• popFront, which should call this mixin's _popFront() and additionally perform any token filtering or shuffling you deem necessary. For example, you can implement popFront to skip comment or tokens.
• A function that serves as the default token lexing function. For most languages this will be the identifier lexing function. This should then be passed to the Lexer template mixin as the template parameter.
• A function that is able to determine if an identifier/keyword has come to an end. This function must return bool and take a single size_t argument representing the number of bytes to skip over before looking for a separating character.
• Any functions referred to in the tokenHandlers template paramater. These functions must be marked pure nothrow, take no arguments, and return a token
• A constructor that initializes the range field as well as calls popFront() exactly once (to initialize the _front field).

Parameters

Examples

struct CalculatorLexer
{
   mixin Lexer!(IdType, Token, defaultTokenFunction, isSeparating,
       staticTokens, dynamicTokens, possibleDefaultTokens, tokenHandlers);

   this (ubyte[] bytes)
   {
       this.range = LexerRange(bytes);
       popFront();
   }

   void popFront() pure
   {
       _popFront();
   }

   Token lexNumber() pure nothrow @safe
   {
       // implementation goes here
   }

   Token lexWhitespace() pure nothrow @safe
   {
       // implementation goes here
   }

   Token defaultTokenFunction() pure nothrow @safe
   {
       // There is no default token in the example calculator language, so
       // this is always an error.
       range.popFront();
       return Token(tok!"");
   }

   bool isSeparating(size_t offset) pure nothrow @safe
   {
       // For this example language, always return true.
       return true;
   }
}

Variables

LexerRange

Token

Functions