Options
All
  • Public
  • Public/Protected
  • All
Menu

Class Lexer

Hierarchy

  • Lexer

Index

Constructors

constructor

  • Parameters

    • tokenClasses: TokenConstructor[]

      constructor functions for the Tokens types this scanner will support These constructors must be in one of THESE forms:

      1. With a PATTERN property that has a RegExp value for tokens to match: example: -->class Integer extends Token { static PATTERN = /[1-9]\d }<--

      2. With a PATTERN property that has the value of the var Lexer.NA defined above. This is a convenience form used to avoid matching Token classes that only act as categories. example: -->class Keyword extends Token { static PATTERN = NA }<--

      The following RegExp patterns are not supported: a. '$' for match at end of input b. /b global flag c. /m multi-line flag

      The Lexer will identify the first pattern the matches, Therefor the order of Token Constructors passed To the SimpleLexer's constructor is meaningful. If two patterns may match the same string, the longer one should be before the shorter one.

      Note that there are situations in which we may wish to place the longer pattern after the shorter one. For example: keywords vs Identifiers. 'do'(/do/) and 'done'(/w+)

      • If the Identifier pattern appears before the 'do' pattern both 'do' and 'done' will be lexed as an Identifier.

      • If the 'do' pattern appears before the Identifier pattern 'do' will be lexed correctly as a keyword. however 'done' will be lexed as TWO tokens keyword 'do' and identifier 'ne'.

        To resolve this problem, add a static property on the keyword's Tokens constructor named: LONGER_ALT example:

        export class Identifier extends Keyword { static PATTERN = /[_a-zA-Z][_a-zA-Z0-9]/ } export class Keyword extends Token {

         static PATTERN = lex.NA
         static LONGER_ALT = Identifier
        

        } export class Do extends Keyword { static PATTERN = /do/ } export class While extends Keyword { static PATTERN = /while/ } export class Return extends Keyword { static PATTERN = /return/ }

        The lexer will then also attempt to match a (longer) Identifier each time a keyword is matched

    • Default value deferDefinitionErrorsHandling: boolean = false

    Returns Lexer

Properties

Protected allPatterns

allPatterns: RegExp[]

Protected emptyGroups

emptyGroups: object

Type declaration

  • [groupName: string]: Token

lexerDefinitionErrors

lexerDefinitionErrors: Array<any>

Protected patternIdxToCanLineTerminator

patternIdxToCanLineTerminator: boolean[]

Protected patternIdxToClass

patternIdxToClass: Function[]

Protected patternIdxToGroup

patternIdxToGroup: boolean[]

Protected patternIdxToLongerAltIdx

patternIdxToLongerAltIdx: number[]

Protected tokenClasses

tokenClasses: TokenConstructor[]

constructor functions for the Tokens types this scanner will support These constructors must be in one of THESE forms:

  1. With a PATTERN property that has a RegExp value for tokens to match: example: -->class Integer extends Token { static PATTERN = /[1-9]\d }<--

  2. With a PATTERN property that has the value of the var Lexer.NA defined above. This is a convenience form used to avoid matching Token classes that only act as categories. example: -->class Keyword extends Token { static PATTERN = NA }<--

The following RegExp patterns are not supported: a. '$' for match at end of input b. /b global flag c. /m multi-line flag

The Lexer will identify the first pattern the matches, Therefor the order of Token Constructors passed To the SimpleLexer's constructor is meaningful. If two patterns may match the same string, the longer one should be before the shorter one.

Note that there are situations in which we may wish to place the longer pattern after the shorter one. For example: keywords vs Identifiers. 'do'(/do/) and 'done'(/w+)

If the Identifier pattern appears before the 'do' pattern both 'do' and 'done' will be lexed as an Identifier.

If the 'do' pattern appears before the Identifier pattern 'do' will be lexed correctly as a keyword. however 'done' will be lexed as TWO tokens keyword 'do' and identifier 'ne'.

To resolve this problem, add a static property on the keyword's Tokens constructor named: LONGER_ALT example:

export class Identifier extends Keyword { static PATTERN = /[_a-zA-Z][_a-zA-Z0-9]/ } export class Keyword extends Token { static PATTERN = lex.NA static LONGER_ALT = Identifier } export class Do extends Keyword { static PATTERN = /do/ } export class While extends Keyword { static PATTERN = /while/ } export class Return extends Keyword { static PATTERN = /return/ }

The lexer will then also attempt to match a (longer) Identifier each time a keyword is matched

Static NA

NA: RegExp

Methods

tokenize

  • Will lex(Tokenize) a string. Note that this can be called repeatedly on different strings as this method does not modify the state of the Lexer.

    Parameters

    • text: string

      the string to lex

    Returns ILexingResult

    }}

Object literals

Static SKIPPED

SKIPPED: object

description

description: string

Generated using TypeDoc