icon_lexer

Lexer for Icon: converts a sequence of lines containing Icon code into a sequence of tokens, preserving location and source contents.

Limitations:

Procedures

icon_lexer (line_generator, include_comments)

Generates tokens from a given line generator. 

  > every token := icon_lexer(["2 + 3"]) do write(token.kind, " ", token.text)
  INT_LIT 2
  PLUS +
  INT_LIT 3
  EOFX

line_generator
A line generator (list or file) to process. A string will be converted to a one-element list containing that string.
include_comments
optional non-null value to include a token for comments

Records

lexer_token (kind, text, line_number, line_column)

Tokens represent individual elements in an Icon program, such as number literals, keywords or operators. The token record holds information about each token, including its type and position in the source file.

kind
is the Icon type, such as "BREAK" or "INT_LIT"
text
is the raw text, as appearing in the source code
line_number
is the number of the line this token is on
line_column
is the column number of the start of this token
Home