"Fossies" - the Fresh Open Source Software Archive

Member "gentle-97/html/descrtok.html" (20 Aug 1998, 2138 Bytes) of package /linux/misc/old/gentle-97.tar.gz:

Caution: In this restricted "Fossies" environment the current HTML page may not be correctly presentated and may have some non-functional links. You can here alternatively try to browse the pure source code or just view or download the uninterpreted raw source code. If the rendering is insufficient you may try to find and view the page on the gentle-97.tar.gz project site itself.


How To Describe a Token

For each token Token introduced in the Gentle specification, there must be a token description file Token.t . This file must specify a Lex rule that handles the token. Such a rule is given by a regular expression that matches the token, and an action that computes the attributes of the token.

   pattern { action }
If the rule does not fit on one line, the action may be written on several lines, but the opening brace must appear on the first line.

By way of an example, a token introduced in a Gentle specification as

   'token' Number (-> INT)
may be described in file Number.t:

   [0-9]+ {
      yylval.attr[1] = atoi(yytext);
      return Number;
The pattern

matches a non-empty sequence of digits.

The line

   yylval.attr[1] = atoi(yytext);
converts the matched input token ( yytext) into an integer (using the C function atoi) and assigns it as the first attribute. The Lex variable yylval has a field attr which is an array of token attributes. attr[ i] stands for the i-th attribute.

The macro

defines the source position of the token (the source position is stored in yylval.attr[0]). In a Gentle specification, this value may be accessed with the @-predicate.

The line

   return Number;
returns the code of the token. The token code is a constant that has the same name as the token.