Dr. Dobb's | Prototyping Interpreters using Python Lex-Yacc | 1, 2004 - The Diigo Meta page

• 29 Sep 07
  

  York Jong

  Lex PLY Python Yacc compiler parser scanner LALR(1) docstring

  • PLY relies on Python reflection, so there are no separate Lex and Yacc files—everything is in Python.

  •  1. The names of all of the valid token types are declared as a list of strings named tokens.

     

      

     2. Tokens that require no special processing are declared using module-level variables named t_TOKEN_TYPE where TOKEN_TYPE matches some string in the "tokens" list.
  • Each such function has a doc string that is a regular expression that matches the respective token. The order of these functions is important since each regular expression is tried in order.
  • Unlike Lex in C, which uses unions, the returned value has an attribute named "value" that can be set to a value of any type (even an instance). This flexibility is convenient. If None is returned or if no return is encountered, the lexer ignores the current token and moves on to the next token. This can be used to implement whitespace handling, but a module-level variable named t_ignore is more efficient for this task.

  2 more annotations...

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.