PLY (Python Lex-Yacc) - The Diigo Meta page

• 09 Sep 11
  

  Christian Haintz

  python parser grammar toddl lex yacc

• 26 May 11
  

  Eric Shull

  python ply lex

• 02 May 11
  

  bayo opadeyi

  python-lex-yacc tutorial

  python lex yacc ply compiler

  • information
  • lexer.token(). Return the next token. Returns a special LexToken instance on success or None if the end of the input text has been reached.
  • le reserved words, you should write a single rule to match an identifier and do a special name lookup in
  • def t_ID(t):  ...  # Look up symbol table information and return a tuple  t.value = (t.value, symbol_lookup(t.value))  ...  return t
  • table
  • 

  4 more annotations...

• 22 Mar 11
  

  Keith Yang

  python yacc lex reference

• 20 Nov 10
  

  Marcin Kasperski

  python parsing ply yacc lexx

• 11 Nov 10
  

  Hsu Wei-Cheng

  compiler yacc python ply 991course

• 20 Oct 10
  

  Russ

  • 8. Using Python's Optimized Mode

       Because PLY uses information from doc-strings, parsing and lexing information must be gathered while running the Python interpreter in normal mode (i.e., not with the -O or -OO options). However, if you specify optimized mode like this:  

     

    lex.lex(optimize=1) yacc.yacc(optimize=1) 

     

      then PLY can later be used when Python runs in optimized mode. To make this work, make sure you first run Python in normal mode. Once the lexing and parsing tables have been generated the first time, run Python in optimized mode. PLY will use the tables without the need for doc strings.  

     Beware: running PLY in optimized mode disables a lot of error checking. You should only do this when your project has stabilized and you don't need to do any debugging. One of the purposes of optimized mode is to substantially decrease the startup time of your compiler (by assuming that everything is already properly specified and works).
• 03 Mar 10
  

  Ian Weatherhogg

  python yacc parsers lex ply

• 15 Sep 09
  

  J. Felix Ontañón

  python parsing

• 19 Jul 09
  

  alex madon

  python yacc parser

• 27 Dec 08
  

  euribates

  python programming reference lax yacc

• 15 Dec 08
  

  zed usi

  python ply lex yacc

• 07 Nov 08
  

  Daniel Demski

  ply yacc lex python parsing

• 29 Sep 07
  

  York Jong

  LALR(1) Lex PLY Python SLR(1) Yacc compiler parser scanner docstring

  • yacc.py can handle empty productions by defining a rule like this:  

     

    def p_empty(p):     'empty :'     pass 

     

      Now to use the empty production, simply use 'empty' as a symbol. For example:  

     

    def p_optitem(p):     'optitem : item'     '        | empty'     ... 

     

      Note: You can write empty rules anywhere by simply specifying an empty right hand side. However, I personally find that writing an "empty" rule and using "empty" to denote an empty production is easier to read.

  •  Internally, lex.py uses the re module to do its patten matching. When building the master regular expression, rules are added in the following order: 

     

     
    1. All tokens defined by functions are added in the same order as they appear in the lexer file. 
    2. Tokens defined by strings are added next by sorting them in order of decreasing regular expression length (longer expressions are added first). 
     

  •  To handle reserved words, it is usually easier to just match an identifier and do a special name lookup in a function like this:  

     

    reserved = {    'if' : 'IF',    'then' : 'THEN',    'else' : 'ELSE',    'while' : 'WHILE',    ... }  def t_ID(t):     r'[a-zA-Z_][a-zA-Z_0-9]*'     t.type = reserved.get(t.value,'ID')    # Check for reserved words     return t 

     

      This approach greatly reduces the number of regular expression rules and is likely to make things a little faster.
  • The special t_ignore rule is reserved by lex.py for characters that should be completely ignored in the input stream.  Usually this is used to skip over whitespace and other non-essential characters.  Although it is possible to define a regular expression rule for whitespace in a manner similar to t_newline(), the use of t_ignore provides substantially better lexing performance because it is handled as a special case and is checked in a much more efficient manner than the normal regular expression rules.

  •  Literal characters can be specified by defining a variable literals in your lexing module. For example:  

     

    literals = [ '+','-','*','/' ] 

     

      or alternatively  

     

    literals = "+-*/" 

     

      A literal character is simply a single character that is returned "as is" when encountered by the lexer. Literals are checked after all of the defined regular expression rules
  • def p_expression_plus(p):  'expression : expression PLUS term'  # ^ ^ ^ ^  # p[0] p[1] p[2] p[3]   p[0] = p[1] + p[3]
  • For tokens, the "value" of the corresponding p[i] is the same as the p.value attribute assigned in the lexer module. For non-terminals, the value is determined by whatever is placed in p[0] when rules are reduced. This value can be anything at all. However, it probably most common for the value to be a simple Python type, a tuple, or an instance.

  •  If any errors are detected in your grammar specification, yacc.py will produce diagnostic messages and possibly raise an exception. Some of the errors that can be detected include:  

     
    • Duplicated function names (if more than one rule function have the same name in the grammar file). 
    • Shift/reduce and reduce/reduce conflicts generated by ambiguous grammars. 
    • Badly specified grammar rules. 
    • Infinite recursion (rules that can never terminate). 
    • Unused rules and tokens 
    • Undefined rules and tokens
  • When combining grammar rules into a single function, it is usually a good idea for all of the rules to have a similar structure (e.g., the same number of terms). Otherwise, the corresponding action code may be more  complicated than necessary. However, it is possible to handle simple cases using len(). For example:  

     

    def p_expressions(p):     '''expression : expression MINUS expression                   | MINUS expression'''     if (len(p) == 4):         p[0] = p[1] - p[3]     elif (len(p) == 3):         p[0] = -p[2]
  • If desired, a grammar may contain tokens defined as single character literals. For example:  

     

    def p_binary_operators(p):     '''expression : expression '+' term                   | expression '-' term        term       : term '*' factor                   | term '/' factor'''     if p[2] == '+':         p[0] = p[1] + p[3]     elif p[2] == '-':         p[0] = p[1] - p[3]     elif p[2] == '*':         p[0] = p[1] * p[3]     elif p[2] == '/':         p[0] = p[1] / p[3] 

     

      A character literal must be enclosed in quotes such as '+'. In addition, if literals are used, they must be declared in the corresponding lex file through the use of a special literals declaration.  

     

    # Literals.  Should be placed in module given to lex() literals = ['+','-','*','/' ]
  • When an ambiguous grammar is given to yacc.py it will print messages about "shift/reduce conflicts"  or a "reduce/reduce conflicts". A shift/reduce conflict is caused when the parser generator can't decide whether or not to reduce a rule or shift a symbol on the parsing stack.

  • To resolve ambiguity, especially in expression grammars, yacc.py allows individual tokens to be assigned a precedence level and associativity. This is done by adding a variable precedence to the grammar file like this:  

     

    precedence = (     ('left', 'PLUS', 'MINUS'),     ('left', 'TIMES', 'DIVIDE'), ) 

     

      This declaration specifies that PLUS/MINUS have the same precedence level and are left-associative and that  TIMES/DIVIDE have the same precedence and are left-associative.  Within the precedence declaration, tokens are ordered from lowest to highest precedence. Thus, this declaration specifies that TIMES/DIVIDE have higher precedence than PLUS/MINUS (since they appear later in the precedence specification).

  •  It is also possible to specify non-associativity in the precedence table. This would be used when you don't want operations to chain together. For example, suppose you wanted to support comparison operators like < and > but you didn't want to allow combinations like a < b < c. To do this, simply specify a rule like this:  

     

    precedence = (     ('nonassoc', 'LESSTHAN', 'GREATERTHAN'),  # Nonassociative operators     ('left', 'PLUS', 'MINUS'),     ('left', 'TIMES', 'DIVIDE'),     ('right', 'UMINUS'),            # Unary minus operator ) 

     

      

     If you do this, the occurrence of input text such as a < b < c will result in a syntax error. However, simple expressions such as a < b will still be fine.
  • However, if you insert an embedded action into one of the rules like this,  

     

    def p_foo(p):     """foo : abcd            | abcx"""  def p_abcd(p):     "abcd : A B C D"  def p_abcx(p):     "abcx : A B seen_AB C X"  def p_seen_AB(p):     "seen_AB :" 

     

      an extra shift-reduce conflict will be introduced. This conflict is caused by the fact that the same symbol C appears next in  both the abcd and abcx rules. The parser can either shift the symbol (abcd rule) or reduce the empty rule seen_AB (abcx rule).
  • The default parsing method is LALR. To use SLR instead, run yacc() as follows:  

     

    yacc.yacc(method="SLR") 

     

     Note: LALR table generation takes approximately twice as long as SLR table generation. There is no difference in actual parsing performance---the same code is used in both cases. LALR is preferred when working with more complicated grammars since it is more powerful.
  • To redirect the debugging output to a filename of your choosing, use:  

     

    yacc.parse(debug=1, debugfile="debugging.out")

  14 more annotations...

Page Comments

• 
  bayo opadeyi on 2011-05-02
  to add extra information to the token fxn value, set t.value to a tuple

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.