York 's Bookmarks tagged parser →  View Popular

Internally, lex.py uses the re module to do its patten matching. When building the master regular expression,
rules are added in the following order:

1. All tokens defined by functions are added in the same order as they appear in the lexer file.
   
2. Tokens defined by strings are added next by sorting them in order of decreasing regular expression length (longer expressions
   are added first).
   

def p_foo(p):
    """foo : abcd
           | abcx"""

def p_abcd(p):
    "abcd : A B C D"

def p_abcx(p):
    "abcx : A B seen_AB C X"

def p_seen_AB(p):
    "seen_AB :"

yacc.yacc(method="SLR")

yacc.parse(debug=1, debugfile="debugging.out")

def p_empty(p):
    'empty :'
    pass

def p_optitem(p):
    'optitem : item'
    '        | empty'
    ...

def p_binary_operators(p):
    '''expression : expression '+' term
                  | expression '-' term
       term       : term '*' factor
                  | term '/' factor'''
    if p[2] == '+':
        p[0] = p[1] + p[3]
    elif p[2] == '-':
        p[0] = p[1] - p[3]
    elif p[2] == '*':
        p[0] = p[1] * p[3]
    elif p[2] == '/':
        p[0] = p[1] / p[3]

# Literals.  Should be placed in module given to lex()
literals = ['+','-','*','/' ]

def p_expressions(p):
    '''expression : expression MINUS expression
                  | MINUS expression'''
    if (len(p) == 4):
        p[0] = p[1] - p[3]
    elif (len(p) == 3):
        p[0] = -p[2]

If any errors are detected in your grammar specification, yacc.py will produce
diagnostic messages and possibly raise an exception. Some of the errors that can be detected include:

• Duplicated function names (if more than one rule function have the same name in the grammar file).
  
• Shift/reduce and reduce/reduce conflicts generated by ambiguous grammars.
  
• Badly specified grammar rules.
  
• Infinite recursion (rules that can never terminate).
  
• Unused rules and tokens
  
• Undefined rules and tokens

To resolve ambiguity, especially in expression grammars, yacc.py allows individual
tokens to be assigned a precedence level and associativity. This is done by adding a variable
precedence to the grammar file like this:

precedence = (
    ('left', 'PLUS', 'MINUS'),
    ('left', 'TIMES', 'DIVIDE'),
)

It is also possible to specify non-associativity in the precedence table. This would
be used when you don't want operations to chain together. For example, suppose
you wanted to support comparison operators like < and > but you didn't want to allow
combinations like a < b < c. To do this, simply specify a rule like this:

precedence = (
    ('nonassoc', 'LESSTHAN', 'GREATERTHAN'),  # Nonassociative operators
    ('left', 'PLUS', 'MINUS'),
    ('left', 'TIMES', 'DIVIDE'),
    ('right', 'UMINUS'),            # Unary minus operator
)

If you do this, the occurrence of input text such as a < b < c will result in a syntax error. However, simple
expressions such as a < b will still be fine.

To handle reserved words, it is usually easier to just match an identifier and do a special name lookup in a function
like this:

reserved = {
   'if' : 'IF',
   'then' : 'THEN',
   'else' : 'ELSE',
   'while' : 'WHILE',
   ...
}

def t_ID(t):
    r'[a-zA-Z_][a-zA-Z_0-9]*'
    t.type = reserved.get(t.value,'ID')    # Check for reserved words
    return t

Literal characters can be specified by defining a variable literals in your lexing module. For example:

literals = [ '+','-','*','/' ]

literals = "+-*/"

Here each function accepts a single argument, t,
which is a tuple. The values of t[i] are mapped to grammar symbols as
shown here:

def p_statement_plus(t):
    'statement : statement ADDOP term'
    #   ^            ^        ^    ^
    #  t[0]         t[1]     t[2] t[3]

    t[0] = t[1] + t[3]

1. The names of all of the valid token types are declared as a list of strings named tokens.

2. Tokens that require no special processing are declared using module-level variables named t_TOKEN_TYPE where TOKEN_TYPE matches some string in the "tokens" list.