The Python Standard Library » 31. Python Language Services

The Python Standard Library »

31. Python Language Services

>>> for pp in dir(parser):

... print pp

...

ASTType

ParserError

STType

__copyright__

__doc__

__file__

__name__

__package__

__version__

_pickler

ast2list

ast2tuple

compileast

compilest

expr

isexpr()

issuite()

sequence2ast

sequence2st

st2list

st2tuple

suite

tuple2ast

tuple2st

31.2. Abstract Syntax Trees

31.2.1. Node classes

class ast.AST

This is the base of all AST node classes. The actual node classes are derived from the Parser/Python.asdl file, which is reproduced below. They are defined in the _ast C module and re-exported in ast.

There is one class defined for each left-hand side symbol in the abstract grammar (for example, ast.stmt or ast.expr). In addition, there is one class defined for each constructor on the right-hand side; these classes inherit from the classes for the left-hand
side trees. For example, ast.BinOp inherits from ast.expr. For production rules with alternatives (aka “sums”), the left-hand side class is abstract: only instances of specific constructor nodes are ever created.

_fields

Each concrete class has an attribute _fields which gives the names of all child nodes.

Each instance of a concrete class has one attribute for each child node, of the type as defined in the grammar. For example, ast.BinOp instances have an attribute left of type ast.expr.

If these attributes are marked as optional in the grammar (using a question mark), the value might be None. If the attributes can have zero-or-more values (marked with an asterisk), the values are represented as Python lists. All possible attributes
must be present and have valid values when compiling an AST with compile().

lineno

col_offset

Instances of ast.expr and ast.stmt subclasses have lineno and col_offset attributes. The lineno is the line number of source text (1-indexed so the first line is line 1) and the col_offset is the UTF-8 byte offset of the first token that generated the
node. The UTF-8 offset is recorded because the parser uses UTF-8 internally.

The constructor of a class ast.T parses its arguments as follows:

If there are positional arguments, there must be as many as there are items in T._fields; they will be assigned as attributes of these names.

If there are keyword arguments, they will set the attributes of the same names to the given values.

For example, to create and populate an ast.UnaryOp node, you could use

node = ast.UnaryOp()

node.op = ast.USub()

node.operand = ast.Num()

node.operand.n = 5

node.operand.lineno = 0

node.operand.col_offset = 0

node.lineno = 0

node.col_offset = 0

or the more compact

node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0),

lineno=0, col_offset=0)

New in version 2.6: The constructor as explained above was added. In Python 2.5 nodes had to be created by calling the class constructor without arguments and setting the attributes afterwards.

31.2.2. Abstract Grammar

The module defines a string constant __version__ which is the decimal Subversion revision number of the file shown below.

The abstract grammar is currently defined as follows:

-- ASDL's five builtin types are identifier, int, string, object, bool

module Python version "$Revision: 62047 $"

{

mod = Module(stmt* body)

| Interactive(stmt* body)

| Expression(expr body)

-- not really an actual node but useful in Jython's typesystem.

| Suite(stmt* body)

stmt = FunctionDef(identifier name, arguments args,

stmt* body, expr* decorator_list)

| ClassDef(identifier name, expr* bases, stmt* body, expr *decorator_list)

| Return(expr? value)

| Delete(expr* targets)

| Assign(expr* targets, expr value)

| AugAssign(expr target, operator op, expr value)

-- not sure if bool is allowed, can always use int

| Print(expr? dest, expr* values, bool nl)

-- use 'orelse' because else is a keyword in target languages

| For(expr target, expr iter, stmt* body, stmt* orelse)

| While(expr test, stmt* body, stmt* orelse)

| If(expr test, stmt* body, stmt* orelse)

| With(expr context_expr, expr? optional_vars, stmt* body)

-- 'type' is a bad name

| Raise(expr? type, expr? inst, expr? tback)

| TryExcept(stmt* body, excepthandler* handlers, stmt* orelse)

| TryFinally(stmt* body, stmt* finalbody)

| Assert(expr test, expr? msg)

| Import(alias* names)

| ImportFrom(identifier module, alias* names, int? level)

-- Doesn't capture requirement that locals must be

-- defined if globals is

-- still supports use as a function!

| Exec(expr body, expr? globals, expr? locals)

| Global(identifier* names)

| Expr(expr value)

| Pass | Break | Continue

-- XXX Jython will be different

-- col_offset is the byte offset in the utf8 string the parser uses

attributes (int lineno, int col_offset)

-- BoolOp() can use left & right?

expr = BoolOp(boolop op, expr* values)

| BinOp(expr left, operator op, expr right)

| UnaryOp(unaryop op, expr operand)

| Lambda(arguments args, expr body)

| IfExp(expr test, expr body, expr orelse)

| Dict(expr* keys, expr* values)

| ListComp(expr elt, comprehension* generators)

| GeneratorExp(expr elt, comprehension* generators)

-- the grammar constrains where yield expressions can occur

| Yield(expr? value)

-- need sequences for compare to distinguish between

-- x < 4 < 3 and (x < 4) < 3

| Compare(expr left, cmpop* ops, expr* comparators)

| Call(expr func, expr* args, keyword* keywords,

expr? starargs, expr? kwargs)

| Repr(expr value)

| Num(object n) -- a number as a PyObject.

| Str(string s) -- need to specify raw, unicode, etc?

-- other literals? bools?

-- the following expression can appear in assignment context

| Attribute(expr value, identifier attr, expr_context ctx)

| Subscript(expr value, slice slice, expr_context ctx)

| Name(identifier id, expr_context ctx)

| List(expr* elts, expr_context ctx)

| Tuple(expr* elts, expr_context ctx)

-- col_offset is the byte offset in the utf8 string the parser uses

attributes (int lineno, int col_offset)

expr_context = Load | Store | Del | AugLoad | AugStore | Param

slice = Ellipsis | Slice(expr? lower, expr? upper, expr? step)

| ExtSlice(slice* dims)

| Index(expr value)

boolop = And | Or

operator = Add | Sub | Mult | Div | Mod | Pow | LShift

| RShift | BitOr | BitXor | BitAnd | FloorDiv

unaryop = Invert | Not | UAdd | USub

cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn

comprehension = (expr target, expr iter, expr* ifs)

-- not sure what to call the first argument for raise and except

excepthandler = ExceptHandler(expr? type, expr? name, stmt* body)

attributes (int lineno, int col_offset)

arguments = (expr* args, identifier? vararg,

identifier? kwarg, expr* defaults)

-- keyword arguments supplied to call

keyword = (identifier arg, expr value)

-- import name with optional 'as' alias.

alias = (identifier name, identifier? asname)

}

31.2.3. ast Helpers

New in version 2.6.

Apart from the node classes, ast module defines these utility functions and classes for traversing abstract syntax trees:

ast.parse(expr, filename='<unknown>', mode='exec')

ast.literal_eval(node_or_string)

ast.get_docstring(node, clean=True)

ast.fix_missing_locations(node)

ast.increment_lineno(node, n=1)

ast.copy_location(new_node, old_node)

ast.iter_fields(node)

ast.iter_child_nodes(node)

ast.walk(node)

class ast.NodeVisitor

class ast.NodeTransformer

ast.dump(node, annotate_fields=True, include_attributes=False)

31.3. symtable — Access to the compiler’s symbol tables

Symbol tables are generated by the compiler from AST just before bytecode is generated. The symbol table is responsible for calculating the scope of every identifier in the code. symtable provides an interface to examine these tables.

31.3.1. Generating Symbol Tables

symtable.symtable(code, filename, compile_type)

31.3.2. Examining Symbol Tables

class symtable.SymbolTable

A namespace table for a block. The constructor is not public.

get_type()

get_id()

get_name()

get_lineno()

is_optimized()

is_nested()

has_children()

has_exec()

has_import_star()

get_identifiers()

lookup(name)

get_symbols()

get_children()

class symtable.Function

A namespace for a function or method. This class inherits SymbolTable.

get_parameters()

get_locals()

get_globals()

get_frees()

class symtable.Class

A namespace of a class. This class inherits SymbolTable.

get_methods()

class symtable.Symbol

An entry in a SymbolTable corresponding to an identifier in the source. The constructor is not public.

get_name()

is_referenced()

is_imported()

is_parameter()

is_global()

is_declared_global()

is_local()

is_free()

is_assigned()

is_namespace()

get_namespaces()

get_namespace()

>>> for st in dir(symtable):

... print st

...

CELL

Class

DEF_BOUND

DEF_GLOBAL

DEF_IMPORT

DEF_LOCAL

DEF_PARAM

FREE

Function

GLOBAL_EXPLICIT

GLOBAL_IMPLICIT

LOCAL

OPT_BARE_EXEC

OPT_EXEC

OPT_IMPORT_STAR

SCOPE_MASK

SCOPE_OFF

Symbol

SymbolTable

SymbolTableFactory

USE

__all__

__builtins__

__doc__

__file__

__name__

__package__

_newSymbolTable

_symtable

symtable

weakref

31.4. symbol — Constants used with Python parse trees

This module provides constants which represent the numeric values of

internal nodes of the parse tree. Unlike most Python constants,

these use lower-case names. Refer to the file Grammar/Grammar in the

Python distribution for the definitions of the names in the context

of the language grammar. The specific numeric values which the names

map to may change between Python versions.

This module also provides one additional data object:

symbol.sym_name

Dictionary mapping the numeric values of the constants defined in this

module back to name strings, allowing more human-readable representati

on of parse trees to be generated.

>>> for sy in dir(symbol):

... print sy

...

__builtins__

__doc__

__file__

__name__

__package__

_name

_value

and_expr

and_test

arglist

argument

arith_expr

assert_stmt

atom

augassign

break_stmt

classdef

comp_for

comp_if

comp_iter

comp_op

comparison

compound_stmt

continue_stmt

decorated

decorator

decorators

del_stmt

dictorsetmaker

dotted_as_name

dotted_as_names

dotted_name

encoding_decl

eval_input

except_clause

exec_stmt

expr

expr_stmt

exprlist

factor

file_input

flow_stmt

for_stmt

fpdef

fplist

funcdef

global_stmt

if_stmt

import_as_name

import_as_names

import_from

import_name

import_stmt

lambdef

list_for

list_if

list_iter

listmaker

main

not_test

old_lambdef

old_test

or_test

parameters

pass_stmt

power

print_stmt

raise_stmt

return_stmt

shift_expr

simple_stmt

single_input

sliceop

small_stmt

stmt

subscript

subscriptlist

suite

sym_name

term

test

testlist

testlist1

testlist_comp

testlist_safe

trailer

try_stmt

varargslist

while_stmt

with_item

with_stmt

xor_expr

yield_expr

yield_stmt

31.5. token — Constants used with Python parse trees

This module provides constants which represent the numeric values

of leaf nodes of the parse tree (terminal tokens). Refer to the file

Grammar/Grammar in the Python distribution for the definitions of

the names in the context of the language grammar. The specific

numeric values which the names map to may change between Python

versions.

This module also provides one data object and some functions. The

functions mirror definitions in the Python C header files.

token.tok_name

Dictionary mapping the numeric values of the constants defined in

this module back to name strings, allowing more human-readable

representation of parse trees to be generated.

token.ISTERMINAL(x)

token.ISNONTERMINAL(x)

token.ISEOF(x)

>>> for to in dir(token):

... print to

...

AMPER

AMPEREQUAL

AT

BACKQUOTE

CIRCUMFLEX

CIRCUMFLEXEQUAL

COLON

COMMA

DEDENT

DOT

DOUBLESLASH

DOUBLESLASHEQUAL

DOUBLESTAR

DOUBLESTAREQUAL

ENDMARKER

EQEQUAL

EQUAL

ERRORTOKEN

GREATER

GREATEREQUAL

INDENT

ISEOF

ISNONTERMINAL

ISTERMINAL

LBRACE

LEFTSHIFT

LEFTSHIFTEQUAL

LESS

LESSEQUAL

LPAR

LSQB

MINEQUAL

MINUS

NAME

NEWLINE

NOTEQUAL

NT_OFFSET

NUMBER

N_TOKENS

OP

PERCENT

PERCENTEQUAL

PLUS

PLUSEQUAL

RBRACE

RIGHTSHIFT

RIGHTSHIFTEQUAL

RPAR

RSQB

SEMI

SLASH

SLASHEQUAL

STAR

STAREQUAL

STRING

TILDE

VBAR

VBAREQUAL

__builtins__

__doc__

__file__

__name__

__package__

main

tok_name

31.6. keyword — Testing for Python keywords

This module allows a Python program to determine if a string is a

keyword.

keyword.iskeyword(s)

keyword.kwlist

Sequence containing all the keywords defined for the interpreter.

If any keywords are defined to only be active when particular

__future__ statements are in effect, these will be included as well.

>>> dir(keyword)

['__all__', '__builtins__', '__doc__', '__file__', '__name__',

'__package__', 'iskeyword', 'kwlist', 'main']

31.7. tokenize — Tokenizer for Python source

The tokenize module provides a lexical scanner for Python source

code, implemented in Python. The scanner in this module returns

comments as tokens as well, making it useful for implementing

“pretty-printers,” including colorizers for on-screen displays.

The primary entry point is a generator:

tokenize.generate_tokens(readline)

An older entry point is retained for backward compatibility:

tokenize.tokenize(readline[, tokeneater])

tokenize.COMMENT

tokenize.NL

Another function is provided to reverse the tokenization process:

tokenize.untokenize(iterable)

>>> for tn in dir(tokenize):

... print tn

...

AMPER

AMPEREQUAL

AT

BACKQUOTE

Binnumber

Bracket

CIRCUMFLEX

CIRCUMFLEXEQUAL

COLON

COMMA

COMMENT

Comment

ContStr

DEDENT

DOT

DOUBLESLASH

DOUBLESLASHEQUAL

DOUBLESTAR

DOUBLESTAREQUAL

Decnumber

Double

Double3

ENDMARKER

EQEQUAL

EQUAL

ERRORTOKEN

Expfloat

Exponent

Floatnumber

Funny

GREATER

GREATEREQUAL

Hexnumber

INDENT

ISEOF

ISNONTERMINAL

ISTERMINAL

Ignore

Imagnumber

Intnumber

LBRACE

LEFTSHIFT

LEFTSHIFTEQUAL

LESS

LESSEQUAL

LPAR

LSQB

MINEQUAL

MINUS

NAME

NEWLINE

NL

NOTEQUAL

NT_OFFSET

NUMBER

N_TOKENS

Name

Number

OP

Octnumber

Operator

PERCENT

PERCENTEQUAL

PLUS

PLUSEQUAL

PlainToken

Pointfloat

PseudoExtras

PseudoToken

RBRACE

RIGHTSHIFT

RIGHTSHIFTEQUAL

RPAR

RSQB

SEMI

SLASH

SLASHEQUAL

STAR

STAREQUAL

STRING

Single

Single3

Special

StopTokenizing

String

TILDE

Token

TokenError

Triple

Untokenizer

VBAR

VBAREQUAL

Whitespace

__all__

__author__

__builtins__

__credits__

__doc__

__file__

__name__

__package__

any

double3prog

endprogs

generate_tokens

group

main

maybe

printtoken

pseudoprog

re

single3prog

single_quoted

string

t

tabsize

tok_name

tokenize

tokenize_loop

tokenprog

triple_quoted

untokenize

31.9. pyclbr — Python class browser support

pyclbr.readmodule(module[, path=None])

pyclbr.readmodule_ex(module[, path=None])

31.9.1. Class Objects

Class.module

Class.name

Class.super

Class.methods

Class.file

Class.lineno

31.9.2. Function Objects

Function.module

Function.name

Function.file

Function.lineno

>>> for pc in dir(pyclbr):

... print pc

...

Class

DEDENT

Function

NAME

OP

__all__

__builtins__

__doc__

__file__

__name__

__package__

_getname

_getnamelist

_main

_modules

_readmodule

imp

itemgetter

readmodule

readmodule_ex

sys

tokenize

-------------------------------end of file------------------