Python exposes an abstract syntax tree (AST) representing the compiled form of Python source code in the AST module. The AST module allows user code to inspect and manipulate of the AST representation, in between parsing of source and compilation of bytecode.
Although the meaning of Python code is defined by the language reference, the AST module is a CPython implementation detail, and is not required to be implemented in other Python implementations.
Compatibility of the AST
As part of work to rewrite the CPython peephole optimizer to work on the AST (rather than on the raw bytecode, as is currently the case), Eugene Toder needed to make some changes to the structure of the AST. As a CPython implementation detail, it wasn't immediately clear what backward compatibility policies applied to the AST. So, Eugene asked the question on python-dev. Was it necessary, when changing the AST, to ensure backward compatibility?
The general consensus was that compatibility is not required. The AST module exposes a constant, ast.__version__, which provides a means for user code to vary its behaviour depending on the version of the AST it encounters. This was viewed as sufficient compatibility for an implementation-specific module.
Other Python Implementations
In actual fact, both Jython and IronPython maintainers pointed out that their respective implementations either had a compatible AST module, or intended to provide one. Even so, they did not feel that this meant that the AST should be frozen, and were happy that as long as the ast.__version__ constant changed, the AST could be modified in incompatible ways.
One point that was raised is that a full suite of tests in test_ast.py would help other implementations to ensure that their AST representations were compatible with CPython. Increasing the coverage of test_ast.py would make a good project for someone who wanted to get involved with Python internals!
What Will Happen Next?
The patch which started the discussion is not yet included in CPython. So possibly, nothing will happen. However, if it does get committed, the AST will change in an incompatible way. The ast.__version__ constant will change to reflect this, so user code will know, but changes will be needed. More generally, this will be the way AST changes will be handled in future.
The Python developers are interested in how widely the AST is used, and how much impact this policy will have. If any readers have code that will be affected by the change, they are encouraged to participate in the discussion on python-dev.