A proposal for /usr/bin/python between Python 2 and 3

I just got back from PyCon, where a few debian-python team members met to discuss some goals for Python packaging in Debian over the next release cycle. One item of interest is moving away from installing Python 2 by default (expecting it to be desupported in 2020), which raises questions about what /usr/bin/python should mean. At the moment, it very strongly means Python 2 specifically, so there's a question about what it should be on a system with only Python 3 installed—should it become Python 3? Should it be nonexistent?

PEP 0394 recommends that Python 2 should be installed as python2 and Python 3 as python3, and that python should mean Python 2 for now. I think it's important that python continue to mean Python 2, for the reasons described in the "Migration Notes" section of that PEP. In particular, I think it's very important that you should be able to install Python 2 (for the near future), even if your system did not ship with Python 2, and that the system version of Python 3 should not prevent you from using python to run Python 2 applications.

However, not shipping /usr/bin/python by default also has its downsides. I made a suggestion on the debian-python list for handling this: this blog post is a more detailed version of that proposal.

The basic motivation for this proposal is that third-party script authors love Python in part because it's just about universally available. #!/usr/bin/env python, today, is sort of like #!/bin/sh: you don't necessarily get nice things (Python 3-only features and bash-only features, respectively), but it works. If the python command stops existing, this is inconvenient for end users, who will need to manually edit scripts or install a version of Python themselves, and authors, who may just decide to use a worse language like #!/bin/sh. It's also bad for the Python language community, since it removes an incentive to use Python.

So it would be nice to keep the python command working usefully on both Python 2-only systems and Python 3-only systems. Fortunately, it's pretty doable to port code to work on both Python 2 and 3 with the same source. Especially for third-party scripts with the goal of working out-of-the-box in as many places as possible, writing code to these restrictions isn't particularly onerous, since they needed to stay Python 2-compatible anyway. I've been writing a bunch of Python code recently as polyglot Python 2/3, and the biggest problem has been limiting myself to features in Python 2.7, not getting things to work in both versions of the language.

So here's the proposal: we install a wrapper binary as python, that defaults to launching Python 2 (or reporting an error if Python 2 is not installed). However, Python 2/3-compatible scripts can include a specific marker indicating they can be run on Python 3, which makes these scripts able to run on both Python 2-only systems and Python 3-only systems.

This marker is based on the existing coding: syntax from PEP 0263: it's a "magic comment" on the first or second line of the form pyversions=2.7+,3.3+, indicating which Python major and minor versions are supported. A script compatible with both Python 2 and 3 should include one of these magic comments, and also include a shebang line launching just python. Here's an example based on one from PEP 0263:

#!/usr/bin/env python
# -*- coding=utf-8 -*- pyversions=2.6+,3.3+

from __future__ import print_function

print(u"Hello world!")

On new systems, the python command itself is a wrapper binary that knows what versions of the Python interpreter are installed. If it detects a pyversions comment, it will exec the newest Python interpreter compatible with the comment. For this example, if Python 3.3 is installed, it will launch that: if only Python 3.2 and 2.7 are installed, it will use Python 2.7, since Python 3.2 does not support the u"" literal syntax. Otherwise, it will assume that code is Python 2-only, and exec the newest Python 2 version installed. In either case, if a compatible interpreter cannot be found, it will print an error and exit.

On legacy systems, the python command refers to Python 2. So Python 2/3-compatible scripts will still be able to find an interpreter they are compatible with.

For legacy scripts that use a shebang stating just python, on both new and legacy systems, they will only ever run on Python 2, or not at all. This preserves the existing API, and avoids the poor user experience of running Python 2 scripts with the Python 3 interpreter, as PEP 0394 warns against doing.

However, the python wrapper supports running Python 2/3-compatible scripts with the Python 3 interpreter, which is useful for Python 3-only systems. A future version of Debian can choose to ship the Python 3 interpreter only, and remain compatible with third-party scripts that include the pyversions magic comment. Meanwhile, these third-party scripts can state #!/usr/bin/env python in their shebang, and remain compatible with legacy Python 2-only systems, including those (like Mac OS X) that do not include the python2 symbolic link.

The python wrapper can run in three possible modes:

• As an interpreter, it follows the rules previously discussed. The "interpreter" mode covers both invocation via the shebang of a Python script, and invocation of the form python script.py.
• For interactive use, it will just run the newest version of Python on the system. This allows, for instance, teachers of the language to have their students just run python, without the pedagogical speed bump of having to explain versions of the language. Running the latest major version of Python is safe, since the interpreter will print out its own version at startup.
• Uses such as python -c are considered scripted use, since in this context, the python command is also serving as an API, and existing users of the API expect Python 2 just as much as scripts do. (Imagine, for instance, a shell script with KEY=$(python -c "import local_settings; print SECRET_KEY")), which will fail if python means Python 3.) In this mode, the wrapper will instead look for an environment variable named PYVERSIONS. If this is set, it will be parsed as the value of the pyversions magic comment. So shell scripts that include Python 2/3-compatible python -c commands can e.g. export PYVERSIONS=2.7+,3.3+ at the top of the script, and work on systems with either just Python 2 or just Python 3.

The end result here is that third-party script authors can continue writing polyglot Python 2/3 code for the indefinite future, and be compatible with both existing Python 2-only distributions and newer Python 3-only distributions. Meanwhile, distributions can move towards installing Python 3 only by default and avoid installing Python 2 as far as possible, without closing off the ability to install Python 2 when needed, or breaking legacy Python 2-only code.

This is a good transition story for distributions like Debian, Ubuntu, and Fedora that are currently trying to move to Python 3 only, but don't want to break existing code. It's also a good transition story for distributions like Arch that have already started moving /usr/bin/python to Python 3: interactive and scripted use of the python command can continue to launch Python 3, but compatibility with third-party Python 2 scripts is regained. And most importantly, it's good for third-party script authors, who want to write one script that works on Debian, Ubuntu, Fedora, Arch, and Mac OS X alike, and for their end users.

I'm interested in initial feedback on this idea: if it generally seems like a good plan, I'll firm up the details and write it up as a formal PEP. There are a few details to work out, like how this interacts with the py.exe Windows launcher described in PEP 0397 (if at all), but my focus is making /usr/bin/python useful and backwards-compatible on UNIX-like platforms.

Edit 3 May 2015: I've uploaded a proof-of-concept implementation of this launcher to GitHub to see what the overhead and complexity of this approach looks like.