Coding Across Python Versions

The asyncio library in Python is not very old, relative to the language: it was introduced in Python 3.4. But even during its short lifetime, asyncio has undergone numerous API changes. One important example is the lineage of the asyncio.run function. Introduced in Python 3.7, asyncio.run is a high-level API to the older asyncio.loop.run_until_complete method.

So if code runs under Python 3.7+ then it can make use of run, otherwise it needs to rely on asyncio.get_event_loop().run_until_complete.

This isn’t a problem if you are writing an application, since you’ll probably run that application under a well-known version of Python – the version you choose.

But if you are writing a library, you might need to support Python <=3.6 and Python3.7+. The code might look something like this:

# code courtesy of @mark-nick-o on mypy issue #12331: https://github.com/python/mypy/issues/12331
if (sys.version_info.major == 3 and sys.version_info.minor >= 7):
    asyncio.run(main())
elif (sys.version_info.major == 3 and sys.version_info.minor <= 6) and (sys.version_info.major == 3 and sys.version_info.minor >= 5):
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.wait(main()))
else:
    print("you currently need to have python 3.5 at least")

That code isn’t too bad, but can we make it better?

compatlib

This is the reason I decided to write compatlib.

Instead of branching if-else statements, compatlib lets you perform method overloading based on the version of the python interpreter:

# my_module.py

import asyncio
from compatlib import compat

@compat.after(3, 4)
def main() -> None:
    asyncio.get_event_loop().run_until_complete(coro())
    return 3.4

@compat.after(3, 7)
def main() -> None:
    asyncio.run(coro())
    return 3.7

This means that invoking my_module.main() on a Python 3.6 interpreter will return 3.4, while running it on Python 3.7 will return 3.7.

To my eyes, the compatlib implementation is much more readable: instead of grokking sys.version_info checks, you can clearly see (major, minor) version tuples and the associated code. But that doesn’t make the compatlib code better than the first example – and it definitely doesn’t make compatlib a good idea.

It wasn’t until I shared compatlib with the world that I realized how dangerous it is.

The conversation that formed around compatlib was far more incisive and interesting than I could have imagined. Folks were discussing the merits and prevalence of librarires supporting multiple Python versions, the quickest and easiest ways to reproduce certain subsets of compatlib features, the parallels with awk's BEGIN blocks, the absence of macros in Python.

And one comment stood out to me – far beyond the rest, because it succinctly captured my thoughts on compatlib: don’t use this.

compatlib makes code more “magical”: by hiding the machinery of “interpreter version checks” behind compat.after, you introduce a lot of unecessary indirection to your code. And, if compatlib breaks (which it very well could, because it hasn’t been thoroughly tested), you now have to debug a library, instead of debugging a few if-else checks.

In fact, it was a recent post from David Bieber in which David shares “magic functions”, sometimes known as implicit arguments. David is very direct:

The short answer is that you should never use magic functions.

It’s amazing to see what the Python community accepts and rejects; what goes into and out of vogue. Hopefully, compatlib stays out of vogue, in favor of simpler, less magical alternatives.