Jay Taylor's notes

back to listing index

Python is Actually Portable

[web search]
Original source (ahgamut.github.io)
Tags: python single-file-multi-os-executable ape actually-portable cosmopolitan-libc ahgamut.github.io
Clipped on: 2022-07-31

Python is Actually Portable

July 13, 2021

Update (2022-07-27) : This post describes a proof-of-concept Python executable (2.7.18 and 3.6.14) built on Cosmopolitan Libc, which allows it to run on six different operating systems (Linux, Mac, Windows, NetBSD, FreeBSD, OpenBSD). It’s been more than a year since I put this together, and now Python3.6 and its test suite are part of the Cosmopolitan Libc monorepo. There’s been a LOT of work done to improve python.com over vanilla Python3.6 – a custom sys.meta_path entry for faster loading, size reductions of the unicodedata, a cosmo module for Cosmopolitan-specific goodies, a revamp of Python’s build and testing system, a superb REPL experience, backports from Python 3.7, and a lot more! So some of the details in this blog post are likely out of date. Check out the history of APE Python here, build it via the Cosmopolitan monorepo, or download python.com from here. I’m also trying to port the newer versions of Python and some well-known Python packages like numpy. Join the discussion about Python and Cosmopolitan Libc in the redbean Discord server: https://discord.gg/rGQja6kS

Back in February, I put together Lua 5.4 using Cosmopolitan Libc as a quick proof-of-concept, and that led to Lua 5.4 being vendored as part of the Cosmopolitan repository on Github, along with many other interesting developments. It’s pretty exciting to try and compile well-known C projects using Cosmopolitan; the portability reward is great motivation, and I get to observe the design, coding style, and build system of high-quality codebases.

However, my initial plan with Cosmopolitan was not to compile Lua, it was to compile Python. Since February, I’ve been trying to understand how Cosmopolitan works, reading the repo code and submitting PRs occasionally, and finally I have an actually portable version of Python 2.7.18 (and 3.6.141) – you can build it from the repository here. It’s not solid as Lua 5.4 because it currently passes only a third of the regression tests, but most of the parts are there. For example, here’s a GIF showing a simple Flask webapp running via the python.com APE.

Image (Asset 1/1) alt=
  1. Did you click on this before reading the remainder of the post because you were wondering about Python 3? I know Python 2.7 reached EOL last year, but the Python 2.7 codebase was easier to read, change, and debug, so I was able to get a better idea of where I had to make changes. I put together a Python 3.6.14 APE (build it from the repo here), and it took much less time and experimentation because I knew exactly where to look. ↩︎

  2. I originally uploaded this post like twelve hours, in the excitement of seeing a Flask webapp work on the APE. Unfortunately, I forgot to test on Windows before putting the post out, and Windows, true to its nature caused several annoyances which together took a couple of hours to sort out. However, I think it ought to work now. ↩︎

  3. I originally tried the complicated solution of writing a new configure script from scratch, but I got lost in the docs for autoconf, and resorted to the simpler fix of just editing the shell script. This issue only happened because I used dummy headers with the amalgamation; if you have the Cosmopolitan repo nearby, you can use -isystem cosmopolitan/libc/isystem instead of -I dummy_headers -include cosmopolitan.h, which would avoid confusing configure↩︎

  4. I remembered this a few days back: if a name in your C file clashes with a header name somewhere, you can do something like:

    #ifdef name
    #define name __name
    #undef name

    and avoid the name clash instead of doing a find-and-replace. ↩︎

  5. Over the course of getting the APE to work, my muscle memory is to call python -BESsuv, which avoids a bunch of normal startup stuff and adds stderr information about imports. This is super useful when building python, but one time I was running a local script and couldn’t figure out why it was failing before I saw that I had typed python -BESsuv script.py every single time. ↩︎

  6. I was pleasantly surprised to find out that _sqlite and _bz2 could be compiled, because I didn’t expect SQLite or libbz2 would be easy to compile from source with Cosmopolitan. SQLite is now vendored in the Cosmopolitan repo.

    On the other hand, I wanted readline because I like to have the (press-Up-arrow-for-previously-typed-line) in the REPL, but I didn’t know how terminfo worked. _ctypes was also a disappointment because I had to compile libffi from source, and then I saw most of _ctypes' use comes from having dlopen↩︎

  7. for example, greenlet contains a C extension called _greenlet which it refers to by doing things like the following in __init__.py:

    from ._greenlet import foo
    from greenlet._greenlet import bar

    now __init__.py expects _greenlet.so to be present in the same directory, but that is not possible because the extension is compiled into the interpreter statically. So I had to manually change this to use from _greenlet without the dot. This also means I have to change the extension name in the PyModuleDef part of the C source code: it’s not greenlet._greenlet anymore, it’s just _greenlet↩︎

  8. This works because most libraries use only the high-level threading API to handle their threads. A notable counter-example is Django: I tried getting Django to work before figuring out how _greenlet could be compiled, but Django imports the low-level _thread API to handle things. ↩︎

  9. It is possible to have SSL support: you just have to build OpenSSL with Cosmopolitan Libc. I was able to build libssl.a, link it with python.com via Modules/Setup, and pip worked. However, I have not tested it with the OpenSSL test suite. ↩︎

  10. The Cosmopolitan repo builds python.com using PYOBJ.COM, a minimal Python executable that creates .pyc files and symbols for the linker to resolve dependencies. ↩︎