Misery of manual installation

Developer to the user: This server is a small Python script, and it depends on Concurrence only.

• Install Python from package.
• Start the script. (Fails, needs Concurrence.)
• Try to compile Concurrence. (Fails, needs libevent.)
• Install libevent. (Will fail, package too old.)
• Compile libevent. (Fails first because of header file conflict.)
• Compile and install Concurrence. (Fails, needs Stackless Python.)
• Compile Stackless Python. (Fails, needs libc6-dev, libz-dev, libreadline-dev.)
• Install all library dependencies (hopefully).
• Repeat on every machine.

How to precompile Python programs?

Create single-file executable or installer:

• PyInstaller (cross-platform)
• bbfreeze (cross-platfrom)
• cx_Freeze (cross-platform)
• py2exe for Windows
• py2app for Mac OS X

Bundle all C code into a single-file executable:

• StaticPython (cross-platform)

Cross-platform

• Linux (84.7%)
  • 32-bit i386
  • statically linked
• Mac OS X (3.27%)
  • 32-bit i386, 10.5 or later
  • uses only libSystem.B.dylib and libncurses.5.dylib
• FreeBSD (1.64%)
  • runs the Linux binaries unmodified
  • needs kldload linux
• not supported: Windows (6.56%)
• rest (3.82%)

What's included

• (C)Python 2.7.1 or (C)Python 3.2
• all built-in .py modules
• all (applicable) built-in C extension modules, including proper thread, readline, sqlite3
• greenlet 0.3.1
• full interactive interpreter (python -i)
• all Python command-line flags
• missing: ctypes and .so extension loading

The stackless binary contains:

• Stackless Python 2.7.1
• and everything above, including greenlet

More goodies in the stacklessco binary

• SSL sockets (_ssl) and hashes (_hashlib) using OpenSSL
• PyCrypto
• alo-aes
• gevent with libevent2
• gevent-MySQL
• Syncless with libevent2
• Concurrence

stacklessco also runs these at full speed:

• Twisted
• Eventlet
• Tornado

How many bytes is the 2.7 binary?

How is the binary constructed?

• upx --brute for C code was considered, but it's not used, because it makes startup slow
• advzip -z -4 for Python code
• concatenate
• svn commit
  • Linux and FreeBSD binaries
  • Mac OS X binaries

Distribution-specific changes

• force config.guess to i386 (32-bit)
• force HAVE_DYNAMIC_LOADING off
• reimplement calculate_path() to read from argv[0] as a .zip file
• add all C extensions to Modules/Setup (with manual filenames)
• remove -g, force -O2 for gcc in Makefiles
• remove unapplicable lib/plat-*
• set OS to Linux in the ELF headers, for FreeBSD
• STACKLESS_FRHACK for the ebx register on the Mac OS X

Distribution-specific changes for modules

• sqlite needs an extra #include "util.h"
• force good event.h for libevent2 in gevent
• run pyrexc manually for Concurrence (setup.py is never run!)
• proxy for non-toplevel C extensions, e.g. gevent/core.py:

  import sys; sys.modules[__name__] = __import__('_gevent_core')

The build system (build.sh)

• fully automated
• fully deterministic and repeatable
  • on Linux: bundled and statically linked gcc, ld, busybox, make, perl
  • all source .tar.gz files in the repository: Python, zlib, bz2, ncurses, readline, sqlite3, libevent2.
• not incremental, rebuilds from scratch
• has some very rough and manual incrementality support
• can be extended to build custom binaries

Porting issues

Mac OS X:

• use Xcode for Mac OS X 10.5 to build GCC 4.4 using MacPorts
• compile using gcc-mp-4.4 -m32 -static-libcc
• no support for static binaries in the operating system
• rename to libreadline-staticpython.a, because -lreadline would link against libreadline.*.dylib
• remove the spwd module, no shadow passwords

FreeBSD:

• no epoll(4) emulation for FreeBSD, libevent2 reverts to poll(2)
• everything else works by magic (no userspace libararies needed)

?