pip and friends: Packaging

(largely borrowed from Zamboni)

Your app will depend on lots of tasty open source Python libraries. The list of all your dependencies should exist in several places:

  1. requirements/prod.txt
  2. As a submodule of vendor or vendor-local

Ultimately your app code will run against the libraries under vendor/vendor-local via mod_wsgi.

Why requirements? For development, you can use virtualenvs and pip to have a tidy self-contained environment. If run from the Django runserver command, you don’t even need a web server.

The vendor library

The /vendor library is supposed to contain all packages and repositories. It enables the project to be deployed as one package onto many machines, without relying on PyPI-based installations on each target machine.

By default, the vendor lib is checked out as a git submodule under vendor/. If you do need to check it out separately, do:

git clone --recursive git://github.com/mozilla/playdoh-lib.git ./vendor

Once the playdoh-lib repo has been downloaded to /vendor, you only need to install the compiled packages (as defined in requirements/compiled.txt). These can come from your system package manager or from:

pip install -r requirements/compiled.txt

The vendor-local library

The /vendor-local directory is laid out exactly like vendor and works the same way. All of your custom pure Python dependencies should go here so that you can freely merge with playdoh’s vendor directory if necessary. See the section on adding new packages below.

Global vs. local library

Playdoh provides its default library in the vendor/ directory. You may fork and change it, but that will make it hard to pull updates from the upstream library later.

If you want to make only a few local additions or override some of the libs in vendor/, make those changes to the directory vendor-local/ instead, which (in manage.py) is given precedence over playdoh’s vendor dir.

compiled.txt vs prod.txt

If a Python library requires compilation, it should be recorded in compiled.txt. These aren’t as portable and cannot be shipped in the vendor library. For local development, it’s nice to pip install these into a virtualenv. A common practise is to use virtualenv only for compiled libraries and vendor for the rest of your dependencies.

Adding new packages

If we wanted to add a new dependency called cheeseballs to playdoh, you would add it to requirements/prod.txt. If your library isn’t used in production, then put it in requirements/dev.txt. This makes it available to users installing into virtualenvs.

We also need to add the new package to the vendor-local lib, since that is what runs in production...

First, we need to add the source. There are two ways, depending on how this project is hosted:

Non-git based repos (hg, CVS, tarball)

For such repos or for packages coming from PyPI, do:

pip install -I --install-option="--home=`pwd`/vendor-local" cheeseballs
cd vendor-local
git add lib/python/cheeseballs
git commit

Optionally, if the package installs executables add them too. For example:

cd vendor-local
git add bin/cheeseballer
git commit

For hg repos that are not on PyPI, they can be installed with pip too but omit the --home option and use the --src instead. For example:

pip install -I --src='vendor-local/src' \
-e hg+http://bitbucket.org/jespern/django-piston@default#egg=django-piston
cd vendor-local
git add src/django-piston
git commit


Installed source packages need to be appended to vendor-local/vendor.pth. See note below. For example:

echo src/django-piston >> vendor-local/vendor.pth

git-based repositories

For a git-based package, add it as a git submodule:

cd vendor-local
git submodule add git://github.com/mozilla/cheeseballs.git src/cheeseballs
git commit vendor.pth .gitmodules src/cheeseballs

Further, you then need to update vendor-local/vendor.pth. Python uses .pth files to dynamically add directories to sys.path (docs).

The file format is simple. Consult vendor/vendor.pth for reference.

Some packages (like html5lib and selenium) are troublesome, because their source lives inside an extra subdirectory src/ inside their checkout. So they need to be sourced with src/html5lib/src, for example. Hopefully you won’t hit any snags like that.

Done. Try ./manage.py shell and then import cheeseballs to make sure it worked.

Testing Your Vendor Change

It’s critical that you test your app running under mod_wsgi. Although you may use runserver day to day, go ahead and run some code through WSGI to prove vendor is setup properly. (throw an import into your view, etc)

Advanced Topics

TODO [automate these instructions](<https://github.com/mozilla/playdoh/issues/30)

Initial creation of the vendor library

The vendor repo was seeded with

pip install -I --install-option="--home=`pwd`/vendor" --src='vendor/src' -r requirements/dev.txt

# ..delete some junk from vendor/lib/python...

# Create the .pth file so Python can find our src libs.
find src -type d -depth 1 >> vendor.pth

# Add all the submodules.
for f in src/*; do
    pushd $f >/dev/null && REPO=$(git config remote.origin.url) && popd > /dev/null && git submodule add $REPO $f
git add .

Adding lots of git submodules

As noted in Adding new packages, git-based packages are git submodules inside the vendor library. To set up the first batch of submodules, something like the following happened:

for f in src/*
    pushd $f && REPO=$(git config remote.origin.url) && popd && git submodule add $REPO $f

For reference: pip

The classical method of installing is using pip. We have our packages separated into three files:

All packages that require (or go faster with) compilation. These can’t be distributed cross-platform, so they need to be installed through your system’s package manager or pip.
The minimal set of packages you need to run zamboni in production. You also need to get requirements/compiled.txt.
All the packages needed for running tests and development servers. This automatically includes requirements/prod.txt.

With pip, you can get a development environment with:

pip install -r requirements/dev.txt -r requirements/compiled.txt