pip and friends: Packaging¶
(largely borrowed from Zamboni)
Your app will depend on lots of tasty open source Python libraries. The list of all your dependencies should exist in several places:
- As a submodule of vendor or vendor-local
Ultimately your app code will run against the libraries under vendor/vendor-local via mod_wsgi.
Why requirements? For development, you can use virtualenvs and pip to have a tidy self-contained environment. If run from the Django runserver command, you don’t even need a web server.
The vendor library¶
The /vendor library is supposed to contain all packages and repositories. It enables the project to be deployed as one package onto many machines, without relying on PyPI-based installations on each target machine.
By default, the vendor lib is checked out as a git submodule under vendor/. If you do need to check it out separately, do:
git clone --recursive git://github.com/mozilla/playdoh-lib.git ./vendor
Once the playdoh-lib repo has been downloaded to /vendor, you only need to install the compiled packages (as defined in requirements/compiled.txt). These can come from your system package manager or from:
pip install -r requirements/compiled.txt
The vendor-local library¶
The /vendor-local directory is laid out exactly like vendor and works the same way. All of your custom pure Python dependencies should go here so that you can freely merge with playdoh’s vendor directory if necessary. See the section on adding new packages below.
Global vs. local library¶
Playdoh provides its default library in the vendor/ directory. You may fork and change it, but that will make it hard to pull updates from the upstream library later.
If you want to make only a few local additions or override some of the libs in vendor/, make those changes to the directory vendor-local/ instead, which (in manage.py) is given precedence over playdoh’s vendor dir.
compiled.txt vs prod.txt¶
If a Python library requires compilation, it should be recorded in compiled.txt. These aren’t as portable and cannot be shipped in the vendor library. For local development, it’s nice to pip install these into a virtualenv. A common practise is to use virtualenv only for compiled libraries and vendor for the rest of your dependencies.
Adding new packages¶
If we wanted to add a new dependency called cheeseballs to playdoh, you would add it to requirements/prod.txt. If your library isn’t used in production, then put it in requirements/dev.txt. This makes it available to users installing into virtualenvs.
We also need to add the new package to the vendor-local lib, since that is what runs in production...
First, we need to add the source. There are two ways, depending on how this project is hosted:
Non-git based repos (hg, CVS, tarball)¶
For such repos or for packages coming from PyPI, do:
pip install -I --install-option="--home=`pwd`/vendor-local" cheeseballs cd vendor-local git add lib/python/cheeseballs git commit
Optionally, if the package installs executables add them too. For example:
cd vendor-local git add bin/cheeseballer git commit
For hg repos that are not on PyPI, they can be installed with pip too but omit the --home option and use the --src instead. For example:
pip install -I --src='vendor-local/src' \ -e hg+http://bitbucket.org/jespern/django-piston@default#egg=django-piston cd vendor-local git add src/django-piston git commit
Installed source packages need to be appended to vendor-local/vendor.pth. See note below. For example:
echo src/django-piston >> vendor-local/vendor.pth
For a git-based package, add it as a git submodule:
cd vendor-local git submodule add git://github.com/mozilla/cheeseballs.git src/cheeseballs git commit vendor.pth .gitmodules src/cheeseballs
Further, you then need to update vendor-local/vendor.pth. Python uses .pth files to dynamically add directories to sys.path (docs).
The file format is simple. Consult vendor/vendor.pth for reference.
Some packages (like html5lib and selenium) are troublesome, because their source lives inside an extra subdirectory src/ inside their checkout. So they need to be sourced with src/html5lib/src, for example. Hopefully you won’t hit any snags like that.
Done. Try ./manage.py shell and then import cheeseballs to make sure it worked.
Testing Your Vendor Change¶
It’s critical that you test your app running under mod_wsgi. Although you may use runserver day to day, go ahead and run some code through WSGI to prove vendor is setup properly. (throw an import into your view, etc)
TODO [automate these instructions](<https://github.com/mozilla/playdoh/issues/30)
Initial creation of the vendor library¶
The vendor repo was seeded with
pip install -I --install-option="--home=`pwd`/vendor" --src='vendor/src' -r requirements/dev.txt # ..delete some junk from vendor/lib/python... # Create the .pth file so Python can find our src libs. find src -type d -depth 1 >> vendor.pth # Add all the submodules. for f in src/*; do pushd $f >/dev/null && REPO=$(git config remote.origin.url) && popd > /dev/null && git submodule add $REPO $f done git add .
Adding lots of git submodules¶
As noted in Adding new packages, git-based packages are git submodules inside the vendor library. To set up the first batch of submodules, something like the following happened:
for f in src/* pushd $f && REPO=$(git config remote.origin.url) && popd && git submodule add $REPO $f
For reference: pip¶
The classical method of installing is using pip. We have our packages separated into three files:
- All packages that require (or go faster with) compilation. These can’t be distributed cross-platform, so they need to be installed through your system’s package manager or pip.
- The minimal set of packages you need to run zamboni in production. You also need to get requirements/compiled.txt.
- All the packages needed for running tests and development servers. This automatically includes requirements/prod.txt.
With pip, you can get a development environment with:
pip install -r requirements/dev.txt -r requirements/compiled.txt