Building and uploading packages with conda

For quite some time I’ve been using Anaconda as a portable and stable target for running experiments in Python. While many packages already exist in the conda packaging system, there are occasionally things that are a bit harder to find. And rather than installing the package one-off for each machine I might need this on, it’s actually quite easy to roll our own package.

Below I will create a configuration for the NLopt package, but the basics are pretty generic. First we need to create a directory nlopt/ containing files config.yaml and build.sh. The config.yaml file describes where to find the package source as well as any requirements:

package:
    name: nlopt
    version: 2.4.2

source:
    fn: nlopt-2.4.2.tar.gz
    url: http://ab-initio.mit.edu/nlopt/nlopt-2.4.2.tar.gz
    md5: d0b8f139a4acf29b76dbae69ade8ac54

requirements:
    build:
        - python
        - numpy
    run:
        - python
        - numpy

test:
    imports:
        - nlopt

More information on the layout of this file can be found here. Since in this instance I’m building this primarily for the Python bindings, the main purpose of the test section is to make sure that the package is built and can be imported. Next, the build.sh script describes how to build the package which in this case looks something like:

./configure --prefix="$PREFIX" --enable-shared
make
make install

The main important bit here is the $PREFIX variable which will be set by the build system which we’ll call momentarily. But first we need to actually install the build system with

conda install conda-build anaconda-client

Technically the anaconda-client package isn’t completely necessary, but it will make our life easier in a few minutes. At this point, from the directory that contains nlopt/ we can run

conda build nlopt

We could then just install the package using conda install --use-local nlopt, but we can take this a step further by uploading the package to anaconda.org. For this you do need to create an account; mine is mwhoffman which we’ll see shortly. With account information in hand we can then upload the package by running

anaconda login
anaconda upload /usr/local/anaconda/conda-bld/osx-64/nlopt-2.4.2-py27_0.tar.bz2

The location of the file will depend on your anaconda installation, but this will be displayed as output during the build step. Once this is done, we can install the package on any anaconda installation by running

conda install -c https://conda.anaconda.org/mwhoffman nlopt

The -c option to conda tells it to also search this channel (note the mwhoffman there) for packages. But the last two steps can be simplified even further by adding the the following to our ~/.condarc file:

anaconda_upload: True
channels:
  - mwhoffman
  - defaults

The first line will automatically upload successful builds, allowing us to skip the anaconda upload step. The other lines will use the mwhoffman channel by default; however under this configuration conda will only use the listed channels so we have to make sure to to add the defaults channel as well. After doing this conda install nlopt should work directly.

Note, however, that this assumes that any additional machines share the same system (osx-64 for the above package). If this is not the case, though, the same build system can be used and conda build nlopt can be run on the new system which will enable and upload the package for all further uses of that system.