Bootstrapping pipenv on Debian/Ubuntu

03 Apr 2018

For a long time, I have been happily using pyenv as my support infrastructure when setting up a Python environment. But now apparently the powers that be in the Python world have given their blessing to make pipenv the official tool for such things.

The problem I have found with pipenv is that it wants to run in a non-virtualenv environment, so that it can manage the virtualenvs itself. But usually I really try to avoid using the system Python for my own projects, as the system usually has assumptions and settings that may conflict with my intentions or needs. (This is why pyenv is so handy.)

I’ve found a technique that does the minimal steps needed to get pipenv working, using the system Python installation only for the initial bootstrap.

NOTE: These instructions assume that you are working in a Debian-like environment (e.g. Debian itself, Ubuntu, Linux Mint)

Step 1: install prerequisite packages for pyenv

This installs the system packages required for pyenv to build Python from source code, and include all the extra modules such as SSL, SQLite, curses, etc.

$ sudo apt-get install \
    git build-essential libssl-dev zlib1g-dev \
    libbz2-dev libreadline-dev libsqlite3-dev \
    wget curl llvm libncurses5-dev

Step 2: install pyenv itself, and associated plugins

This uses the very handy pyenv-installer script to set everything up nicely from the github repos.

$ curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash

Step 3: setup pyenv in our .bash_profile script

The exact form of this step may be different for you, as you may have other ways of initializing your environment, but the commands have to be run someplace in order for pyenv to run properly.

This method tacks the pyenv initialization commands onto the end of your .bash_profile script.

$ cat >>~/.bash_profile <<EOD
export PATH="\$HOME/.pyenv/bin:\$PATH"
eval "\$(pyenv init -)"
eval "\$(pyenv virtualenv-init -)"
EOD

Be sure to either re-run your .bash_profile (or whatever init script you added the commands to) or just log out and back in, to get the environment set up for the next steps. Something like source $HOME/.bash_profile works well for me.

Step 4: install system packages for pipenv support

$ sudo apt install python3-pip python3-setuptools

Step 5: install pipenv itself

This step is the crucial one. We need to install pipenv as a local (“user”) install, not in the system area. So we need to tell pip this specifically:

$ python3 -m pip install --user pipenv

Step 6: install a version of python for pipenv to use

Use the version of your choice here; you can see the (very long) list with pyenv install --list. You have to be very specific here.

$ pyenv install 3.6.5 # latest version as of this writing

Step 7: add the .local/bin directory to your shell path

We need to add a specific directory to the shell’s path in order to allow us to run pipenv without using the full path every time, which would be very annoying.

The command here looks very similar to the one for pyenv in step 3, and as in that step, you may need to adapt it for your specific environment or tastes.

Also, as in step 3, be sure to re-run your shell init scripts with source ~/.bash_profile or similar commands.

$ cat >>~/.bash_profile <<EOD
export PATH="\$HOME/.local/bin:\$PATH"
EOD

Step 8: create an environment for your project

You can be a little less specific in the version of python you ask for in this step. pipenv will grab the latest patch version of e.g. Python 3.6 if you ask for just 3.6.

$ mkdir myproject; cd myproject
$ pipenv --python 3.6 # match the version installed in step 6

Now you should be able to follow the rest of the pipenv instructions on creating a Pipfile, etc. (Actually pipenv will have created one for you at this point, so you can just fill out the rest of what you need inside it.)

Have fun!

Update 2018-05-21

At this point my personal technique is to install pipenv directly into the virtual environment I’m working with, having set it up with pyenv beforehand:

pyenv virtualenv 3.6.5 foo
pyenv local foo
pip install pipenv

I prefer the way that pyenv creates its virtualenvs (more specifically, it lets you control the names of the virtualenvs instead of appending some random string and throwing it in your .local directory.)

pipenv does complain for some reason about the --system flag being used although I didn’t actually use it. I found a bug report which mentions that this will be fixed soon, so perhaps there’s no reason to worry much about it. In any case, pipenv detects that it’s already running in a virtualenv and works fine with it.

Now if I could only get PyCharm to recognize and use the .python-version file that pyenv creates, I’d be in for much smoother sailing…

Update 2018-06-11

It seems that the bug which thinks the --system flag has been used, when it hasn’t, has mutated into a more serious version that causes pipenv to abort in a couple different circumstances. Fortunately, until this is properly fixed (as I believe it will be shortly), there is a monkeypatch which will work for now.

To patch this, you need to find where pipenv is actually installed. In my case (with pyenv) I can use $PYENV_VIRTUAL_ENV/lib/python*/site-packages/pipenv. This bug seems to be specific to pyenv (from my limited experience) so that may be all you need.

Within the site-packages/pipenv directory, you need to edit core.py.

There are two lines that need to be edited, in a similar fashion. In my version of pipenv these are lines 1322 and 1349. They are both if statements; the first one is an elif and the other one is a standard if.

The condition they are testing needs to be changed from system or allow_global to (system or allow_global) and not PIPENV_VIRTUALENV.

This should fix the problem, as far as I can see. May your luck be as good or better than mine. :)

Update 2018-06-24

The problem seems to be fixed in pyenv’s github repo, but hasn’t been released yet. Currently I am using the following command to install pipenv (this checks out a specific git commit from the pyenv repo, so it will quickly be out of date):

pip install git+https://github.com/pypa/pipenv.git@1afee9a118982a46d570e8ce7ce00dfd37f8da85

Update 2018-06-28

There is a new release of pipenv to the pypi package index, which seems to have solved the aforementioned errors. Installing the specific git commit is therefore no longer necessary.