Stop Homebrew Version-Jacking Your Python All The Time

It happened again. I do a careless brew upgrade and homebrew blows away¹ the stable python install I had 20 virtualenvs pointing to in favor of a new major python version released two days ago that many extensions and modules won’t even support building against for another 30-90 days.

Let’s stop homebrew from upgrading us past where we expect to be² with pyenv and all the shell hacks required to run it.

pyenv makes available many python versions and forks (pypy, stackless, jython, micropython, etc) and installs them all from source in a user directory for non-conflicting system development. As of this writing, pyenv install --list reports 438³ unique python implementations available for building.

Install pyenv

First clone pyenv (and no, do not install pyenv from homebrew either, it also causes problems⁴):

 git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Setup pyenv

Remainder of the setup process requires modifying your $PATH and setting some variables, which depends on your system and shell, so pick the right combination for your platforms then open a new shell to continue using pyenv with your shim-adjusted path.

Also note: any previously system-wide installed python helpers (like /usr/local/bin/poetry, etc) placed into your path should also be removed in favor of letting the pyenv shims at the start of your path pick the correct version for you. The pip/homebrew /usr/local/bin packages are pinned to your system or package manager python version and they don’t check your environment python for any usage.

Install your first private python version

Because pyenv builds each python version from source, you need to have all the python build dependencies installed before asking pyenv to isolate a python version for you.

Check out the pyenv build environment page for which packages you likely need across various platforms.

After your system has all the prereq packages, just pyenv install 3.9.1, wait 5 minutes (python builds fast under defaults), and you have a new python version available!

You can customize the build process in many ways to turn on optimizations and LTO (increases build time to 15+ minutes) using export PYTHON_CONFIGURE_OPTS="--enable-optimizations --enable-ipv6 --with-lto or use different libraries than the default configure scripts will find.

Use your private python version

Now with a python version installed, you can set your local directory python version using pyenv local 3.9.1 and pyenv will drop a file named .python-version in your current directory with the contents 3.9.1.

If you successfully updated your $PATH in the previous steps so pyenv shims are before your system python paths, when you run python, you’ll get the version specified by the nearest .python-version file without needing to do any virtualenv trickery.

How pyenv Works

pyenv works using shell magic and local version pinning files:

• pyenv overrides your path python and python3 to use a version lookup script instead of being a binary or symlink itself.
  • (pyenv also overrides/intercepts all other python utilities like pip and easy_install and virtualenv, etc; in fact every utility gets the same⁵ shim just re-exec’ing the called script name with the requested .python-version path)
• pyenv looks to see if the current directory has a .python-version file
  • if not, pyenv checks the next highest directory, all the way up.
  • if so, when you call python (which in your path is now ~/.pyenv/shims/python) the script actually calls $PYENV_ROOT/versions/<version in your file>/bin/python

All Done

and now you’re done! You have a way to install multiple python versions at once, each with correct access to their own helper utilities like pip and poetry without any other path conflicts or manual trickery required.