Fix Homebrew Python and Unify Development

Stop Homebrew Version-Jacking Your Python All The Time

It happened again. I do a careless brew upgrade and homebrew blows away1 the stable python install I had 20 virtualenvs pointing to in favor of a new major python version released two days ago that many extensions and modules won’t even support building against for another 30-90 days.

Let’s stop homebrew from upgrading us past where we expect to be2 with pyenv and all the shell hacks required to run it.

pyenv makes available many python versions and forks (pypy, stackless, jython, micropython, etc) and installs them all from source in a user directory for non-conflicting system development. As of this writing, pyenv install --list reports 4383 unique python implementations available for building.

Install pyenv

First clone pyenv (and no, do not install pyenv from homebrew either, it also causes problems4):

 git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Setup pyenv

Remainder of the setup process requires modifying your $PATH and setting some variables, which depends on your system and shell, so pick the right combination for your platforms then open a new shell to continue using pyenv with your shim-adjusted path.

Also note: any previously system-wide installed python helpers (like /usr/local/bin/poetry, etc) placed into your path should also be removed in favor of letting the pyenv shims at the start of your path pick the correct version for you. The pip/homebrew /usr/local/bin packages are pinned to your system or package manager python version and they don’t check your environment python for any usage.

Install your first private python version

Because pyenv builds each python version from source, you need to have all the python build dependencies installed before asking pyenv to isolate a python version for you.

Check out the pyenv build environment page for which packages you likely need across various platforms.

After your system has all the prereq packages, just pyenv install 3.9.1, wait 5 minutes (python builds fast under defaults), and you have a new python version available!

You can customize the build process in many ways to turn on optimizations and LTO (increases build time to 15+ minutes) using export PYTHON_CONFIGURE_OPTS="--enable-optimizations --enable-ipv6 --with-lto or use different libraries than the default configure scripts will find.

Use your private python version

Now with a python version installed, you can set your local directory python version using pyenv local 3.9.1 and pyenv will drop a file named .python-version in your current directory with the contents 3.9.1.

If you successfully updated your $PATH in the previous steps so pyenv shims are before your system python paths, when you run python, you’ll get the version specified by the nearest .python-version file without needing to do any virtualenv trickery.

How pyenv Works

pyenv works using shell magic and local version pinning files:

  • pyenv overrides your path python and python3 to use a version lookup script instead of being a binary or symlink itself.
    • (pyenv also overrides/intercepts all other python utilities like pip and easy_install and virtualenv, etc; in fact every utility gets the same5 shim just re-exec’ing the called script name with the requested .python-version path)
  • pyenv looks to see if the current directory has a .python-version file
    • if not, pyenv checks the next highest directory, all the way up.
    • if so, when you call python (which in your path is now ~/.pyenv/shims/python) the script actually calls $PYENV_ROOT/versions/<version in your file>/bin/python

All Done

and now you’re done! You have a way to install multiple python versions at once, each with correct access to their own helper utilities like pip and poetry without any other path conflicts or manual trickery required.

  1. yes, I know they are still there, at least until brew cleanup destroys them further.↩︎

  2. yes, it is possible to install old brew packages if you manually point to an old git revision, but then another upgrade will just overwrite your choices again in the future. everything ends up being a game of brew roulette—“will this update completely destroy all my development environments? let’s find out!”↩︎

  3. % pyenv install --list
Available versions:
  2.1.3
  2.2.3
  2.3.7
  2.4.0
  2.4.1
  2.4.2
  2.4.3
  2.4.4
  2.4.5
  2.4.6
  2.5.0
  2.5.1
  2.5.2
  2.5.3
  2.5.4
  2.5.5
  2.5.6
  2.6.6
  2.6.7
  2.6.8
  2.6.9
  2.7.0
  2.7-dev
  2.7.1
  2.7.2
  2.7.3
  2.7.4
  2.7.5
  2.7.6
  2.7.7
  2.7.8
  2.7.9
  2.7.10
  2.7.11
  2.7.12
  2.7.13
  2.7.14
  2.7.15
  2.7.16
  2.7.17
  2.7.18
  3.0.1
  3.1.0
  3.1.1
  3.1.2
  3.1.3
  3.1.4
  3.1.5
  3.2.0
  3.2.1
  3.2.2
  3.2.3
  3.2.4
  3.2.5
  3.2.6
  3.3.0
  3.3.1
  3.3.2
  3.3.3
  3.3.4
  3.3.5
  3.3.6
  3.3.7
  3.4.0
  3.4-dev
  3.4.1
  3.4.2
  3.4.3
  3.4.4
  3.4.5
  3.4.6
  3.4.7
  3.4.8
  3.4.9
  3.4.10
  3.5.0
  3.5-dev
  3.5.1
  3.5.2
  3.5.3
  3.5.4
  3.5.5
  3.5.6
  3.5.7
  3.5.8
  3.5.9
  3.5.10
  3.6.0
  3.6-dev
  3.6.1
  3.6.2
  3.6.3
  3.6.4
  3.6.5
  3.6.6
  3.6.7
  3.6.8
  3.6.9
  3.6.10
  3.6.11
  3.6.12
  3.7.0
  3.7-dev
  3.7.1
  3.7.2
  3.7.3
  3.7.4
  3.7.5
  3.7.6
  3.7.7
  3.7.8
  3.7.9
  3.8.0
  3.8-dev
  3.8.1
  3.8.2
  3.8.3
  3.8.4
  3.8.5
  3.8.6
  3.9.0
  3.9-dev
  3.9.1
  3.10-dev
  activepython-2.7.14
  activepython-3.5.4
  activepython-3.6.0
  anaconda-1.4.0
  anaconda-1.5.0
  anaconda-1.5.1
  anaconda-1.6.0
  anaconda-1.6.1
  anaconda-1.7.0
  anaconda-1.8.0
  anaconda-1.9.0
  anaconda-1.9.1
  anaconda-1.9.2
  anaconda-2.0.0
  anaconda-2.0.1
  anaconda-2.1.0
  anaconda-2.2.0
  anaconda-2.3.0
  anaconda-2.4.0
  anaconda-4.0.0
  anaconda2-2.4.0
  anaconda2-2.4.1
  anaconda2-2.5.0
  anaconda2-4.0.0
  anaconda2-4.1.0
  anaconda2-4.1.1
  anaconda2-4.2.0
  anaconda2-4.3.0
  anaconda2-4.3.1
  anaconda2-4.4.0
  anaconda2-5.0.0
  anaconda2-5.0.1
  anaconda2-5.1.0
  anaconda2-5.2.0
  anaconda2-5.3.0
  anaconda2-5.3.1
  anaconda2-2018.12
  anaconda2-2019.03
  anaconda2-2019.07
  anaconda3-2.0.0
  anaconda3-2.0.1
  anaconda3-2.1.0
  anaconda3-2.2.0
  anaconda3-2.3.0
  anaconda3-2.4.0
  anaconda3-2.4.1
  anaconda3-2.5.0
  anaconda3-4.0.0
  anaconda3-4.1.0
  anaconda3-4.1.1
  anaconda3-4.2.0
  anaconda3-4.3.0
  anaconda3-4.3.1
  anaconda3-4.4.0
  anaconda3-5.0.0
  anaconda3-5.0.1
  anaconda3-5.1.0
  anaconda3-5.2.0
  anaconda3-5.3.0
  anaconda3-5.3.1
  anaconda3-2018.12
  anaconda3-2019.03
  anaconda3-2019.07
  anaconda3-2019.10
  anaconda3-2020.02
  anaconda3-2020.07
  graalpython-20.1.0
  graalpython-20.2.0
  ironpython-dev
  ironpython-2.7.4
  ironpython-2.7.5
  ironpython-2.7.6.3
  ironpython-2.7.7
  jython-dev
  jython-2.5.0
  jython-2.5-dev
  jython-2.5.1
  jython-2.5.2
  jython-2.5.3
  jython-2.5.4-rc1
  jython-2.7.0
  jython-2.7.1
  jython-2.7.2
  micropython-dev
  micropython-1.9.3
  micropython-1.9.4
  micropython-1.10
  micropython-1.11
  micropython-1.12
  micropython-1.13
  miniconda-latest
  miniconda-2.2.2
  miniconda-3.0.0
  miniconda-3.0.4
  miniconda-3.0.5
  miniconda-3.3.0
  miniconda-3.4.2
  miniconda-3.7.0
  miniconda-3.8.3
  miniconda-3.9.1
  miniconda-3.10.1
  miniconda-3.16.0
  miniconda-3.18.3
  miniconda2-latest
  miniconda2-3.18.3
  miniconda2-3.19.0
  miniconda2-4.0.5
  miniconda2-4.1.11
  miniconda2-4.3.14
  miniconda2-4.3.21
  miniconda2-4.3.27
  miniconda2-4.3.30
  miniconda2-4.3.31
  miniconda2-4.4.10
  miniconda2-4.5.1
  miniconda2-4.5.4
  miniconda2-4.5.11
  miniconda2-4.5.12
  miniconda2-4.6.14
  miniconda2-4.7.10
  miniconda2-4.7.12
  miniconda3-latest
  miniconda3-2.2.2
  miniconda3-3.0.0
  miniconda3-3.0.4
  miniconda3-3.0.5
  miniconda3-3.3.0
  miniconda3-3.4.2
  miniconda3-3.7.0
  miniconda3-3.8.3
  miniconda3-3.9.1
  miniconda3-3.10.1
  miniconda3-3.16.0
  miniconda3-3.18.3
  miniconda3-3.19.0
  miniconda3-4.0.5
  miniconda3-4.1.11
  miniconda3-4.2.12
  miniconda3-4.3.11
  miniconda3-4.3.14
  miniconda3-4.3.21
  miniconda3-4.3.27
  miniconda3-4.3.30
  miniconda3-4.3.31
  miniconda3-4.4.10
  miniconda3-4.5.1
  miniconda3-4.5.4
  miniconda3-4.5.11
  miniconda3-4.5.12
  miniconda3-4.6.14
  miniconda3-4.7.10
  miniconda3-4.7.12
  pypy-c-jit-latest
  pypy-c-nojit-latest
  pypy-dev
  pypy-stm-2.3
  pypy-stm-2.5.1
  pypy-1.5-src
  pypy-1.5
  pypy-1.6
  pypy-1.7
  pypy-1.8
  pypy-1.9
  pypy-2.0-src
  pypy-2.0
  pypy-2.0.1-src
  pypy-2.0.1
  pypy-2.0.2-src
  pypy-2.0.2
  pypy-2.1-src
  pypy-2.1
  pypy-2.2-src
  pypy-2.2
  pypy-2.2.1-src
  pypy-2.2.1
  pypy-2.3-src
  pypy-2.3
  pypy-2.3.1-src
  pypy-2.3.1
  pypy-2.4.0-src
  pypy-2.4.0
  pypy-2.5.0-src
  pypy-2.5.0
  pypy-2.5.1-src
  pypy-2.5.1
  pypy-2.6.0-src
  pypy-2.6.0
  pypy-2.6.1-src
  pypy-2.6.1
  pypy-4.0.0-src
  pypy-4.0.0
  pypy-4.0.1-src
  pypy-4.0.1
  pypy-5.0.0-src
  pypy-5.0.0
  pypy-5.0.1-src
  pypy-5.0.1
  pypy-5.1-src
  pypy-5.1
  pypy-5.1.1-src
  pypy-5.1.1
  pypy-5.3-src
  pypy-5.3
  pypy-5.3.1-src
  pypy-5.3.1
  pypy-5.4-src
  pypy-5.4
  pypy-5.4.1-src
  pypy-5.4.1
  pypy-5.6.0-src
  pypy-5.6.0
  pypy-5.7.0-src
  pypy-5.7.0
  pypy-5.7.1-src
  pypy-5.7.1
  pypy2-5.3-src
  pypy2-5.3
  pypy2-5.3.1-src
  pypy2-5.3.1
  pypy2-5.4-src
  pypy2-5.4
  pypy2-5.4.1-src
  pypy2-5.4.1
  pypy2-5.6.0-src
  pypy2-5.6.0
  pypy2-5.7.0-src
  pypy2-5.7.0
  pypy2-5.7.1-src
  pypy2-5.7.1
  pypy2.7-5.8.0-src
  pypy2.7-5.8.0
  pypy2.7-5.9.0-src
  pypy2.7-5.9.0
  pypy2.7-5.10.0-src
  pypy2.7-5.10.0
  pypy2.7-6.0.0-src
  pypy2.7-6.0.0
  pypy2.7-7.0.0-src
  pypy2.7-7.0.0
  pypy2.7-7.1.0-src
  pypy2.7-7.1.0
  pypy2.7-7.1.1-src
  pypy2.7-7.1.1
  pypy2.7-7.2.0-src
  pypy2.7-7.2.0
  pypy2.7-7.3.0-src
  pypy2.7-7.3.0
  pypy2.7-7.3.1-src
  pypy2.7-7.3.1
  pypy3-2.3.1-src
  pypy3-2.3.1
  pypy3-2.4.0-src
  pypy3-2.4.0
  pypy3.3-5.2-alpha1-src
  pypy3.3-5.2-alpha1
  pypy3.3-5.5-alpha-src
  pypy3.3-5.5-alpha
  pypy3.5-c-jit-latest
  pypy3.5-5.7-beta-src
  pypy3.5-5.7-beta
  pypy3.5-5.7.1-beta-src
  pypy3.5-5.7.1-beta
  pypy3.5-5.8.0-src
  pypy3.5-5.8.0
  pypy3.5-5.9.0-src
  pypy3.5-5.9.0
  pypy3.5-5.10.0-src
  pypy3.5-5.10.0
  pypy3.5-5.10.1-src
  pypy3.5-5.10.1
  pypy3.5-6.0.0-src
  pypy3.5-6.0.0
  pypy3.5-7.0.0-src
  pypy3.5-7.0.0
  pypy3.6-7.0.0-src
  pypy3.6-7.0.0
  pypy3.6-7.1.0-src
  pypy3.6-7.1.0
  pypy3.6-7.1.1-src
  pypy3.6-7.1.1
  pypy3.6-7.2.0-src
  pypy3.6-7.2.0
  pypy3.6-7.3.0-src
  pypy3.6-7.3.0
  pypy3.6-7.3.1-src
  pypy3.6-7.3.1
  pyston-0.5.1
  pyston-0.6.0
  pyston-0.6.1
  stackless-dev
  stackless-2.7-dev
  stackless-2.7.2
  stackless-2.7.3
  stackless-2.7.4
  stackless-2.7.5
  stackless-2.7.6
  stackless-2.7.7
  stackless-2.7.8
  stackless-2.7.9
  stackless-2.7.10
  stackless-2.7.11
  stackless-2.7.12
  stackless-2.7.14
  stackless-3.2.2
  stackless-3.2.5
  stackless-3.3.5
  stackless-3.3.7
  stackless-3.4-dev
  stackless-3.4.1
  stackless-3.4.2
  stackless-3.4.7
  stackless-3.5.4
  stackless-3.7.5
    ↩︎

  4. what problems? pyenv doesn’t always cut a new release when it adds new python versions to its library, and homebrew only uses released versions, so maybe you do want a python version released in the past month but pyenv hasn’t cut a release in two months… you end up having to clone pyenv manually anyway (also, the source of python versions is only inside the pyenv repo itself, it doesn’t maintain an independent download list).↩︎

  5. % openssl sha512 ~/.pyenv/shims/*
SHA512(/Users/matt/.pyenv/shims/2to3)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/2to3-3.9)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/chardetect)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/doesitcache)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/easy_install)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/easy_install-3.9)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/idle)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/idle3)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/idle3.9)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/keyring)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/pip)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/pip3)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/pip3.9)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/pkginfo)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/poetry)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/pydoc)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/pydoc3)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/pydoc3.9)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/python)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/python-config)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/python3)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/python3-config)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/python3.9)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/python3.9-config)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/python3.9-gdb.py)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
SHA512(/Users/matt/.pyenv/shims/virtualenv)= 22c8ec0e34dc5b480a0550d87be2d3279e7ab3af0e58dcf5d30bb1d2bdced6a36e74220f444a716fbfc48f74c54a4c0fa9c76a339e285601705871e59540f415
    ↩︎