Passions Play

Screenshot showing the process of using Poetry to install Python dev dependencies.

Python Dev Environment under Ubuntu in 2021

Let’s work through how to get a stable and flexible Python development environment configured for Ubuntu 21.04.

Background

Throughout my time in tech, I’ve mostly been a generalist and have employed various languages to just get things done. I’ve never focused solely on learning Python, but I do know enough to wrangle data and create some effective visualizations.

Occasionally I like to completely wipe my personal Linux machine and install things from scratch. There’s something wonderful about nuking the cruft and getting back to the bare essentials for getting things done. So here I am; a mostly blank installation of Ubuntu 21.04, ready to get a Python environment installed and set up. Let’s document this process so that future me won’t have to think about how to get up and running with Python dev!

Since I’m not solely focused on Python projects, it makes sense that I should probably search for how “the pros” configure their environment to be productive. There are a lot of guides out there that say they have the answer, and in sometimes that true. Sometimes you go down a fragile rabbit hole. Many of the tutorials and blog posts covered older versions of Python, or in some way didn’t feel right for how I like to configure my system.

In the end, the best post I found was from Jacob Kaplan-Moss (co-creator of Django). It’s specific, direct, and I get the feeling that like me, a crufty machine won’t be tolerated. Even though the post is from 2019, the general ideas and package recommendations are still good:

  1. pyenv
    • Helps to manage multiple versions of Python. Think of nvm, but for Python.
  2. pipx
    • Manage your user’s “global” tooling. Something along the lines of installing a Javascript tool for your “global” user. (think npm install <package> --global )
  3. poetry
    • Dependency and virtual environment management for specific projects. Think npm or composer, where you declare a project’s dependencies and let the tool build the environment. Additionally, this configuration is “frozen” with a lock file.

So, what are the specifics for getting these things working on a clean install of Ubuntu?

Installing Ubuntu Dependencies

In Jacob’s post above, the installation is for a MacOS system. It took a bit of trial and error, but I was able to zero in on the specific mix of packages that need to be installed. There might be other packages that creep in as I start installing and using various Python tools, but for now, this should get me what I need:

sudo apt-get update;
sudo apt-get install --no-install-recommends \
     make build-essential libssl-dev zlib1g-dev \
     libbz2-dev libreadline-dev libsqlite3-dev \
     wget curl llvm libncurses5-dev xz-utils \
     tk-dev libxml2-dev libxmlsec1-dev \
     libffi-dev liblzma-dev

pyenv

Pyenv is pretty neat and does it’s fancy management of Python versions by shimming your $PATH.

Installing is easy and is basically:

  1. Clone the repo to your home directory.
  2. Update your shell config to reference that repo.

The above process is outlined in more detail within the Basic Github checkout section of the readme. Here’s what that looks like:

git clone https://github.com/pyenv/pyenv.git ~/.pyenv

And this shell configuration added to my ~/.profile file:

if [ -d "$HOME/.pyenv" ]; then
    export PYENV_ROOT="$HOME/.pyenv"
    export PATH="$PYENV_ROOT/bin:$PATH"
    eval "$(pyenv init --path)"
fi

Once Pyenv is configured, we can list and install individual versions of Python from the terminal.

★  src/python-sandbox % pyenv install --list
pyenv install --list
Available versions:
2.1.3
2.2.3
...
★  src/python-sandbox % pyenv install 3.9.1
pyenv install 3.9.1
★  src/python-sandbox % pyenv global 3.9.1
pyenv global 3.9.1
★  src/python-sandbox % which python
which python
/home/benjamin/.pyenv/shims/python
★  src/python-sandbox % python --version
python --version
Python 3.9.1

pipx

As someone who doesn’t live and breath Python, I mostly turn to it for quick and simple scripts as well as a few essential cli apps. Because most systems make use of some sort of system-installed version of Python, I like the idea of having my user-specific tooling constrained to my own user.

Pipx makes having a separate, user-focused python environment easy, and the installation is quick using pip from our Pyenv installation:

★  src/python-sandbox % python -m pip install --user pipx
python -m pip install --user pipx
Collecting pipx
  Downloading pipx-0.15.6.0-py3-none-any.whl (43 kB)
     |████████████████████████████████| 43 kB 1.6 MB/s 
Collecting packaging>=20.0
  Downloading packaging-20.8-py2.py3-none-any.whl (39 kB)
  ...
★  src/python-sandbox % export PATH="$HOME/.local/bin:${PATH}"
export PATH="$HOME/.local/bin:${PATH}"
★  src/python-sandbox % python -m pipx ensurepath
python -m pipx ensurepath
/home/benjamin/.local/bin is already in PATH.
/home/benjamin/.local/bin is already in PATH.

We now have an isolated environment for our user’s Python cli apps, but what now?

One amazing tool that I keep returning to is Visidata, which is like a spreadsheet tool on CLI steroids. It allows you to explore, interact, and clean data from the command line.

In addition to working with csv files, you can extend Visidata by including additional libraries so that it can handle more file formats. For example, installing Pandas will allow more loaders within visidata – basically, any file format that Pandas knows how to open can be opened in Visidata.

So we want Visidata along with the Pandas boost, but how do we do that with Pipx?

By injecting dependencies!

★  src/python-sandbox % pipx install visidata
pipx install visidata
  installed package visidata 2.1.1, Python 3.9.1
  These apps are now globally available
    - vd
    - visidata
done! ✨ 🌟 ✨
★  src/python-sandbox % pipx inject visidata pandas
pipx inject visidata pandas
  injected package pandas into venv visidata
done! ✨ 🌟 ✨

Poetry

The previous tools have let us sandbox our day-to-day Python tools away from a system configuration of Python, but the missing piece is managing dependencies for an actual Python project.

Poetry handles the actual dependency and virtual environment management for specific projects. Think of something like yarn or composer, where you can declare your dependencies as well as freeze the configuration by using a “lock” file.

Using Pipx again, installation is easy, and we can quickly initialize a new project and add our dependencies:

★  src/python-sandbox % pipx install poetry
pipx install poetry
  installed package poetry 1.1.4, Python 3.9.1
  These apps are now globally available
    - poetry
done! ✨ 🌟 ✨
★  src/python-sandbox % poetry init
poetry init

This command will guide you through creating your pyproject.toml config.

Package name [python-sandbox]:  

...

★  src/python-sandbox % poetry add pandas numpy bokeh networkx
poetry add pandas numpy bokeh networkx
Creating virtualenv python-sandbox-Z3rSXY9z-py3.9 in /home/benjamin/.cache/pypoetry/virtualenvs
Using version ^1.2.0 for pandas
Using version ^1.19.5 for numpy
Using version ^2.2.3 for bokeh
Using version ^2.5 for networkx

Updating dependencies
...

Nice! Pretty easy!

Wrap-up

I’ve been using this setup for a few weeks now and it feels mostly stable and strikes a good balance of:

  1. Being flexible for when I need to switch versions of software.
  2. Isolated so that I’m not accidentally breaking other projects.
  3. Good-enough defaults for when I just need a quick snippet of Python to be run.

Hope this helps get you up and running quickly and that you can get on with solving interesting problems!

Posted

in

by

Benjamin

Tags:

, ,