Skip to content

Working with Python Virtual Environments

When we use Python in our work or personal projects, it is often necessary to use a number of packages that are not distributed as standard Python libraries. We therefore need to install those packages based on the specific requirements of every project. In the scenario of working on multiple projects, it is not uncommon that different projects have conflicting requirements in terms of packages. For example, project A may require version 1.0 of a certain package, while project B may require version 2.0 of the same package. A solution to this conflict is to separate the packages for different projects or purposes with the help of a so-called “virtual environment”.

A Python virtual environment is an isolated run-time environment that makes it possible to install and execute Python packages without interfering with the outside world. Without a virtual environment, Python packages are installed either in the system site directory, which can be located via the following command:

$ python -c 'import site; print(site.getsitepackages())'

or in the so-called Python user base, which is usually in the “$HOME/.local” folder. A Python package installed in this way can have only one version, and it is therefore not possible to work with two or more projects that have conflicting requirements regarding the versions of a certain Python package. With the help of a virtual environment, we can have different Python site directories for different projects and have those site directories isolated from each other and from the system site directory.

This blog post will briefly introduce two ways of creating and managing a Python virtual environment: venv or conda.

Virtual environment with venv

The venv module is a standard Python module that has been available since Python 3.3. It provides a lightweight virtual environment and is very straightforward to use.

For example, if we want to use the venv module on Beskow, we can first load the anaconda module, and then create a virtual environment in the nobackup folder  as follows. (Note: If you do this, you need to replace “username” in the example with your username and “u” with  the first letter of your username.)

$ module load anaconda/2019.03/py37
$ cd /cfs/klemming/nobackup/u/username
$ python3 -m venv my-venv-beskow

After the “python3 -m venv ...” command has been executed, a new folder (named “my-venv-beskow”) will have been created. This folder will contain the files that are used by the virtual environment, such as executables, libraries and scripts. (Note: You can use a different name for the environment, and hence the new folder. However, since the nobackup folder is shared between Beskow and Tegner, it is helpful to include “beskow” in the name of the virtual environment, so you remember that this virtual environment is to be used on Beskow only. )

To use the new virtual environment, we need to activate it first as follows:

$ source my-venv-beskow/bin/activate
(my-venv-beskow) $

After the “source ...” command has been run, we will see the environment name surrounded by parentheses, which reminds us that the “my-venv-beskow” environment is currently active.

We can now check that the Python site directory using the following commands. In the output we can see that the site-packages” folder is inside the “my-venv-beskow” folder:

(my-venv-beskow) $ python3 -c 'import site; print(site.getsitepackages())'
['/cfs/klemming/nobackup/u/username/my-venv-beskow/lib64/python3.6/site-packages', '/cfs/klemming/nobackup/u/username/my-venv-beskow/lib/python3.6/site-packages']

We can also install any necessary packages in the now-active virtual environment. For example we can install yapf (a formatter for Python files):

(my-venv-beskow) $ python3 -m pip install yapf

and then check that yapf is installed under my-venv-beskow”:

(my-venv-beskow) $ which yapf
/cfs/klemming/nobackup/u/username/my-venv-beskow/bin/yapf

To deactivate the virtual environment, use the deactivate” command:

(my-venv-beskow) $ deactivate
$

After the deactivate” command has been run, the my-venv-beskow” environment will become inactive, as indicated by the disappearance of the environment name surrounded by parentheses.

Virtual environment with conda

With the anaconda module loaded on Beskow, it is also natural to use conda to manage virtual environments for Python. conda is an open-source package and environment manager, and is included in both Anaconda and Miniconda.

To use conda, we need to initialize it first. This is often done by modifying the ~/.bashrc script; however, since the ~/.bashrc script is shared between Beskow and Tegner, it is recommended to use a separate script for initializing conda. In practice, we can create a ~/Public/.bashrc.conda.beskow file with the contents in the example below. This script file can then be used to initialize conda for Beskow.

module load anaconda/2019.03/py37

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/pdc/vol/anaconda/2019.03/py37/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/pdc/vol/anaconda/2019.03/py37/etc/profile.d/conda.sh" ]; then
        . "/pdc/vol/anaconda/2019.03/py37/etc/profile.d/conda.sh"
    else
        export PATH="/pdc/vol/anaconda/2019.03/py37/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Since Linux users are used to having such scripts in home directory, we can create a symbolic link in our home directory that points to the initialization script for conda:

$ ln -s ~/Public/.bashrc.conda.beskow ~/.bashrc.conda.beskow

After this we can load the anaconda module and initialize conda with the following command:

$ source ~/.bashrc.conda.beskow
(base) $

After the above command has been executed, we’ll see base surrounded by parentheses, indicating that the default virtual environment is active.

The next step is to configure conda with user-defined directories. For example, we can edit the ~/.condarc file (or create the file if it doesn’t exist) so it contains the following lines.

pkgs_dirs:
    - /cfs/klemming/nobackup/u/username/conda-dirs/pkgs
envs_dirs:
    - /cfs/klemming/nobackup/u/username/conda-dirs/envs

Now we are ready to create our own virtual environment with conda. In practice, we use the conda create ...” command and specify both the --prefix option and the Python version with the following command:

(base) $ conda create --prefix /cfs/klemming/nobackup/u/username/conda-dirs/envs/my-conda-beskow python=3.7

Note that the --prefix option should point to the virtual environment that is under the specified environment directory (env_dirs) in the ~/.condarc file. The output will look like the following:

Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /cfs/klemming/nobackup/u/username/conda-dirs/envs/my-conda-beskow

  added / updated specs:
    - python=3.7

The following packages will be downloaded:
    ...

The following NEW packages will be INSTALLED:
  ...

Proceed ([y]/n)?

After pressing “y” and the enter key, we’ll see the following:

Downloading and Extracting Packages
...

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

# To activate this environment, use
#
# $ conda activate /cfs/klemming/nobackup/u/username/conda-dirs/envs/my-conda-beskow
#
# To deactivate an active environment, use
#
# $ conda deactivate

With conda, the commands to activate and deactivate a virtual environment are conda activate ... and conda deactivate, respectively. Since we have already specified envs_dir in ~/.condarc, we can activate the my-conda-beskow environment using this command:

(base) $ conda activate my-conda-beskow
(my-conda-beskow) $

We can check that the Python site directory is now under my-conda-beskow as follows:

(my-conda-beskow) $ python3 -c 'import site; print(site.getsitepackages())'
['/cfs/klemming/nobackup/u/username/conda-dirs/envs/my-conda-beskow/lib/python3.7/site-packages']

We can also install yapf, now using “conda install” as in the following example (and by pressing the “y” and enter keys to proceed):

(my-conda-beskow) $ conda install yapf

Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /cfs/klemming/nobackup/u/username/conda-dirs/envs/my-conda-beskow

  added / updated specs:
    - yapf

The following packages will be downloaded:
    ...

The following NEW packages will be INSTALLED:
  ...

Proceed ([y]/n)? y

Downloading and Extracting Packages
...

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

We can then check that yapf has been installed under my-conda-beskow using the following command:

(my-conda-beskow) $ which yapf
/cfs/klemming/nobackup/u/username/conda-dirs/envs/my-conda-beskow/bin/yapf

To deactivate the virtual environment, use the conda deactivatecommand:

(my-conda-beskow) $ conda deactivate
(base) $

After the conda deactivate” command has been run, the my-conda-beskowenvironment will become inactive and the default baseenvironment will become active again. The conda deactivate” command can also be used to deactivate the base environment as follows:

(base) $ conda deactivate
$

Summary

We have briefly introduced the creation and management of a Python virtual environment using venv and conda. The former is lightweight and easy-to-use while the latter is more powerful and versatile. You can find more information about them in the following links: