Note: This post has been updated to reflect the modules on Dardel (December 2021).
When we use Python in our work or personal projects, it is often necessary to use a number of packages that are not distributed as standard Python libraries. We therefore need to install those packages based on the specific requirements of every project. In the scenario of working on multiple projects, it is not uncommon that different projects have conflicting requirements in terms of packages. For example, project A may require version 1.0 of a certain package, while project B may require version 2.0 of the same package. A solution to this conflict is to separate the packages for different projects or purposes with the help of a so-called “virtual environment”.
A Python virtual environment is an isolated run-time environment that makes it possible to install and execute Python packages without interfering with the outside world. Without a virtual environment, Python packages are installed either in the system site directory, which can be located via the following command:
$ python -c 'import site; print(site.getsitepackages())'
or in the so-called Python user base, which is usually in the “$HOME/.local” folder. A Python package installed in this way can have only one version, and it is therefore not possible to work with two or more projects that have conflicting requirements regarding the versions of a certain Python package. With the help of a virtual environment, we can have different Python site directories for different projects and have those site directories isolated from each other and from the system site directory.
This blog post will briefly introduce two ways of creating and managing a Python virtual environment: venv or conda.
Virtual environment with venv
The venv module is a standard Python module that has been available since Python 3.3. It provides a lightweight virtual environment and is very straightforward to use.
For example, if we want to use the venv module on Dardel, we can first load the anaconda module, and then create a virtual environment in the home folder as follows. (Note: If you do this, you need to replace “username” in the example with your username and “u” with the first letter of your username.)
$ module load PDC/21.11
$ module load Anaconda3/2021.05
$ cd /cfs/klemming/home/u/username
$ python3 -m venv my-venv-dardel
After the “python3 -m venv ...” command has been executed, a new folder (named “my-venv-dardel”) will have been created. This folder will contain the files that are used by the virtual environment, such as executables, libraries and scripts. (Note: You can use a different name for the environment, and hence the new folder. )
To use the new virtual environment, we need to activate it first as follows:
$ source my-venv-dardel/bin/activate
(my-venv-dardel) $
After the “source ...” command has been run, we will see the environment name surrounded by parentheses, which reminds us that the “my-venv-dardel” environment is currently active.
We can now check that the Python site directory using the following commands. In the output we can see that the “site-packages” folder is inside the “my-venv-dardel” folder:
(my-venv-dardel) $ python3 -c 'import site; print(site.getsitepackages())'
['/cfs/klemming/home/u/username/my-venv-dardel/lib/python3.8/site-packages']
We can also install any necessary packages in the now-active virtual environment. For example we can install yapf (a formatter for Python files):
(my-venv-dardel) $ python3 -m pip install yapf
and then check that yapf is installed under “my-venv-dardel”:
(my-venv-dardel) $ which yapf
/cfs/klemming/home/u/username/my-venv-dardel/bin/yapf
To deactivate the virtual environment, use the “deactivate” command:
(my-venv-dardel) $ deactivate
$
After the “deactivate” command has been run, the “my-venv-dardel” environment will become inactive, as indicated by the disappearance of the environment name surrounded by parentheses.
Virtual environment with conda
With the anaconda module loaded on Dardel, it is also natural to use “conda” to manage virtual environments for Python. conda is an open-source package and environment manager, and is included in both Anaconda and Miniconda.
To use conda, we need to initialize it first. This is often done by modifying the ~/.bashrc script; however, the ~/.bashrc script is automatically loaded upon login, and you may not want to initialize conda every time you log in to Dardel. In practice it is recommended to use a separate script for initializing conda. We can create a ~/.bashrc.conda.dardel file with the contents in the example below, which can then be used to initialize conda for Dardel.
module load PDC/21.11
module load Anaconda3/2021.05
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/pdc/software/21.11/eb/software/Anaconda3/2021.05/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/pdc/software/21.11/eb/software/Anaconda3/2021.05/etc/profile.d/conda.sh" ]; then
. "/pdc/software/21.11/eb/software/Anaconda3/2021.05/etc/profile.d/conda.sh"
else
export PATH="/pdc/software/21.11/eb/software/Anaconda3/2021.05/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
After this we can load the anaconda module and initialize conda with the following command:
$ source ~/.bashrc.conda.dardel
(base) $
After the above command has been executed, we’ll see “base” surrounded by parentheses, indicating that the default virtual environment is active.
The next step is to configure conda with user-defined directories. For example, we can edit the ~/.condarc file (or create the file if it doesn’t exist) so it contains the following lines.
pkgs_dirs:
- /cfs/klemming/home/u/username/conda-dirs/pkgs
envs_dirs:
- /cfs/klemming/home/u/username/conda-dirs/envs
Now we are ready to create our own virtual environment with conda. In practice, we use the “conda create ...” command and specify both the “--prefix” option and the Python version with the following command:
(base) $ conda create --prefix /cfs/klemming/home/u/username/conda-dirs/envs/my-conda-dardel
Note that the “--prefix” option should point to the virtual environment that is under the specified environment directory (env_dirs) in the ~/.condarc file. The output will look like the following:
Collecting package metadata: done
Solving environment: done
## Package Plan ##
environment location: /cfs/klemming/home/u/username/conda-dirs/envs/my-conda-dardel
added / updated specs:
- python=3.8
The following packages will be downloaded:
...
The following NEW packages will be INSTALLED:
...
Proceed ([y]/n)?
After pressing “y” and the enter key, we’ll see the following:
Downloading and Extracting Packages
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
# To activate this environment, use
#
# $ conda activate /cfs/klemming/home/u/username/conda-dirs/envs/my-conda-dardel
#
# To deactivate an active environment, use
#
# $ conda deactivate
With conda, the commands to activate and deactivate a virtual environment are “conda activate ...” and “conda deactivate”, respectively. Since we have already specified envs_dir in ~/.condarc, we can activate the my-conda-dardel environment using this command:
(base) $ conda activate my-conda-dardel
(my-conda-dardel) $
We can check that the Python site directory is now under my-conda-dardel as follows:
(my-conda-dardel) $ python3 -c 'import site; print(site.getsitepackages())'
['/cfs/klemming/home/u/username/conda-dirs/envs/my-conda-dardel/lib/python3.8/site-packages']
We can also install yapf, now using “conda install” as in the following example (and by pressing the “y” and enter keys to proceed):
(my-conda-dardel) $ conda install yapf
Collecting package metadata: done
Solving environment: done
## Package Plan ##
environment location: /cfs/klemming/home/u/username/conda-dirs/envs/my-conda-dardel
added / updated specs:
- yapf
The following packages will be downloaded:
...
The following NEW packages will be INSTALLED:
...
Proceed ([y]/n)? y
Downloading and Extracting Packages
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
We can then check that yapf has been installed under my-conda-dardel using the following command:
(my-conda-dardel) $ which yapf
/cfs/klemming/home/u/username/conda-dirs/envs/my-conda-dardel/bin/yapf
To deactivate the virtual environment, use the “conda deactivate” command:
(my-conda-dardel) $ conda deactivate
(base) $
After the “conda deactivate” command has been run, the “my-conda-dardel” environment will become inactive and the default “base” environment will become active again. The “conda deactivate” command can also be used to deactivate the base environment as follows:
(base) $ conda deactivate
$
Summary
We have briefly introduced the creation and management of a Python virtual environment using venv and conda. The former is lightweight and easy-to-use while the latter is more powerful and versatile. You can find more information about them in the following links: