{"id":553,"date":"2020-11-10T10:00:27","date_gmt":"2020-11-10T09:00:27","guid":{"rendered":"https:\/\/www.kth.se\/blogs\/pdc\/?p=553"},"modified":"2025-05-06T11:52:42","modified_gmt":"2025-05-06T09:52:42","slug":"working-with-python-virtual-environments","status":"publish","type":"post","link":"https:\/\/www.kth.se\/blogs\/pdc\/2020\/11\/working-with-python-virtual-environments\/","title":{"rendered":"Working with Python Virtual Environments"},"content":{"rendered":"
\n

Note: This post has been updated to reflect the modules on Dardel (December 2021).<\/em><\/p>\n

\n

When we use Python in our work or personal projects, it is often necessary to use a number of packages that are not distributed as standard Python libraries. We therefore need to install those packages based on the specific requirements of every project. In the scenario of working on multiple projects, it is not uncommon that different projects have conflicting requirements in terms of packages. For example, project A may require version 1.0 of a certain package, while project B may require version 2.0 of the same package. A solution to this conflict is to separate the packages for different projects or purposes with the help of a so-called \u201cvirtual environment\u201d<\/a>.<\/p>\n

A Python virtual environment is an isolated run-time environment that makes it possible to install and execute Python packages without interfering with the outside world. Without a virtual environment, Python packages are installed either in the system site directory, which can be located via the following command:<\/p>\n

\n
\n
$ python -c 'import site; print(site.getsitepackages())'\r\n<\/pre>\n<\/div>\n<\/div>\n
or in the so-called Python user base, which is usually in the \u201c$HOME\/.local<\/code>\u201d folder. A Python package installed in this way can have only one version, and it is therefore not possible to work with two or more projects that have conflicting requirements regarding the versions of a certain Python package. With the help of a virtual environment, we can have different Python site directories for different projects and have those site directories isolated from each other and from the system site directory.<\/p>\n
This blog post will briefly introduce two ways of creating and managing a Python virtual environment: venv<\/code> or conda<\/code>.<\/p>\n
<\/p>\n
Virtual environment with venv<\/h2>\n
The venv<\/code><\/a> module is a standard Python module that has been available since Python 3.3. It provides a lightweight virtual environment and is very straightforward to use.<\/p>\n
For example, if we want to use the venv<\/code> module on Dardel<\/a>, we can first load the anaconda<\/code> module, and then create a virtual environment in the home<\/code> folder\u00a0 as follows. (Note: If you do this, you need to replace \u201cusername<\/code>\u201d in the example with your username and \u201cu<\/code>\u201d with\u00a0 the first letter of your username.)<\/p>\n
\n
\n
$ module load PDC\/21.11\r\n$ module load Anaconda3\/2021.05\r\n$ cd \/cfs\/klemming\/home\/u\/username\r\n$ python3 -m venv my-venv-dardel<\/pre>\n<\/div>\n<\/div>\n
After the \u201cpython3 -m venv ...<\/code>\u201d command has been executed, a new folder (named \u201cmy-venv-dardel<\/code>\u201d) will have been created. This folder will contain the files that are used by the virtual environment, such as executables, libraries and scripts. (Note: You can use a different name for the environment, and hence the new folder.\u00a0)<\/p>\n
To use the new virtual environment, we need to activate it first as follows:<\/p>\n
\n
\n
$ source my-venv-dardel\/bin\/activate\r\n(my-venv-dardel) $<\/pre>\n<\/div>\n<\/div>\n
After the \u201csource ...<\/code>\u201d command has been run, we will see <\/span>the environment name surrounded by parentheses, which reminds us that the \u201cmy-venv-dardel<\/code>\u201d environment is currently active.<\/span><\/p>\n
We can now check that the Python site directory using the following commands. In the output we can see that the \u201csite-packages<\/code>\u201d folder is inside the \u201cmy-venv-dardel<\/code>\u201d folder:<\/span><\/span><\/p>\n
\n
\n
(my-venv-dardel) $ python3 -c 'import site; print(site.getsitepackages())'\r\n['\/cfs\/klemming\/home\/u\/username\/my-venv-dardel\/lib\/python3.8\/site-packages']<\/pre>\n<\/div>\n<\/div>\n
We can also install any necessary packages in the now-active virtual environment. For example we can install yapf<\/code><\/span><\/span> (a formatter for Python files):<\/span><\/p>\n
\n
\n
(my-venv-dardel) $ python3 -m pip install yapf<\/pre>\n<\/div>\n<\/div>\n
and then check that yapf<\/code><\/span><\/span> is installed under \u201cmy-venv-dardel<\/code>\u201d:<\/span><\/span><\/p>\n
\n
\n
(my-venv-dardel) $ which yapf\r\n\/cfs\/klemming\/home\/u\/username\/my-venv-dardel\/bin\/yapf<\/pre>\n<\/div>\n<\/div>\n
To deactivate the virtual environment, use the \u201cdeactivate<\/code><\/span><\/span>\u201d command:<\/span><\/p>\n
\n
\n
(my-venv-dardel) $ deactivate\r\n$<\/pre>\n<\/div>\n<\/div>\n
After the \u201cdeactivate<\/code><\/span><\/span>\u201d command has been run, the \u201cmy-venv-dardel<\/code>\u201d environment will become inactive, as indicated by the disappearance of <\/span><\/span>the environment name surrounded by parentheses.<\/span><\/span><\/span><\/p>\n
Virtual environment with conda<\/h2>\n
\n
With the anaconda<\/code><\/span><\/span><\/span> module loaded on Dardel<\/a>, it is also natural to use \u201cconda<\/code>\u201d<\/span><\/span><\/span> to manage virtual environments for Python. conda<\/code><\/span><\/span><\/span><\/a> is an open-source package and environment manager, and is included in both Anaconda and Miniconda.<\/p>\n
To use conda<\/code><\/span><\/span><\/span>, we need to initialize it first. This is often done by modifying the ~\/.bashrc<\/code><\/span><\/span><\/span> script; however, the ~\/.bashrc<\/code><\/span><\/span><\/span> script is automatically loaded upon login, and you may not want to initialize conda<\/code><\/span><\/span><\/span> every time you log in to Dardel<\/a>. In practice it is recommended to use a separate script for initializing conda<\/code><\/span><\/span><\/span>. We can create a ~\/.bashrc.conda.dardel<\/code><\/span><\/span><\/span> file with the contents in the example below, which can then be used to initialize conda<\/code><\/span><\/span><\/span> for Dardel.<\/p>\n
\n
\n
module load PDC\/21.11<\/span>\r\nmodule load Anaconda3\/2021.05<\/span>\r\n\r\n# >>> conda initialize >>><\/span>\r\n# <\/span>!! Contents within this block are managed by 'conda init' !!<\/span>\r\n__conda_setup<\/span>=<\/span>\"$('<\/span>\/pdc\/software\/21.11\/eb\/software\/Anaconda3\/2021.05\/bin\/conda<\/span>' 'shell.bash' 'hook' 2> <\/span>\/dev\/null<\/span>)\"<\/span>\r\nif<\/span> [<\/span> $?<\/span> -eq<\/span> 0<\/span> ]<\/span>;<\/span> then<\/span>\r\n    eval<\/span> \"<\/span>$__conda_setup<\/span>\"<\/span>\r\nelse<\/span>\r\n    if<\/span> [<\/span> -f<\/span> \"\/pdc\/software\/21.11\/eb\/software\/Anaconda3\/2021.05\/<\/span><\/span>etc\/profile.d\/conda.sh<\/span>\"<\/span> ]<\/span>;<\/span> then<\/span>\r\n        .<\/span> \"\/pdc\/software\/21.11\/eb\/software\/Anaconda3\/2021.05\/<\/span><\/span>etc\/profile.d\/conda.sh<\/span>\"<\/span>\r\n    else<\/span>\r\n        export<\/span> PATH<\/span>=<\/span>\"\/pdc\/software\/21.11\/eb\/software\/Anaconda3\/2021.05\/<\/span><\/span>bin<\/span>:<\/span>$PATH<\/span>\"<\/span>\r\n    fi<\/span>\r\nfi<\/span>\r\nunset<\/span> __conda_setup\r\n# <<< conda initialize <<<<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n
After this we can load the anaconda module and initialize conda<\/code><\/span><\/span><\/span> with the following command:<\/p>\n
\n
\n
$ source ~\/.bashrc.conda.dardel\r\n(base) $<\/pre>\n<\/div>\n<\/div>\n
After the above command has been executed, we’ll see \u201cbase<\/code>\u201d<\/span><\/span><\/span> surrounded by parentheses, indicating that the default virtual environment is active.<\/p>\n
The next step is to configure conda<\/code><\/span><\/span><\/span> with user-defined directories. For example, we can edit the ~\/.condarc<\/code><\/span><\/span><\/span> file (or create the file if it doesn’t exist) so it contains the following lines.<\/p>\n
\n
\n
pkgs_dirs:\r\n    - \/cfs\/klemming\/home\/u\/username\/conda-dirs\/pkgs\r\nenvs_dirs:\r\n    - \/cfs\/klemming\/home\/u\/username\/conda-dirs\/envs\r\n<\/pre>\n<\/div>\n<\/div>\n
Now we are ready to create our own virtual environment with conda<\/code><\/span><\/span><\/span>. In practice, we use the \u201cconda create ...<\/code>\u201d command <\/span><\/span><\/span>and specify both the \u201c<\/span><\/span><\/span>--prefix<\/code>\u201d<\/span><\/span><\/span> option and the Python version with the following command:<\/p>\n
\n
\n
(base) $ conda create --prefix \/cfs\/klemming\/home\/u\/username\/conda-dirs\/envs\/my-conda-dardel<\/pre>\n<\/div>\n<\/div>\n
Note that the \u201c<\/span><\/span><\/span>--prefix<\/code>\u201d<\/span><\/span><\/span> option should point to the virtual environment that is under the specified environment directory (env_dirs<\/code><\/span><\/span><\/span>) in the ~\/.condarc<\/code><\/span><\/span><\/span> file. The output will look like the following:<\/p>\n
\n
\n
Collecting package metadata: done\r\nSolving environment: done\r\n\r\n## Package Plan ##\r\n\r\n  environment location: \/cfs\/klemming\/home\/u\/username\/conda-dirs\/envs\/my-conda-dardel\r\n\r\n  added \/ updated specs:\r\n    - python=3.8\r\n\r\nThe following packages will be downloaded:\r\n    ...\r\n\r\nThe following NEW packages will be INSTALLED:\r\n  ...\r\n\r\nProceed ([y]\/n)?\r\n<\/pre>\n<\/div>\n<\/div>\n
After pressing \u201cy\u201d<\/span><\/span><\/span> and the enter key, we\u2019ll see the following:<\/p>\n
\n
\n
Downloading and Extracting Packages\r\n...\r\n\r\nPreparing transaction: done\r\nVerifying transaction: done\r\nExecuting transaction: done\r\n\r\n# To activate this environment, use\r\n#\r\n# $ conda activate \/cfs\/klemming\/home\/u\/username\/conda-dirs\/envs\/my-conda-dardel\r\n#\r\n# To deactivate an active environment, use\r\n#\r\n# $ conda deactivate<\/pre>\n<\/div>\n<\/div>\n
With conda<\/code><\/span><\/span><\/span>, the commands to activate and deactivate a virtual environment are \u201cconda activate ...<\/code>\u201d<\/span><\/span><\/span>\u00a0and \u201cconda deactivate<\/code>\u201d<\/span><\/span><\/span>, respectively. Since we have already specified envs_dir<\/code><\/span><\/span><\/span> in ~\/.condarc<\/code><\/span><\/span><\/span>, we can activate the my-conda-dardel<\/code><\/span><\/span><\/span> environment using this command:<\/p>\n
\n
\n
(base) $ conda activate my-conda-dardel\r\n(my-conda-dardel) $<\/pre>\n<\/div>\n<\/div>\n
We can check that the Python site directory is now under my-conda-dardel<\/code> as follows:<\/span><\/span><\/span><\/p>\n
\n
\n
(my-conda-dardel) $ python3 -c 'import site; print(site.getsitepackages())'\r\n['\/cfs\/klemming\/home\/u\/username\/conda-dirs\/envs\/my-conda-dardel\/lib\/python3.8\/site-packages']<\/pre>\n<\/div>\n<\/div>\n
We can also install yapf<\/code><\/span><\/span><\/span>, now using \u201cconda install<\/code><\/span><\/span><\/span>\u201d as in the following example (and by pressing the \u201cy\u201d<\/span><\/span><\/span> and enter keys to proceed):<\/p>\n
\n
\n
(my-conda-dardel) $ conda install yapf\r\n\r\nCollecting package metadata: done\r\nSolving environment: done\r\n\r\n## Package Plan ##\r\n\r\n  environment location: \/cfs\/klemming\/home\/u\/username\/conda-dirs\/envs\/my-conda-dardel\r\n\r\n  added \/ updated specs:\r\n    - yapf\r\n\r\nThe following packages will be downloaded:\r\n    ...\r\n\r\nThe following NEW packages will be INSTALLED:\r\n  ...\r\n\r\nProceed ([y]\/n)? y\r\n\r\nDownloading and Extracting Packages\r\n...\r\n\r\nPreparing transaction: done\r\nVerifying transaction: done\r\nExecuting transaction: done<\/pre>\n<\/div>\n<\/div>\n
We can then check that <\/span>yapf<\/code><\/span><\/span><\/span> has been installed under <\/span>my-conda-dardel<\/code> using the following command:<\/span><\/span><\/span><\/span><\/p>\n
\n
\n
(my-conda-dardel) $ which yapf\r\n\/cfs\/klemming\/home\/u\/username\/conda-dirs\/envs\/my-conda-dardel\/bin\/yapf\r\n<\/pre>\n<\/div>\n<\/div>\n
To deactivate the virtual environment, use the \u201cconda deactivate<\/code>\u201d <\/span><\/span>command:<\/span><\/p>\n
\n
\n
(my-conda-dardel) $ conda deactivate\r\n(base) $<\/pre>\n<\/div>\n<\/div>\n
After the \u201cconda deactivate<\/code><\/span><\/span>\u201d command has been run, the \u201cmy-conda-dardel<\/code>\u201d <\/span><\/span>environment will become inactive and the default \u201cbase<\/code>\u201d <\/span><\/span>environment will become active again. <\/span>The \u201cconda deactivate<\/code><\/span><\/span>\u201d command can also be used to<\/span> deactivate the base environment as follows:<\/p>\n
\n
\n
(base) $ conda deactivate\r\n$<\/pre>\n<\/div>\n<\/div>\n
Summary<\/h2>\n
We have briefly introduced the creation and management of a Python virtual environment using venv<\/code><\/span><\/span><\/span> and conda<\/code><\/span><\/span><\/span>. The former is lightweight and easy-to-use while the latter is more powerful and versatile. You can find more information about them in the following links:<\/p>\n