How the virtual environment libraries work

Have you ever wondered how the virtual environment libraries work in Python? In this article, I propose to get acquainted with the main concept that all libraries use for environments, such as virtualenv, virtualenvwrapper, conda, pipenv.

Initially, in Python there was no built-in ability to create environments, and this feature was implemented as a hack. As it turned out, all libraries are based on a very simple feature of the python interpreter.

When Python starts the interpreter, it starts looking for a site-packages directory. The search starts from the parent directory regarding the physical location of the interpreter executable (python.exe). If the module folder is not found, then Python goes to the next level, and does so until the root directory is reached. In order to understand that this is a directory with modules, Python searches for the os module, which should be in the os.py file and is required for python to work.

Let's imagine that our interpreter is located at /usr/dev/lang/bin/python. Then the search paths will look like this:

/usr/dev/lang/lib/python3.7/os.py
/usr/dev/lib/python3.7/os.py
/usr/lib/python3.7/os.py
/lib/python3.7/os.py

As you can see, Python adds a special prefix ( lib/python$VERSION/os.py) to our path. As soon as the interpreter finds the first match (presence os.py file), it changes sys.prefix, and sys.exec_prefixon this path (to remove the prefix). If for some reason no match is found, then the standard path is used which is compiled into the interpreter.

Now let's see how this is done by one of the oldest and most famous libraries - virtualenv.

user@arb:/usr/home/test# virtualenv ENV
Running virtualenv with interpreter /usr/bin/python3
New python executable in /usr/home/test/ENV/bin/python3
Also creating executable in /usr/home/test/ENV/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

After execution, it creates additional directories:

user@arb:/usr/home/test/ENV# tree -L 3
.
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate_this.py
│   ├── easy_install
│   ├── easy_install-3.7
│   ├── pip
│   ├── pip3
│   ├── pip3.7
│   ├── python
│   ├── python-config
│   ├── python3 -> python
│   ├── python3.7 -> python
│   └── wheel
├── include
│   └── python3.7m -> /usr/include/python3.7m
├── lib
│   └── python3.7
│   ├── __future__.py -> /usr/lib/python3.7/__future__.py
│   ├── __pycache__
│   ├── _bootlocale.py -> /usr/lib/python3.7/_bootlocale.py
│   ├── _collections_abc.py -> /usr/lib/python3.7/_collections_abc.py
│   ├── _dummy_thread.py -> /usr/lib/python3.7/_dummy_thread.py
│   ├── _weakrefset.py -> /usr/lib/python3.7/_weakrefset.py
│   ├── abc.py -> /usr/lib/python3.7/abc.py
│   ├── base64.py -> /usr/lib/python3.7/base64.py
│   ├── bisect.py -> /usr/lib/python3.7/bisect.py
│   ├── codecs.py -> /usr/lib/python3.7/codecs.py
│   ├── collections -> /usr/lib/python3.7/collections
│   ├── config-3.7m-darwin -> /usr/lib/python3.7/config-3.7m-darwin
│   ├── copy.py -> /usr/lib/python3.7/copy.py
│   ├── copyreg.py -> /usr/lib/python3.7/copyreg.py
│   ├── distutils
│   ├── encodings -> /usr/lib/python3.7/encodings
│   ├── enum.py -> /usr/lib/python3.7/enum.py
│   ├── fnmatch.py -> /usr/lib/python3.7/fnmatch.py
│   ├── functools.py -> /usr/lib/python3.7/functools.py
│   ├── genericpath.py -> /usr/lib/python3.7/genericpath.py
│   ├── hashlib.py -> /usr/lib/python3.7/hashlib.py
│   ├── heapq.py -> /usr/lib/python3.7/heapq.py
│   ├── hmac.py -> /usr/lib/python3.7/hmac.py
│   ├── imp.py -> /usr/lib/python3.7/imp.py
│   ├── importlib -> /usr/lib/python3.7/importlib
│   ├── io.py -> /usr/lib/python3.7/io.py
│   ├── keyword.py -> /usr/lib/python3.7/keyword.py
│   ├── lib-dynload -> /usr/lib/python3.7/lib-dynload
│   ├── linecache.py -> /usr/lib/python3.7/linecache.py
│   ├── locale.py -> /usr/lib/python3.7/locale.py
│   ├── no-global-site-packages.txt
│   ├── ntpath.py -> /usr/lib/python3.7/ntpath.py
│   ├── operator.py -> /usr/lib/python3.7/operator.py
│   ├── orig-prefix.txt
│   ├── os.py -> /usr/lib/python3.7/os.py
│   ├── posixpath.py -> /usr/lib/python3.7/posixpath.py
│   ├── random.py -> /usr/lib/python3.7/random.py
│   ├── re.py -> /usr/lib/python3.7/re.py
│   ├── readline.so -> /usr/lib/python3.7/lib-dynload/readline.cpython-37m-darwin.so
│   ├── reprlib.py -> /usr/lib/python3.7/reprlib.py
│   ├── rlcompleter.py -> /usr/lib/python3.7/rlcompleter.py
│   ├── shutil.py -> /usr/lib/python3.7/shutil.py
│   ├── site-packages
│   ├── site.py
│   ├── sre_compile.py -> /usr/lib/python3.7/sre_compile.py
│   ├── sre_constants.py -> /usr/lib/python3.7/sre_constants.py
│   ├── sre_parse.py -> /usr/lib/python3.7/sre_parse.py
│   ├── stat.py -> /usr/lib/python3.7/stat.py
│   ├── struct.py -> /usr/lib/python3.7/struct.py
│   ├── tarfile.py -> /usr/lib/python3.7/tarfile.py
│   ├── tempfile.py -> /usr/lib/python3.7/tempfile.py
│   ├── token.py -> /usr/lib/python3.7/token.py
│   ├── tokenize.py -> /usr/lib/python3.7/tokenize.py
│   ├── types.py -> /usr/lib/python3.7/types.py
│   ├── warnings.py -> /usr/lib/python3.7/warnings.py
│   └── weakref.py -> /usr/lib/python3.7/weakref.py
└── pip-selfcheck.json

As you can see, the virtual environment was created by copying the Python binary into a local folder (ENV / bin / python). We can also notice that the parent folder contains symbolic links to the standard python library files. We cannot create a symbolic link to the executable file, since the interpreter will still name it to the actual path.

Now let's activate our environment:

user@arb:/usr/home/test# source ENV/bin/activate

This command changes the $ PATH environment variable so that the command pythonpoints to our local version of python. This is achieved by substituting the local path of the bin folder to the beginning of the $ PATH line so that the local path takes precedence over all paths to the right.

export"/usr/home/test/ENV/bin:$PATH"echo$PATH

If you run the script from this environment, it will be executed using the binary at /usr/home/test/ENV/bin/python. The interpreter will use this path as a starting point for finding modules. In our case, the modules of the standard library will be found along the way /usr/home/test/ENV/lib/python3.7/.

This is the main hack, thanks to which all libraries work with virtual environments.

Python 3 enhancements

Starting with Python 3.3, a new standard has emerged, referred to as PEP 405 , which introduces a new mechanism for lightweight environments.

This PEP adds an extra step to the search process. If you create a configuration file pyenv.cfg, instead of copying the Python binary and all its modules, you can simply specify their location in this config.

This feature is actively used by the standard venv module , which appeared in Python 3.

user@arb:/usr/home/test2# python3 -m venv ENV
user@arb:/usr/home/test2# tree -L 3
.
└── ENV
  ├── bin
  │   ├── activate
  │   ├── activate.csh
  │   ├── activate.fish
  │   ├── easy_install
  │   ├── easy_install-3.7
  │   ├── pip
  │   ├── pip3
  │   ├── pip3.5
  │   ├── python -> python3
  │   └── python3 -> /usr/bin/python3
  ├── include
  ├── lib
  │   └── python3.7
  ├── lib64 -> lib
  ├── pyvenv.cfg
  └── share
  └── python-wheels

user@arb:/usr/home/test2# cat ENV/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.7.0
user@arb:/usr/home/test2# readlink ENV/bin/python3
/usr/bin/python3

Thanks to this configuration, instead of copying the binary, venv simply creates a link to it. If the parameter is include-system-site-packageschanged to true, then all modules of the standard library will be automatically accessible from the virtual environment.

Despite these changes, most third-party libraries for working with virtual environments use the old approach.

PS: I am the author of this article, you can ask any questions.