July 1, 2019 at 22:05Combining multiple packages into a single Python namespace
Sometimes it becomes necessary to separate several packages lying in the same namespace along different physical paths. For example, if you want to be able to transfer different layouts of plugins, having the ability to subsequently add them without controlling their location, and at the same time, access them through one namespace.
This cheat sheet, which is more suitable for beginners, is dedicated to Python namespaces.
Let's look at how this can be done in different versions of Python, since although Python2 is no longer being supported soon, many of us are right now between two fires, and this is just one of the important nuances in the transition.
Consider this example:
We want to get the package structure:
Module1 file contents
Module2 file contents
At the same time, packages are distributed in the following folder structure:
Suppose that somehow path1 and path2 are already added to sys.path. We need to access module1 and module2:
What happens in Python 3.7 when this code is executed? Everything works wonderfully:
With PEP-420 in Python 3.3, support for implicit namespaces has appeared. In addition, when importing a package from py33, you do not need to create __init__.py files. And when importing namespace, it is just _ forbidden_. If the __init__.py file is present in one or both directories with the name name1, an error will occur on the import of the second package.
Thus, the presence of an insider explicitly determines the package, and packages cannot be combined, it is a single entity. If you are starting a new project, independent of the old development, and the packages will be installed using pip, then you need to stick to this method. However, sometimes we inherit the old code, which also needs to be maintained, at least for a while, or ported to a new version.
Let's move on to Python 2.7 . With this version it is already more interesting, you first need to add __init__.py to each directory to create packages, otherwise the interpreter simply does not recognize the package in this set of files. And then write explicit namespace declaration in __init__ files related to namespace1, otherwise, only the first package will be imported.
What happens with this? When the interpreter reaches the first import, a package with that name is searched in sys.path, it is in path1 / namespace1 and the interpreter executes path1 / namespace1 / __ init__.py. Further search is not conducted. However, the extend_path function itself searches already throughout sys.path, finds all packages with the name namespace1 and the internal name and adds them to the __path__ variable of the namespace1 package, which is used to search for child packages in this namespace.
In official guides, it is recommended that the initials be the same each time namespace1 is placed. In fact, they can be empty all but the first one, which is found during a search in sys.path, in which pkgutil.extend_path should be called, because the rest are not executed. However, of course, it’s better that the call is in every internal office, so as not to tie your logic “in case” and not to guess which team was the first to execute, because the search order can change. For the same reason, you should not place any other logic __init__ files in the variable area.
This will work in future versions, and this code can be used to write compatible code., but you need to consider that you must adhere to the selected method in each distributed package. If on version 3 you put an in-box in some packages in a call to pkgutil.extend_path and leave some without an in-box, this will not work.
In addition, this option is also suitable for the case when you plan to install using python setup.py install.
Another way, which is now considered somewhat outdated, but it can still be found a lot where:
The pkg_resources module comes with the setuptools package. Here the meaning is the same as in pkgutil - it is necessary that each __init__ file contains the same namespace declaration at every location of namespace1 and no other code is present. At the same time, it is necessary to register namespace namespace_packages = ['namespace1'] in setup.py. A more detailed description of creating packages is beyond the scope of this article.
In addition, you can often find such a code
Here the logic is simple - if setuptools is not installed, then we use pkgutil, which is included in the standard library.
If you configure a namespace in one of these ways, you can call another from one module. For example, change namespace1 / package2 / module2
And then we'll see what happens if we mistakenly name a new package as well as an existing one and wrap it with the same namespace. For example, there will be two packages in different places with the name package1.
In this case, only the first one will be imported and there will be no access to module2. Packages cannot be combined.
Summary:
Sources:
Examples can be found here .
This cheat sheet, which is more suitable for beginners, is dedicated to Python namespaces.
Let's look at how this can be done in different versions of Python, since although Python2 is no longer being supported soon, many of us are right now between two fires, and this is just one of the important nuances in the transition.
Consider this example:
We want to get the package structure:
namespace1
package1
module1
package2
module2Module1 file contents
print('package 1')
var1 = 1Module2 file contents
print('package 2')
var2 = 2
At the same time, packages are distributed in the following folder structure:
path1
namespace1
package1
module1
path2
namespace1
package2
module2
Suppose that somehow path1 and path2 are already added to sys.path. We need to access module1 and module2:
from namespace1.package1 import module1
from namespace1.package2 import module2
What happens in Python 3.7 when this code is executed? Everything works wonderfully:
package 1
package 2
With PEP-420 in Python 3.3, support for implicit namespaces has appeared. In addition, when importing a package from py33, you do not need to create __init__.py files. And when importing namespace, it is just _ forbidden_. If the __init__.py file is present in one or both directories with the name name1, an error will occur on the import of the second package.
ModuleNotFoundError: No module named 'namespace1.package2'
Thus, the presence of an insider explicitly determines the package, and packages cannot be combined, it is a single entity. If you are starting a new project, independent of the old development, and the packages will be installed using pip, then you need to stick to this method. However, sometimes we inherit the old code, which also needs to be maintained, at least for a while, or ported to a new version.
Let's move on to Python 2.7 . With this version it is already more interesting, you first need to add __init__.py to each directory to create packages, otherwise the interpreter simply does not recognize the package in this set of files. And then write explicit namespace declaration in __init__ files related to namespace1, otherwise, only the first package will be imported.
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
What happens with this? When the interpreter reaches the first import, a package with that name is searched in sys.path, it is in path1 / namespace1 and the interpreter executes path1 / namespace1 / __ init__.py. Further search is not conducted. However, the extend_path function itself searches already throughout sys.path, finds all packages with the name namespace1 and the internal name and adds them to the __path__ variable of the namespace1 package, which is used to search for child packages in this namespace.
In official guides, it is recommended that the initials be the same each time namespace1 is placed. In fact, they can be empty all but the first one, which is found during a search in sys.path, in which pkgutil.extend_path should be called, because the rest are not executed. However, of course, it’s better that the call is in every internal office, so as not to tie your logic “in case” and not to guess which team was the first to execute, because the search order can change. For the same reason, you should not place any other logic __init__ files in the variable area.
This will work in future versions, and this code can be used to write compatible code., but you need to consider that you must adhere to the selected method in each distributed package. If on version 3 you put an in-box in some packages in a call to pkgutil.extend_path and leave some without an in-box, this will not work.
In addition, this option is also suitable for the case when you plan to install using python setup.py install.
Another way, which is now considered somewhat outdated, but it can still be found a lot where:
#namespace1/__init__.py
__import__('pkg_resources').declare_namespace(__name__)
The pkg_resources module comes with the setuptools package. Here the meaning is the same as in pkgutil - it is necessary that each __init__ file contains the same namespace declaration at every location of namespace1 and no other code is present. At the same time, it is necessary to register namespace namespace_packages = ['namespace1'] in setup.py. A more detailed description of creating packages is beyond the scope of this article.
In addition, you can often find such a code
try:
__import__('pkg_resources').declare_namespace(__name__)
except:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
Here the logic is simple - if setuptools is not installed, then we use pkgutil, which is included in the standard library.
If you configure a namespace in one of these ways, you can call another from one module. For example, change namespace1 / package2 / module2
import namespace1.package1.module1
print(var1)
And then we'll see what happens if we mistakenly name a new package as well as an existing one and wrap it with the same namespace. For example, there will be two packages in different places with the name package1.
namespace1
package1
module1
package1
module2
In this case, only the first one will be imported and there will be no access to module2. Packages cannot be combined.
from namespace1.package1 import module1
from namespace1.package1 import module2
#>>ImportError: cannot import name module2Summary:
- For Python older than 3.3 and installing with pip, it is recommended that you use an implicit namespace declaration.
- In case of support for versions 2 and 3, as well as installation with both pip and python setup.py install, the option with pkgutil is recommended.
- The pkg_resources option is recommended if you need to support older packages using this method, or you need the package to be zip-safe.
Sources:
- packaging.python.org/guides/packaging-namespace-packages
- www.python.org/dev/peps/pep-0420
- Learning Python, 5th Edition by Mark Lutz
Examples can be found here .