June 18, 2019 at 18:03Useful Python Tools
- Transfer
Friends, good evening! We have great news, open recruitment in a new group on the course "Python Developer" . The group starts already in early July, and right now, according to an established tradition, we are sharing a useful translation prepared for students of this course.
When you first start learning Python, someone explains to you that you can add your source folder to the PYTHONPATH environment variable and then your code can be imported from other directories. Very often the explainer forgets to say that in most cases this is a bad idea. Some people find this out on the Internet, others just understand it first hand. But too many people (especially inexperienced programmers) think that there can be no other alternatives.
This article is mainly for them, because even if you know that there is an alternative, it is not always easy to accept it and start using it. Python tools are confusing because they are a lot of software built on top of one another, with a lot of overlaps and problems that result from this. It is not easy to understand how to use these tools correctly in your project.
For this reason, I decided to write this article and consider the most popular tools in it, to figure out when and where they are used and what tasks they solve. I will try to explain on fingers how to use each of these tools. If the tool is on this list, then you, as a pythonist, need to at least know about its existence. I will only talk about those tools that can be applied to any project or workflow, and you should remember about them when you start a new project. However, this does not mean that you should use all the tools presented in each of your projects. It is unnecessary to overload the project with tools, in some cases this can complicate its support.
Setuptools is the standard way to create packages in Python. He works anywhere and does his job well.
Why : create egg, zip or wheel files from source, define metadata for your project, joint structured and standardized work on code.
When used : Always when you write code that should run on someone else's machine.
Alternatives : Poetry, Flit
Virtualenv is a virtual environment manager. Such sandboxes are stand-alone python with a specific set of pre-installed packages. Using virtualenv means that you do not need to install packages on the python system by default.
Why : separation of dependencies, support for different versions of python by one system, easy relocation of dependencies.
When used : You need to write code, and for this you need a python version different from your default python system version.
Alternatives : Docker or something similar.
Pip is the most common package manager in python. It allows you to install local or remote packages in your virtual environment or Python system.
For what : installing and removing packages, tracking the versions of packages that you use.
When used : Always.
Alternatives : Poetry, Conda
distutils is the forerunner of setuptools. The latter actively uses the distutils functionality, so it is often necessary to interact with this tool. Distutils is not exactly the tool that you should have in your arsenal, but you should know how it fits into the big picture.
Pypi or Python Package Index is a large repository that contains all your favorite Python modules. For example, the same pip takes package builds from there.
Why : To publish your code.
When used : When there is a package that you want to show to the community.
Pypiserver is one of the implementations of the Package Index API used by Pypi. You can create your own repository, for example, for your entire company and publish packages without making public releases.
Why : Create your own repositories within the organization.
When used : When your code does not need publicity, but full control is needed over it.
Alternatives : Warehouse (used by Pypi), djangopypi
Poetry is an alternative package management system that replaces setuptools, pip, and some other tools built on their basis. This is an attempt to completely redefine how the package system works in Python. To date, poetry has many positive reviews, but is not the most common tool.
Why : processing and distribution of packages, managing dependencies, preventing problems with resolving dependencies.
When to use : When you are planning a new project and you are not afraid to use highly specialized tools.
Alternatives : Pipenv
Pipenv , like Poetry, is a tool for structuring dependencies and configuring Python projects in a more sane way. With Pipfile, it manages the dependencies of your project and ensures consistency and ease of use.
Why : processing and distribution of packages, dependency management.
When used : you need a tool like Poetry that will cause fewer questions.
Alternatives : Poetry.
Sphinx is a tool for creating documentation. It was originally created to handle Python documentation, but has become a common tool. It is the most common option for Python projects.
Why : create PDF or HTML documents using a markup language from reStructuredText sources.
When used : When your project, API, or code requires external documentation.
Alternatives : Docutils, Doxygen
autodoc is a fundamental extension for Sphinx that allows you to create reStructuredText files from Python source code with signatures for each class, function, module, and so on.
Why : Documenting your code or API.
When used : In fact, every time you use Sphinx.
Alternatives : autosummary
py.test - in my opinion, is the best package for testing in Python. It has many functions, although not all of them are opened properly, so it will take some time to search for all the possibilities that py.test provides.
Why : testing your code.
When used : Always when you are too lazy to test manually.
Alternatives : unittest, nose
Hypothesis is a tool for testing individual properties. In short, it generates random test scripts according to your specifications until it finds a scenario where the test fails. Take some time to learn the principles before you start using this tool.
Why : code testing, especially data processing.
When to use : When you need to test non-trivial logic with a wide range of input values (numbers, strings, structured data).
tox is a virtual environment manager for testing. This means that you can configure it to run tests in clean, customizable virtual environments to ensure that your code can work under various conditions.
Why : for code that should run in various conditions and environments. Also useful for CI.
When to use : When you need your code to be supported by different versions of Python, run in different environments and on different operating systems.
Alternatives : bash scrips, CI pipelines
pyenv is the python version manager. It aims to simplify the local workflow of developers when working with multiple versions.
Why : launch various projects with different versions of Python.
When used : You need to work with global versions of Python and you have a lot of them.
Alternatives : manual management, virtualenv, Poetry, Pipenv
PyScaffold is a tool for initializing a project’s structure in a standardized way and providing some of the tools listed above without having to manually configure them. Very flexible.
Why : for loading projects, working with several projects with the same tools and structure.
When used : always (if you are familiar with this tool, but do not try to use it for the first time when you are in a hurry)
Alternatives : python-project-template, Cookiecutter
flake8 is one of the most popular linters for Python. It runs various scripts to verify that your code matches the requirements of the Python Style Guide ( PEP-8 ).
For what : checking your project for a good writing style.
When used : every time your project should be read by someone or you.
Alternatives : pylint
Black automatically formats the code. This means that instead of just checking your code for compliance with standards, Black will independently change it to fit it.
Why : automatic code formatting.
When used : When you have no problem refusing to manually manage your code.
Alternatives : autopep8, yapf
That's all. Waiting for your comments ;-).
When you first start learning Python, someone explains to you that you can add your source folder to the PYTHONPATH environment variable and then your code can be imported from other directories. Very often the explainer forgets to say that in most cases this is a bad idea. Some people find this out on the Internet, others just understand it first hand. But too many people (especially inexperienced programmers) think that there can be no other alternatives.
This article is mainly for them, because even if you know that there is an alternative, it is not always easy to accept it and start using it. Python tools are confusing because they are a lot of software built on top of one another, with a lot of overlaps and problems that result from this. It is not easy to understand how to use these tools correctly in your project.
For this reason, I decided to write this article and consider the most popular tools in it, to figure out when and where they are used and what tasks they solve. I will try to explain on fingers how to use each of these tools. If the tool is on this list, then you, as a pythonist, need to at least know about its existence. I will only talk about those tools that can be applied to any project or workflow, and you should remember about them when you start a new project. However, this does not mean that you should use all the tools presented in each of your projects. It is unnecessary to overload the project with tools, in some cases this can complicate its support.
Basic tools
Setuptools
Setuptools is the standard way to create packages in Python. He works anywhere and does his job well.
Why : create egg, zip or wheel files from source, define metadata for your project, joint structured and standardized work on code.
When used : Always when you write code that should run on someone else's machine.
Alternatives : Poetry, Flit
virtualenv
Virtualenv is a virtual environment manager. Such sandboxes are stand-alone python with a specific set of pre-installed packages. Using virtualenv means that you do not need to install packages on the python system by default.
Why : separation of dependencies, support for different versions of python by one system, easy relocation of dependencies.
When used : You need to write code, and for this you need a python version different from your default python system version.
Alternatives : Docker or something similar.
Pip
Pip is the most common package manager in python. It allows you to install local or remote packages in your virtual environment or Python system.
For what : installing and removing packages, tracking the versions of packages that you use.
When used : Always.
Alternatives : Poetry, Conda
Package creation and distribution
For a closer look, python.org has a separate page: packaging.python.org
distutils
distutils is the forerunner of setuptools. The latter actively uses the distutils functionality, so it is often necessary to interact with this tool. Distutils is not exactly the tool that you should have in your arsenal, but you should know how it fits into the big picture.
Pypi
Pypi or Python Package Index is a large repository that contains all your favorite Python modules. For example, the same pip takes package builds from there.
Why : To publish your code.
When used : When there is a package that you want to show to the community.
Pypiserver
Pypiserver is one of the implementations of the Package Index API used by Pypi. You can create your own repository, for example, for your entire company and publish packages without making public releases.
Why : Create your own repositories within the organization.
When used : When your code does not need publicity, but full control is needed over it.
Alternatives : Warehouse (used by Pypi), djangopypi
Poetry
Poetry is an alternative package management system that replaces setuptools, pip, and some other tools built on their basis. This is an attempt to completely redefine how the package system works in Python. To date, poetry has many positive reviews, but is not the most common tool.
Why : processing and distribution of packages, managing dependencies, preventing problems with resolving dependencies.
When to use : When you are planning a new project and you are not afraid to use highly specialized tools.
Alternatives : Pipenv
Pipenv
Pipenv , like Poetry, is a tool for structuring dependencies and configuring Python projects in a more sane way. With Pipfile, it manages the dependencies of your project and ensures consistency and ease of use.
Why : processing and distribution of packages, dependency management.
When used : you need a tool like Poetry that will cause fewer questions.
Alternatives : Poetry.
Documentation
Sphinx
Sphinx is a tool for creating documentation. It was originally created to handle Python documentation, but has become a common tool. It is the most common option for Python projects.
Why : create PDF or HTML documents using a markup language from reStructuredText sources.
When used : When your project, API, or code requires external documentation.
Alternatives : Docutils, Doxygen
autodoc
autodoc is a fundamental extension for Sphinx that allows you to create reStructuredText files from Python source code with signatures for each class, function, module, and so on.
Why : Documenting your code or API.
When used : In fact, every time you use Sphinx.
Alternatives : autosummary
Testing
py.test
py.test - in my opinion, is the best package for testing in Python. It has many functions, although not all of them are opened properly, so it will take some time to search for all the possibilities that py.test provides.
Why : testing your code.
When used : Always when you are too lazy to test manually.
Alternatives : unittest, nose
Hypothesis
Hypothesis is a tool for testing individual properties. In short, it generates random test scripts according to your specifications until it finds a scenario where the test fails. Take some time to learn the principles before you start using this tool.
Why : code testing, especially data processing.
When to use : When you need to test non-trivial logic with a wide range of input values (numbers, strings, structured data).
tox
tox is a virtual environment manager for testing. This means that you can configure it to run tests in clean, customizable virtual environments to ensure that your code can work under various conditions.
Why : for code that should run in various conditions and environments. Also useful for CI.
When to use : When you need your code to be supported by different versions of Python, run in different environments and on different operating systems.
Alternatives : bash scrips, CI pipelines
Other tools
pyenv
pyenv is the python version manager. It aims to simplify the local workflow of developers when working with multiple versions.
Why : launch various projects with different versions of Python.
When used : You need to work with global versions of Python and you have a lot of them.
Alternatives : manual management, virtualenv, Poetry, Pipenv
PyScaffold
PyScaffold is a tool for initializing a project’s structure in a standardized way and providing some of the tools listed above without having to manually configure them. Very flexible.
Why : for loading projects, working with several projects with the same tools and structure.
When used : always (if you are familiar with this tool, but do not try to use it for the first time when you are in a hurry)
Alternatives : python-project-template, Cookiecutter
flake8
flake8 is one of the most popular linters for Python. It runs various scripts to verify that your code matches the requirements of the Python Style Guide ( PEP-8 ).
For what : checking your project for a good writing style.
When used : every time your project should be read by someone or you.
Alternatives : pylint
Black
Black automatically formats the code. This means that instead of just checking your code for compliance with standards, Black will independently change it to fit it.
Why : automatic code formatting.
When used : When you have no problem refusing to manually manage your code.
Alternatives : autopep8, yapf
That's all. Waiting for your comments ;-).