10 steps to a successful Python project

Original author: Jeff Hale
  • Transfer
The material, the translation of which we publish today, is devoted to tools that allow equipping Python projects with code formatting, testing, continuous integration and dependency analysis tools. This helps speed up the development process, helps to improve the quality, uniformity and security of the code. It is assumed that the reader of this material already has some experience in Python development and a Python project with which he, during reading, will experiment. If you don’t have such a project, here you can learn how to prepare a development environment and create a Python package. The examples that will be given here are prepared using macOS and Python 3.7.



Step 1. Installing Black



Project code must follow code style conventions. Black is a Python package that automatically formats the code, bringing its appearance to the PEP 8 standard . Black is a relatively new project, but it has already gained over a million downloads. Its use quickly became a sign of good taste in Python development. Here is the Black manual.

I, as a code editor, use Atom, so I added a package to Atom Python-Black. You can find out how to install it here . After installing this package, Atom will reformat the code after saving the file.

While we are talking about Black - let's equip this tool with the development environment of those who work with us on the project. As a result, everyone who works on the project will use the same code formatting rules, otherwise their pull requests will not be accepted.

Add the parameter black==18.9b0to the first found free line of the file requirements_dev.txtand run the command install -r requirements_dev.txt.

Black, by default, sets the length of the line of code to 88 characters. Some style guides, such as Sphinx , require a string length of 79 characters. In the package settings, Black-Atomyou can specify the desired string length.

Now that we have acquired a tool that will help save time on code formatting, we’ll think about how to speed up and simplify sending code to PyPI.

Step 2. Creating the .pypirc File


When twine is used to send application assemblies to TestPyPI and PyPI, you must manually enter login information. If you are not familiar with twine, take a look at this material. Now we will automate this process.

Twine can work with a file .pypirc, which should be in our home directory. This tool, unloading data, takes a URL, username and password from a given file.

So, create a file in the home directory .pypirc:

touch ~/.pypirc

Add the following text to it:

[distutils]
index-servers =
    pypi
    testpypi
[testpypi]
repository: https://test.pypi.org/legacy
username = your_username
password = your_pypitest_password
[pypi]
username = your_username
password = your_pypi_password

It is clear that here you must enter your real username and password. Also, check that this file is saved in the home directory, and not in the current working directory. If you want to protect this file from other users, you can, using the command line tools, configure its permissions:

chmod 600 ~/.pypirc

Now your package can be loaded into TestPyPI using the following command:

twine upload -r testpypi dist/*

In regular PyPI, you can load packages like this:

twine upload dist/*

After you get the file .pypirc, you no longer have to manually enter your username and password.

Now let's add testing tools to our development environment that will allow us to verify the correct operation of the package that we are creating.

Step 3. Install and configure pytest



Pytest is the most popular, easy-to-use library for testing code written in Python. In this example, we will add simple tests to the project. Now , if you are interested in the details of pytest, a good introductory guide to this tool.

Add pytest information to the file requirements_dev.txt:

pytest==4.3.0

Let's install the package:

pip install requirements_dev.txt

Now execute the following command, which will allow pytest to find our package:

pip install -e .

If you deactivated your virtual development environment, then you will need to run both commands again to run the tests pip.

Step 4. Creating Tests


Add the folder testto the root directory of your project. Put the file in it test_your_package_name.py. My file is called test_notebookc.py. If the file name starts with test_, pytest can automatically detect such a file. I added the following test to the

file test_notebookc.py, which is aimed at checking whether the function displays the correct name. Modify this code so that the file and function names used in it match yours, describe your own tests in it.

"""Tests for `notebookc` package."""
import pytest
from notebookc import notebookc
def test_convert(capsys):
    """Correct my_name argument prints"""
    notebookc.convert("Jill")
    captured = capsys.readouterr()
    assert "Jall" in captured.out

What's going on here?

First we import our module here. Then we create a function whose name is built according to the template test_my_function_name. This function naming convention allows other people reading your project code to quickly understand what exactly is being tested in the tests. In addition, this is necessary for a package that helps control the coverage of code with tests, which we will discuss below.

After that, we call the function ( convert), passing it the name as an argument Jill. Next - capture what the function displays. Here it is worth saying that the function in question is extremely simple. She takes a parameter my_nameand does the following:

print(f"I’ll convert a notebook for you some day, {my_name}.")

Pytest checks to see if Jallthe function prints. This line should not be there, since we are passing functions Jill. Here is the pytest documentation where you can find information on intercepting output.

Run the test by typing on the command line pytest. This test should fail. Error information is displayed in red.


An error was detected during the test.

It is recommended to check the tests for correctness, describing them so that, under certain conditions, they would end with an error. You should not write tests that give out only green messages, as otherwise it may turn out that the tests do not check at all what they are written to verify.

After we have ascertained that the test failed, change the statement Jallto Jilland start the test again. Now it should complete successfully.


Successful completion of the test.

Now everything is fine. The test allows you to make sure that if someone passes a line to our function, this line will fall into the text that this function displays.

You can also write a test that checks the function on how it handles the data passed to it. Namely, if it receives data whose type is different from string, it should cause an error TypeError. Here is some good stuff on exceptions and error handling in Python.

When we created the previous test, we wrote code that leads to the successful completion of the test. This is called Test-Driven Development (TDD). TDD is a proven programming approach that helps you write code that has fewer errors than it would without TDD. Here is some useful TDD material.

Now, as an exercise, try writing a test that checks the function convert()so that when passing something different from the string to it, it will throw an error, and implement the appropriate mechanisms for this function. Note that integers, lists, and dictionaries are converted to strings.

After the package has successfully passed the tests, we are ready to take advantage of the continuous integration system.

Step 5. Registration in the Travis CI service and its configuration



Travis CI is a "distributed web service for building and testing software." It was recently bought by Idera . There are other continuous integration systems, but Travis CI is a popular, open-source and well-documented tool, so we will use it.

Travis CI allows you to integrate into your project only the code that passes the tests and meets the standards. Here you can read more about Travis CI, and here about continuous integration .

Create an account at https://travis-ci.org/ . Next, click on the linkReview and add your authorized organizationson the profile page. You will be prompted for a password to access GitHub. Click on Grantthe section Organization access.


Configuring the Travis CI

Account I needed to synchronize the account so that information about notebooktoallthe repository appeared in the account notebookc. Typically, for Travis CI to work with code, it takes about a minute. After that, you need to activate the repository using the switch shown in the following figure.


Activating the repository

Now you need to click on the button Settings. Here you need to indicate whether Travis can build based on pull requests or branches sent to the repository.


Setting up the project assembly

Now it's time to set up the project we are working on, which will enable Travis to build the project for each pull request.

Step 6. Creating the .travis.yml file


In the project root folder, create a file .travis.ymlwith the following contents:

dist: xenial
language: python
python: 3.7.2
install:
  - pip install -r requirements_dev.txt
  - pip install -e .
script:
  - pytest

The line is dist: xenialneeded to indicate to Travis the need to use Ubuntu Xenial 16.04 for the organization of the virtual environment. To test Python 3.7 code, Ubuntu Xenial is needed, details about this can be found here .

The section installallows you to install packages used in the development of the project. The team pip install -e .installs our package in the Travis virtual environment. After that, Travis, starting pytest, will be able to find our package.

Step 7. Testing in Travis CI


Commit the changes, submit them to GitHub, do the PR. Travis should start working within seconds.


Travis at work.

This is what Travis does in handling the project.


Actions performed by Travis during project processing

If PR was unsuccessful - Travis will inform about it. Please note that if the pull request is unsuccessful, you can send the changes to the same branch and Travis will automatically start working.

Go to your repository page on the Travis website and look around there. Here you can find a lot of interesting things about assemblies. Probably in the future you will become a frequent guest of this page when you try to understand what caused the failed assembly.

If we assume that everything went well, if the page contains green labels, then the verification and assembly of the project were successful.


The project was completed successfully.

If there are no green or red labels on the page, open the menu More optionsand select the item Requests. If you see red error messages here, analyze them. If you see an error message Build config file is required, it means that Travis cannot find your file in the repository .travis.yml. Correct it and the error will disappear.

Travis sends users emails in cases where the assembly of the project is unsuccessful, and in cases where it is possible to fix it.

Remember that you can send commits to open PR and Travis will automatically restart the project build process.

Now let’s analyze our project for covering the code with tests.

Step 8. Assessing code coverage with tests


The report on code coverage with tests allows you to find out which part of the project code, albeit a small one, has been tested. To create such reports, we will use the pytest-cov package .

Add the requirements_dev.txtfollowing line to the file:

pytest-cov==2.6.1

Run the following command:

pytest --cov=my_project_name

In my case, after the command pytest --cov=notebookcwas executed , the following report was displayed.


Report on code coverage with tests

As it turned out, all the project code is provided with tests. Such indicators are very easy to achieve if the entire project consists of several lines of code.

Now let's talk about a tool that allows you to keep a public history of the state of the project in terms of covering its code with tests.

Step 9. Using Coveralls


The Coveralls project allows you to maintain historical information about code coverage with tests.


Coveralls

In order to take advantage of the capabilities of this project, you need to register on the site https://coveralls.io/ , using your GitHub account information. Then you need to connect the repository.

The file requirements_dev.txtyou need to add a line coveralls==1.6.0. This file, by the way, at this stage of work on the project should look like this:

pip==19.0.3
wheel==0.33.0
twine==1.13.0
pytest==4.3.0
pytest-cov==2.6.1
coveralls==1.6.0

We’ll edit the file .travis.ymlto make it look like this (in your case, the name of your project will be here):

dist: xenial
language: python
python: 3.7.2
install:
  — pip install -r requirements_dev.txt
  — pip install -e .
script:
  — pytest --cov=my_package_name
after_success:
  — coveralls

Now, when Travis will build the project, he will install the necessary packages, run the tests and create a report on the coverage of the code with tests. Then this report will be sent to the Coveralls service.

Commit, submit the code to GitHub, and watch what happens. It might take a few minutes for the code coverage test report to get into Coveralls.


Project processing, test coverage report code

Now, among PR checks, there is also a check performed by Coveralls tools.

On the Coveralls page, you can verify that the project is 100% covered in tests.


Information about code coverage with tests

Now let's equip our project with another useful tool.

Step 10. Working with PyUp


The PyUp.io service allows the developer to find out if the dependencies used by him are outdated and whether they have vulnerabilities. This service automatically executes pull requests aimed at updating the package on GitHub. In order to take advantage of the capabilities of this project, you need to register using a GitHub account, on its website - https://pyup.io/ . When adding a repository, it is recommended to set the update frequency ( Update Schedules) to a value every week. With this approach, if your project has many dependencies, you will not encounter too many pull requests.


Configuring Updates

Here's what the package details look like, some of which are outdated, on the PyUp.io website.


Information about packages

Using this service, you will always know about when the latest versions of the packages you use come out. Knowledge, as they say, is half the victory. And the second half is, obviously, automatic pull-requests for updating dependencies.

Summary


In this article, you learned how to use tools such as Black, pytest, Travis CI, Coveralls, and PyUp when developing Python projects. They help control project dependencies, format and test code, and verify and build projects. We hope you find these tools useful.

Dear readers! What tools do you use when developing Python projects?


Also popular now: