Comparison of BDD Python frameworks: pros and cons / OTUS Blog. Online education

Hello. Already this month , the Python QA Engineer course starts at OTUS . In anticipation of the start of this course, we are sharing with you the translation of interesting material.

Almost all major programming languages ​​have frameworks for BDD testing , and Python is no exception. And in fact, he has several of them! So how do you compare them and which one is the best? Let's try to figure it out.

behave

behave is one of the most popular BDD python frameworks. Despite the fact that it is not officially part of the Cucumber project , its functionality is quite similar to the Cucumber framework.

Sources:

• Python Testing 101: Behave
• Behavior-driven python
• GitHub Behave Project
• GitHub behave-parallel project
• Behav documentation

Logo

Pros:

• Fully supports the Gherkin language;
• The functionality of the environment and fixtures make installation and removal easy;
• There is integration with Django and Flask;
• Popular with BDD pythonists;
• Good documentation and manuals;
• There is support for PyCharm Professional Edition .

Minuses:

• There is no support for parallel execution, this requires a separate behave-parallel framework ;
• behave is a standalone framework;
• Sharing steps between feature files can be troublesome.

pytest-bdd

bdd-pytest - is a plugin for pytest , which allows users to write tests in the form of feature-files Gherkin instead of test functions. Since it is integrated into pytest, it can work with any other pytest plugins, such as pytest-html for creating beautiful reports and pytest-xdist for parallel testing. It also uses pytest fixtures to inject dependencies.

Sources:

• Python Testing 101: pytest-bdd
• Python Testing 101: pytest
• Project pytest-bdd on GitHub
• Documentation for pytest-bdd
• pytest.org

Logo

Pros:

• Fully compatible with pytest and its main plugins;
• You can enjoy all the benefits of the pytest community;
• Fixtures are a great way to manage context between steps;
• Tests can be filtered and run along with other pytest tests;
• Step and hook definitions can be easily shared using conftest.py;
• Tabular data can be better processed for data-driven tests ;
• Excellent online documentation and tutorials;
• There is support for PyCharm Professional Edition .

Minuses:

• Step definition modules must have explicit declarations for feature files (via “@scenario” or the “scenarios” function);
• The scenario plan should be handled differently.

radish

radish is a highlighted BDD framework: it adds new syntax to the Gherkin language. Language features such as script loops, script prerequisites, and constants make Gherkin's radish more programmatic for test cases.

Sources:

• Radish project on GitHub;
• radish-bdd.io
• Radish documentation

Logo

Pros:

• Gherkin language extensions enable testers to write tests better;
• Website, documentation and logo - everything is in place;
• Feature files and step definitions are very clear.

Minuses:

• It is a standalone framework with a limited number of extensions;
• BDD purists may not like the addition of Gherkin syntax.

lettuce

lettuce is another “vegetable” BDD Python framework that has been known for several years. However, his site and code have not been updated for a long time.

Sources:

• The lettuce project on GitHub;
• lettuce.it .

Logo:

Pros:

• Simple code
• Verified and good.

Minuses:

• Not as rich functionality as in other frameworks;
• He seems to have no active ongoing support.

freshen

freshen is one of the very first frameworks for BDD testing in Python. He was a plugin for nose . However, freshen and nose are no longer supported, and their documentation page clearly states the use of other frameworks.

Recommendations

Ни один из приведенных фреймворков не идеален, однако у многих из них есть существенные преимущества. В целом, я бы порекомендовал использовать pytest-bdd, поскольку он имеет все преимущества сильных сторон pytest. Я считаю, что pytest – это один из лучших фреймворков для тестирования на любом языке из-за его лаконичности, фикстур, assert’ов и плагинов. 2018 Python Developers Survey показал, что pytest является самым популярным фреймворком для тестирования и на Python. Несмотря на то, что pytest-bdd не выглядит настолько «отполированным», как behave, я считаю, что некоторые TLC из комьюнити с открытым исходным кодом в силах это исправить.
И еще немного рекомендаций:

• Используйте behave, если вам нужен надежный, чистый опыт разработки и большое комьюнити.
• Use pytest-bdd if you need integration with other plugins that already contain a bunch of pytest tests, or if you need to run tests in parallel.
• Use radish if you want more software control over testing at the Gherkin level.
• Do not use lettuce and freshen.

What is your favorite BDD framework? Please share in the comments!