Mypy extension with plugins

Good afternoon friends. And we continue to increase the intensity of launching new courses and are now happy to announce that classes at the "Web-developer in Python" course will start in late April . In this regard, we traditionally share the translation of useful material. Let's get started.

Python is known to be a dynamic typing language. It's very easy to write DSL-like frameworks that are hard to parse with static type checking tools. Despite this, with mypy's latest functional innovations such as protocols and literal types, as well as with basic support for metaclasses and descriptor support, we can often get exact types, but it’s still difficult to avoid false positives and other negative factors. To solve this problem and avoid the need to customize the type system for each framework, mypy supports a plug-in system . Plugins are modules in Python that provide plugin hooks that mypywill call when checking the types of classes and functions that interact with the library or framework. Thus, it is possible to more accurately distinguish the type of the returned function, which is otherwise extremely difficult to express, or automatically generate some class methods to reflect the effects of the decorator. To learn more about the architecture of the plug-in system and see the full list of features, check out the documentation .



Related Plugins for the Standard Library

Mypy comes with default plugins for implementing basic functions and classes, as well as modules ctypes, contextlib and dataclasses. It also includes plugins for attrs (it has historically been the first third-party plugin written for mypy) These plugins allow mypy to more accurately determine types and correctly check code for type using these library functions. To show this with an example, take a look at a code snippet:

from dataclasses import dataclass
    from typing import Generic, TypeVar
    @dataclass
    class TaggedVector(Generic[T]):
        data: List[T]
        tag: str
    position = TaggedVector([0, 0, 0], 'origin')

Above, get_class_decorator_hook()called when defining a class. This adds auto-generated methods, including __init__(), to the function body. Mypy uses such a constructor to correctly calculate TaggedVector[int]as the type for position. As you can see from the example, plugins work even with generic classes.

Here is another piece of code:

from contextlib import contextmanager
    @contextmanager
    def timer(title: str) -> Iterator[float]:
        ...
    with timer(9000) as tm:
        ...

This get_function_hook()provides an exact return type for the decorator contextmanager, so calls to the decorated function can be checked for compliance with a specific type. Now mypy may recognize the error: the argument for timer()must be a string.

Combining plugins and stubs

In addition to using the dynamic functions of Python, frameworks often run into the problem of having large APIs. Mypy needs stub files for libraries to test the code that uses these libraries (only if the library does not contain built-in annotations, which is not so common). Distributing stubs for large frameworks with typeshed is not a common practice:

• Typeshed has a relatively slow release cycle (shipped with mypy ).
• Incomplete stubs can lead to false calls, which will be extremely difficult to avoid.
• Don't just mix stubs from different typeshed versions .

Stub packages introduced in PEP 561 do the following:

• Developers can release stub packages as often as they want.
• Users who have not chosen to use the package will not see false positives.
• You can safely install arbitrary versions of several different stub packages.

Moreover, pipit allows combining different stubs for libraries and corresponding mypy plugins into one distribution. Stubs for the mypy framework or corresponding plugin can be easily developed and put together into one distribution, which is extremely useful since plugins fill in missing or inaccurate definitions in stubs.

The latest example of such a package is SQLAlchemy stubs and plugin , with the first public release of version 0.1, which was published some time ago on PyPI. Despite the fact that this project is in the early Alpha version, we can safely use it in DropBox to improve type checking. The plugin understands the basic ORM declarations:

from sqlalchemy.ext.declarative import declarative_base
    from sqlalchemy import Column, Integer, String
    Base = declarative_base()
    class User(Base):
        __tablename__ = 'users'
        id = Column(Integer, primary_key=True)
        name = Column(String)

In the code snippet above, the plugin uses get_dynamic_class_hook()to tell mypy that Base is a valid base class, even if it doesn't look like that. It get_base_class_hook()is then called to define User, and adds several automatically generated attributes. Next, we create an instance of the model:

user = User(id=42, name=42)

Called get_function_hook(), so mypy may indicate an error: a type value is received integerinstead of the username.

Stubs are defined Columnas a generic descriptor, so that the model attributes get the correct types:

id_col = User.id  # Inferred type is "Column[int]"
name = user.name  # Inferred type is "Optional[str]"

We welcome PRs that add more precise types to stubs (progress for core modules is tracked here ).

Here are a few pitfalls we discovered while working on plugs:

• Use __getattr__()to avoid false positives in the early stages when stubs are not completed (this prevents mypy errors if module attributes are missing). You can also use this in files __init__.pyif any submodules are missing.
• Descriptors often help with more accurate type definitions for custom attribute access (as in the Column example we reviewed above). Using descriptors is fine even if the actual implementation of the runtime uses a more complex mechanism, including a metaclass, for example.
• Without hesitation, declare the framework classes as generalized. Despite the fact that they are not such at runtime, this technique allows you to more accurately determine the type of some elements of the framework, while runtime errors can be easily circumvented . (We hope that frameworks will gradually add built-in support for generic types, explicitly inheriting the corresponding classes from typing.Generic.)

Recently released mypy plugins There are

already several plugins available for the popular Python frameworks. Apart from the SQLAlchemy plugin mentioned above , other noteworthy sample packages with stubs and the built-in mypy plugin include stubs for the Django and Zope interfaces . Active work is underway on these projects.

Installing and connecting stub and plugin packages

Use pip to install a plugin package for mypy and / or stub into a virtual environment where mypy is already installed :

 $ pip install sqlalchemy-stubs

Mypy will automatically detect installed stubs. To connect installed plugins, include them directly in mypy.ini (or in the user configuration file):

[mypy]
plugins = sqlmypy, mypy_django_plugin.main

Developing mypy plugins and writing stubs

If you want to develop a package of stubs and plugins for the framework you use, we can use the sqlalchemy-stubs repository as a template. It includes a file setup.py, infrastructure testing using data-driven tests, and an example plug-in class with a set of hooks for the plug-in (plugin hooks). We recommend using stubgen to automatically generate the stubs that come with mypy to start using them. Stubgenimproved somewhat in mypy 0.670.

Check out the documentation if you want to know more about the mypy plugin system. You can also search the Internet for the source codes of the plugins discussed in the article. If you have questions, you can ask them here .

April 15 will be a free open webinar on the course, which will be held by one of the organizers of the Moscow Python community - Vladimir Filonov , sign up, it will be interesting. And now we are waiting for your comments on the translated material.