Python: metaprogramming in production. Part two

    We continue to talk about metaprogramming in Python. When used properly, it allows you to quickly and elegantly implement complex design patterns. In the last part of this article, we showed how metaclasses can be used to change the attributes of instances and classes.



    Now let's see how you can change the method calls. You can learn more about metaprogramming features in the Advanced Python course .


    Debugging and Call Tracing


    As you already understood, with the help of the metaclass any class can be transformed beyond recognition. For example, replace all class methods with others or apply an arbitrary decorator to each method. This idea can be used to debug application performance.


    The following metaclass measures the execution time of each method in the class and its instances, as well as the creation time of the instance itself:


    from contextlib import contextmanager  
import logging  
import time  
import wrapt  
@contextmanager  deftiming_context(operation_name):"""Этот контекст менеджер замеряет время выполнения произвольной операции"""
    start_time = time.time()  
    try:  
        yieldfinally:  
        logging.info('Operation "%s" completed in %0.2f seconds', 
                          operation_name, time.time() - start_time)  
@wrapt.decorator  deftiming(func, instance, args, kwargs):"""
    Замеряет время выполнения произвольной фукнции или метода.
    Здесь мы используем библиотеку https://wrapt.readthedocs.io/en/latest/
    чтобы безболезненно декорировать методы класса и статические методы
    """with timing_context(func.__name__):  
        return func(*args, **kwargs)  
classDebugMeta(type):def__new__(mcs, name, bases, attrs):for attr, method in attrs.items():  
            ifnot attr.startswith('_'):  
                # оборачиваем все методы декоратором            
                attrs[attr] = timing(method)  
        return super().__new__(mcs, name, bases, attrs)  
    def__call__(cls, *args, **kwargs):with timing_context(f'{cls.__name__} instance creation'):  
            # замеряем время выполнения создания экземпляраreturn super().__call__(*args, **kwargs)

    Let's look at debugging in action:


    classUser(metaclass=DebugMeta):def__init__(self, name):  
        self.name = name  
        time.sleep(.7)  
    deflogin(self):  
        time.sleep(1)  
    deflogout(self):  
        time.sleep(2)  
    @classmethod  defcreate(cls):  
        time.sleep(.5)  
user = User('Michael')  
user.login()  
user.logout()  
user.create()
# Вывод логгера
INFO:__main__:Operation "User instance creation" completed in0.70 seconds
INFO:__main__:Operation "login" completed in1.00 seconds
INFO:__main__:Operation "logout" completed in2.00 seconds
INFO:__main__:Operation "create" completed in0.50 seconds

    Try to independently expand DebugMetaand log the signature of methods and their stack-trace.


    The loner pattern and the prohibition of inheritance


    And now let us turn to the exotic cases of using metaclasses in Python projects.


    Surely many of you use the usual Python module to implement the loner design pattern (aka Singleton), because it is much more convenient and faster than writing the corresponding metaclass. However, let's write one of its implementations for the sake of academic interest:


    classSingleton(type):  
    instance = Nonedef__call__(cls, *args, **kwargs):if cls.instance isNone:  
            cls.instance = super().__call__(*args, **kwargs)  
     return cls.instance  
classUser(metaclass=Singleton):def__init__(self, name):  
        self.name = name  
    def__repr__(self):returnf'<User: {self.name}>'
u1 = User('Pavel')  
# Начиная с этого момента все пользователи будут Павлами
u2 = User('Stepan')
>>> id(u1) == id(u2)
True>>> u2
<User: Pavel>
>>> User.instance
<User: Pavel>
# Как тебе такое, Илон?>>> u1.instance.instance.instance.instance
<User: Pavel>

    This implementation has an interesting nuance - since the class constructor is not invoked a second time, you can make a mistake and not pass the desired parameter there, and nothing happens at runtime if the instance has already been created. For example:


    >>> User('Roman')
<User: Roman>
>>> User('Alexey', 'Petrovich', 66)  # конструктор не принимает столько параметров!
<User: Roman>
# Но если бы конструктор User до этого момента еще не вызывался# мы бы получили TypeError!

    And now let's take a look at an even more exotic option: a ban on inheriting from a certain class.


    classFinalMeta(type):def__new__(mcs, name, bases, attrs):for cls in bases:  
            if isinstance(cls, FinalMeta):  
                raise TypeError(f"Can't inherit {name} class from 
                                final {cls.__name__}") 
        return super().__new__(mcs, name, bases, attrs)  
classA(metaclass=FinalMeta):"""От меня нельзя наследоваться!"""passclassB(A):pass# TypeError: Can't inherit B class from final A# Ну я же говорил!

    Parametrization of metaclasses


    In the previous examples, we used metaclasses to customize the creation of classes, but you can go even further and start parameterizing the behavior of the metaclasses.


    For example, you can metaclasspass a function into a parameter when declaring a class and return different instances of metaclasses from it, depending on some conditions, for example:


    defget_meta(name, bases, attrs):if SOME_SETTING:
        return MetaClass1(name, bases, attrs)
    else:
        return MetaClass2(name, bases, attrs)
classA(metaclass=get_meta):pass

    But a more interesting example is the use of extra_kwargsparameters when declaring classes. Suppose you want to change the behavior of certain methods in a class with the help of a metaclass, and for each class these methods can be called differently. What to do? And that's what


    # Параметризуем наш `DebugMeta` метакласс из примера вышеclassDebugMetaParametrized(type):def__new__(mcs, name, bases, attrs, **extra_kwargs):  
        debug_methods = extra_kwargs.get('debug_methods', ())  
        for attr, value in attrs.items():  
            # Замеряем время исполнения только для методов, имена которых  # переданы в параметре `debug_methods`:  if attr in debug_methods:  
                attrs[attr] = timing(value)  
        return super().__new__(mcs, name, bases, attrs)
classUser(metaclass=DebugMetaParametrized, debug_methods=('login', 'create')):
    ...
user = User('Oleg')  
user.login()
# Метод "logout" залогирован не будет. 
user.logout()
user.create()

    In my opinion, it turned out very elegant! You can think up quite a lot of patterns for using such parameterization, but remember the main rule - everything is good in moderation.


    Examples of using the method __prepare__


    Finally, I will tell you about the possible use of the method __prepare__. As mentioned above, this method should return a dictionary object that the interpreter fills at the time of parsing the body of the class, for example, if it __prepare__returns an object d = dict(), then when reading the next class:


    classA:
    x = 12
    y = 'abc'
    z = {1: 2}

    The interpreter will perform the following operations:


    d['x'] = 12
d['y'] = 'abc'
d['z'] = {1: 2}

    There are several possible uses for this feature. They are all of different utility levels, so:


    1. In versions of Python = <3.5, if we needed to preserve the order of declaring methods in the class, we could return collections.OrderedDictfrom the method __prepare__, in versions of older, the built-in dictionaries already preserve the order of adding keys, so there is OrderedDictno need for it.
    2. The standard library module enumuses a custom dict-like object to determine when a class attribute is duplicated during the declaration. The code can be found here .
    3. Not a production-ready code at all, but a very good example is support for parametric polymorphism .

    For example, consider the following class with three implementations of the same polymorphic method:


    classTerminator:defterminate(self, x: int):  
        print(f'Terminating INTEGER {x}')  
    defterminate(self, x: str):  
        print(f'Terminating STRING {x}')  
    defterminate(self, x: dict):  
        print(f'Terminating DICTIONARY {x}')  
t1000 = Terminator()  
t1000.terminate(10)  
t1000.terminate('Hello, world!')  
t1000.terminate({'hello': 'world'})
# Вывод
Terminating DICTIONARY 10
Terminating DICTIONARY Hello, world!
Terminating DICTIONARY {'hello': 'world'}

    Obviously, the last declared method terminateoverwrites the implementation of the first two, and we need the method to be chosen depending on the type of the argument passed. To achieve this, we program a couple of additional wrapper objects:


    classPolyDict(dict):"""  
    Словарь, который при сохранении одного и того же ключа 
    оборачивает все его значения в один PolyMethod.  
    """def__setitem__(self, key: str, func):ifnot key.startswith('_'):  
            if key notin self:  
                super().__setitem__(key, PolyMethod())  
            self[key].add_implementation(func)
            returnNonereturn super().__setitem__(key, func)
classPolyMethod:"""  
    Обертка для полиморфного метода, которая хранит связь между типом аргумента
    и реализацией метода для данного типа. Для данного объекта мы реализуем 
    протокол дескриптора, чтобы поддержать полиморфизм для всех типов методов:
    instance method, staticmethod, classmethod.
    """def__init__(self):  
        self.implementations = {}  
        self.instance = None  
        self.cls = Nonedef__get__(self, instance, cls):  
        self.instance = instance  
        self.cls = cls  
        return self  
    def_get_callable_func(self, impl):# "достаем" функцию classmethod/staticmethodreturn getattr(impl, '__func__', impl)
    def__call__(self, arg):
        impl = self.implementations[type(arg)]
        callable_func = self._get_callable_func(impl)
        if isinstance(impl, staticmethod):
            return callable_func(arg)
        elif self.cls and isinstance(impl, classmethod):
            return callable_func(self.cls, arg)
        else:
            return callable_func(self.instance, arg)
    defadd_implementation(self, func):
        callable_func = self._get_callable_func(func)
        # расчитываем на то, что метод принимает только 1 параметр
        arg_name, arg_type = list(callable_func.__annotations__.items())[0]
        self.implementations[arg_type] = func

    The most interesting thing in the code above is an object PolyMethodthat stores the registry with implementations of the same method depending on the type of argument passed to this method. PolyDictWe will return the object from the method __prepare__and thereby save different implementations of the methods with the same name terminate. The important point is that when the class body is read and the object is created, the attrsinterpreter places so-called unboundfunctions there, these functions do not yet know which class or instance they will be called. We had to implement a protocol descriptor to determine the context of functions during a call and pass the first parameter or selfeither clsor transmit nothing if called staticmethod.


    As a result, we will see the following magic:


    classPolyMeta(type):    @classmethoddef__prepare__(mcs, name, bases):return PolyDict()  
classTerminator(metaclass=PolyMeta):  
   ...  
t1000 = Terminator()  
t1000.terminate(10)  
t1000.terminate('Hello, world!')  
t1000.terminate({'hello': 'world'})
# Вывод
Terminating INTEGER 10
Terminating STRING Hello, world!
Terminating DICTIONARY {'hello': 'world'}
>>> t1000.terminate
<__main__.PolyMethod object at 0xdeadcafe>

    If you know any other interesting uses of the method __prepare__, please write in the comments.


    Conclusion


    Metaprogramming is one of many topics that I have been telling on Advanced Python intensive . As part of the course, I will also explain how to effectively use the principles of SOLID and GRASP in the development of large projects in Python, design the architecture of applications and write high-performance and high-quality code. I would be glad to see you in the walls of the Binary District!

    Tags:

    Also popular now: