Using metaclasses in Python

Some metaprogramming tools are not used as often in daily
work as usual in OOP classes or the same decorators. To understand the goals of
introducing such tools into the language, specific examples of industrial
applications are required , some of which are given below.

Introduction to Metaclasses

So, classic OOP implies only classes and objects.
The class is the template for the object; when declaring a class, all the mechanics
of each concrete “embodiment” are indicated : the data encapsulated
in the object and the methods for working with this data are specified .


Python expands the classical paradigm, and the classes themselves in it also become
equal objects that can be changed, assigned to a variable, and
passed to functions. But if the class is an object, then to which class does it correspond?
By default, this class (metaclass) is called type.

You can inherit from a metaclass to get a new metaclass, which, in
turn, can be used to define new classes. Thus,
a new “dimension” of inheritance appears, added to the
class inheritance hierarchy : metaclass -> class -> object.

Simple example

Suppose we are tired of setting attributes in the __init __ constructor (self, * args,
** kwargs). I would like to speed up this process so that it is
possible to set attributes directly when creating an object of the class.
This will not work with a regular class:

  >>> class Man (object):
  >>> pass
  >>> me = Man (height = 180, weight = 80)
  Traceback (most recent call last):
  File "", line 20, in 
      TypeError: object .__ new __ () takes no parameters

An object is constructed by calling the class with the "()" operator. We will inherit from
type a metaclass that overrides this statement:

  >>> class AttributeInitType (type):
  >>> def __call __ (self, * args, ** kwargs):
  >>> "" "Calling the class creates a new object." ""
  >>> # First things first, create the object itself ...
  >>> obj = type .__ call __ (self, * args)
  >>> # ... and add to it the arguments passed in the call as attributes.
  >>> for name in kwargs:
  >>> setattr (obj, name, kwargs [name])
  >>> # return the finished object
  >>> return obj

Now create a class using the new metaclass:

  >>> class Man (object):
  >>> __metaclass__ = AttributeInitType

Voila:

  >>> me = Man (height = 180, weigth = 80)
  >>> print me.height
  180

Language Extension (abstract classes)

The Python core is relatively small and simple, the set of built-in tools is
small, which allows developers to quickly learn the language.


At the same time, programmers involved in creating, for example, frameworks and
related special sub-languages ​​(Domain Specific Languages) are provided with
quite flexible tools.

Abstract classes (or their slightly different form - interfaces) are a common
and popular method among programmers for defining the interface part of a
class. Typically, such concepts are embedded in the core of the language (as in Java or C ++), while
Python allows you to elegantly and easily implement them using your own means, in
particular, using metaclasses and decorators.

Consider the operation of the abc library from the implementation proposal for the standard library.

abc

Using abstract classes is very easy. Let's create an abstract base class
with a virtual method and try to create an inheritor class without defining this method:

>>> from abc import ABCMeta, abstractmethod
>>> class A (object):
>>> __metaclass __ = ABCMeta
>>> @abstractmethod
>>> def foo (self): pass
>>> 
>>> A ()  
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Can't instantiate abstract class A with abstract methods foo

It didn’t work out. Now we define the desired method:

  
>>> class C (A):
>>> def foo (self): print (42)
>>> C

>>> a = C ()
>>> a.foo ()
42

We learn how this is implemented in the metaclass (omitting some other features of
the abc module) ABCMeta:

>>> class ABCMeta (type):
>>> def __new __ (mcls, name, bases, namespace):
>>> bases = _fix_bases (bases)
>>> cls = super (ABCMeta, mcls) .__ new __ (mcls, name, bases, namespace)
>>> # We will find many (set) names of abstract methods among proper
>>> # methods and methods of ancestors
>>> abstracts = set (name
>>> for name, value in namespace.items ()
>>> if getattr (value, "__isabstractmethod__", False))
>>> for base in bases:
>>> for name in getattr (base, "__abstractmethods__", set ()):
>>> value = getattr (cls, name, None)
>>> if getattr (value, "__isabstractmethod__", False):
>>> abstracts.add (name)
>>> cls .__ abstractmethods__ = frozenset (abstracts)
>>> return cls

The _fix_bases method adds the hidden _Abstract class to the number of ancestors of the
abstract class. _Abstract itself checks to see if there is anything left in the
set __abstractmethods__; if left, throws an exception.

>>> class _Abstract (object):
>>> def __new __ (cls, * args, ** kwds):
>>> am = cls .__ dict __. get ("__ abstractmethods__")
>>> if am:
>>> raise TypeError ("can't instantiate abstract class% s"
>>> "with abstract methods% s"%
>>> (cls .__ name__, "," .join (sorted (am))))
>>> return super (_Abstract, cls) .__ new __ (cls, * args, ** kwds)
>>>
>>> def _fix_bases (bases):
>>> for base in bases:
>>> if issubclass (base, _Abstract):
>>> # _Abstract is already among the ancestors
>>> return bases
>>> if object in bases:
>>> # Replace object with _Abstract if the class directly inherits from object
>>> # and not listed among other ancestors
>>> return tuple ([_ Abstract if base is object else base
>>> for base in bases])
>>> # Add _Abstract to the end otherwise
>>> return bases + (_Abstract,)

Each abstract class is stored according to the “frozen” set (frozenset) of
abstract methods; that is, those methods (function objects) that have
the __isabstractmethod__ attribute set by the corresponding decorator:

>>> def abstractmethod (funcobj):
>>> funcobj .__ isabstractmethod__ = True
>>> return funcobj

So, the abstract method gets the __isabstractmethod__ attribute when assigning it a
decorator. Attributes after inheriting from an abstract class are collected in the
set "__abstractmethods__" of the inheriting class. If the set is not empty, and the
programmer tries to create an object of the class, a
TypeError exception will be thrown with a list of undefined methods.

Conclusion

Simply? Simply. Is the language expanded? Extended. Comments, as they say, are unnecessary.

DSL in Django

One of the most advanced examples of DSL is the Django ORM mechanism using the Model class and
ModelBase metaclass as an example. Specifically, the connection with the database is not interesting here, it
makes sense to concentrate on creating an instance of the inheritor class of the Model class.

Most of the next subsection is a detailed analysis of
ModelBase code . For readers who do not need more details, just read the output
at the end of the Django section.

Parsing the ModelBase Metaclass

All the mechanics of working of the ModelBase metaclass are concentrated in the place of
overriding the __new__ method, called immediately before creating an
instance of the model class:

  >>> class ModelBase (type):
  >>> "" "
  >>> Metaclass for all models.
  >>> "" "
  >>> def __new __ (cls, name, bases, attrs):
  >>> super_new = super (ModelBase, cls) .__ new__
  >>> parents = [b for b in bases if isinstance (b, ModelBase)]
  >>> if not parents:
  >>> # If this isn't a subclass of Model, don't do anything special.
  >>> return super_new (cls, name, bases, attrs)

At the very beginning of the method, an instance of the class is simply created and, if this class does not
inherit from Model, it is simply returned.

All specific model class options are collected in the attribute of the _meta class, which
can be created from scratch, inherited from the ancestor, or adjusted in the
local Meta class:

  >>> #Class creation
  >>> module = attrs.pop ('__ module__')
  >>> new_class = super_new (cls, name, bases, {'__module__': module})
  >>> attr_meta = attrs.pop ('Meta', None)
  >>> abstract = getattr (attr_meta, 'abstract', False)
  >>> if not attr_meta:
  >>> meta = getattr (new_class, 'Meta', None)
  >>> else:
  >>> meta = attr_meta
  >>> base_meta = getattr (new_class, '_meta', None)

In addition, we see that the class may be abstract, not corresponding
to any table in the database.

The moment of truth in the process of creating a model class occurs when the
default parameters are entered into it :

  >>> new_class.add_to_class ('_ meta', Options (meta, ** kwargs))

add_to_class either calls the contribute_to_class method of the argument, or, if
not, simply adds the named attribute to the class.

The Options class in its contribute_to_class makes the _meta attribute a reference to
itself and collects various parameters in it, such as the name of the database table
, the list of model fields, the list of virtual model fields, access rights, and
others. He also checks links with other models for the uniqueness
of field names in the database.

Then, in the __new__ method, named
exceptions are added to the non-abstract class :

  >>> if not abstract:
  >>> new_class.add_to_class ('DoesNotExist',
  >>> subclass_exception ('DoesNotExist', ObjectDoesNotExist, module))
  >>> new_class.add_to_class ('MultipleObjectsReturned',
  >>> subclass_exception ('MultipleObjectsReturned',
  >>> MultipleObjectsReturned, module))

If the parent class is not abstract, and the parameters are not set explicitly in the local
Meta class, then we inherit the ordering and get_latest_by parameters:

  >>> if base_meta and not base_meta.abstract:
  >>> if not hasattr (meta, 'ordering'):
  >>> new_class._meta.ordering = base_meta.ordering
  >>> if not hasattr (meta, 'get_latest_by'):
  >>> new_class._meta.get_latest_by = base_meta.get_latest_by

The default manager must be zero. If such a model already exists, we complete the processing by returning this model:

  >>> if getattr (new_class, '_default_manager', None):
  >>> new_class._default_manager = None
  >>>        
  >>> m = get_model (new_class._meta.app_label, name, False)
  >>> if m is not None:
  >>> return m

Nothing special, the attributes with which it was
created are simply added to the model class :

  >>> for obj_name, obj in attrs.items ():
  >>> new_class.add_to_class (obj_name, obj)

Now you need to go through the fields of the model and find one-to-one relationships
that will be used just below:

  >>> # Do the appropriate setup for any model parents.
  >>> o2o_map = dict ([(f.rel.to, f) for f in new_class._meta.local_fields
  >>> if isinstance (f, OneToOneField)])

Pass through the ancestors of the model to inherit various fields, discarding those
that are not inheritors of the Model. The following comments are translated, which are
enough to understand what is happening:

  >>> for base in parents:
  >>> if not hasattr (base, '_meta'):
  >>> # Models without _meta are not valid and are of no interest
  >>> continue
  >>>
  >>> # All fields of an arbitrary type for this model
  >>> new_fields = new_class._meta.local_fields + \
  >>> new_class._meta.local_many_to_many + \
  >>> new_class._meta.virtual_fields
  >>> field_names = set ([f.name for f in new_fields])
  >>>
  >>> if not base._meta.abstract:
  >>> # We process "concrete" classes ...
  >>> if base in o2o_map:
  >>> field = o2o_map [base]
  >>> field.primary_key = True
  >>> new_class._meta.setup_pk (field)
  >>> else:
  >>> attr_name = '% s_ptr'% base._meta.module_name
  >>> field = OneToOneField (base, name = attr_name,
  >>> auto_created = True, parent_link = True)
  >>> new_class.add_to_class (attr_name, field)
  >>> new_class._meta.parents [base] = field
  >>>
  >>> else:
  >>> # .. and abstract.
  >>>
  >>> # Check for name collisions between classes,
  >>> # declared in this class and in the abstract ancestor
  >>> parent_fields = base._meta.local_fields + base._meta.local_many_to_many
  >>> for field in parent_fields:
  >>> if field.name in field_names:
  >>> raise FieldError ('Local field% r in class% r clashes' \
  >>> 'with field of similar name from' \
  >>> 'abstract base class% r'% \
  >>> (field.name, name, base .__ name__))
  >>> new_class.add_to_class (field.name, copy.deepcopy (field))
  >>>
  >>> # All non-abstract parents are transferred to the heir
  >>> new_class._meta.parents.update (base._meta.parents)
  >>>
  >>> # Base Managers inherit from abstract classes
  >>> base_managers = base._meta.abstract_managers
  >>> base_managers.sort ()
  >>> for _, mgr_name, manager in base_managers:
  >>> val = getattr (new_class, mgr_name, None)
  >>> if not val or val is manager:
  >>> new_manager = manager._copy_to_model (new_class)
  >>> new_class.add_to_class (mgr_name, new_manager)
  >>>
  >>> # Virtual fields (like GenericForeignKey) we take from the parent
  >>> for field in base._meta.virtual_fields:
  >>> if base._meta.abstract and field.name in field_names:
  >>> raise FieldError ('Local field% r in class% r clashes' \
  >>> 'with field of similar name from' \
  >>> 'abstract base class% r'% \
  >>> (field.name, name, base .__ name__))
  >>> new_class.add_to_class (field.name, copy.deepcopy (field))
  >>>

Abstract model classes are not registered anywhere:

  >>> if abstract:
  >>> #Abstract models cannot be instantiated and do not appear
  >>> # in the list of models for the application, so they turn around a little differently than
  >>> #normal models
  >>> attr_meta.abstract = False
  >>> new_class.Meta = attr_meta
  >>> return new_class

Normal ones are registered and returned already from the list of registered
model classes:

  >>> new_class._prepare ()
  >>> register_models (new_class._meta.app_label, new_class)
  >>> return get_model (new_class._meta.app_label, name, False)

Conclusion

So to summarize. Why are metaclasses needed?

1) The model class must have a set of required parameters (table name, name of the
django application, list of fields, relations with other models and many others) in
the _meta attribute, which are defined when creating each class that inherits
from Model.

2) These parameters are complexly inherited from ordinary and abstract
ancestor classes, which is ugly to lay in the class itself.

3) There is an opportunity to hide what is happening from a programmer using a
framework.

Notice

1) If you do not explicitly specify the inheritance of the class from object, the class uses the
metaclass specified in the global variable __metaclass__, which can sometimes
be convenient when reusing your own metaclass within the
same module. The simple example at the beginning of the note can be redone
as follows:

  class AttributeInitType (type):
      def __call __ (self, * args, ** kwargs):
      obj = type .__ call __ (self, * args)
      for name in kwargs:
      setattr (obj, name, kwargs [name])
      return obj
  __metaclass__ = AttributeInitType
  class Man:
      pass
  me = Man (height = 180, weigth = 80)
  print me.height
  The following will be output to the standard stream:
  180

2) There is such a pythonic supergur, Tim Peters. He said very well about the
use of metaclasses and similar tools from the category of python black magic:

    Metaclasses are deeper magic than 99% of users should ever worry
    about. If you wonder whether you need them, you don't (the
    people who actually need them know with certainty that they need
    them, and don't need an explanation about why).

In Russian, it sounds something like this:

    Metaclasses are superfluous for most users. If at all you wonder
    the question is whether they are needed, then they are definitely not needed. Only people use them,
    who know exactly what they are doing and do not need explanation.

The moral is simple: do not be smart. Metaclasses in most cases are superfluous. The pythonist should be guided by the principle of least surprise;
It’s not worth changing the classical OOP work scheme just for the sake of narcissism.

References based on

English Wikipedia - a simple example
PEP-3119 was borrowed from here -
abstract classes in their full version are described here .
Movie
in English , a detailed conversation about metaclasses in Python with examples of
use. There, through the links, you can find the article itself with examples, very
instructive.