qrazydraqon June 14, 2012 at 20:09

Python metaclasses

Transfer

As one StackOverflow user said, "using SO is like doing lookups with a hashtable instead of a linked list." We again turn to this wonderful resource, which comes across extremely detailed and understandable answers to a variety of questions.

This time we’ll discuss what metaclasses are, how, where, and why to use them, and why you usually don’t.

Classes as objects

Before you study metaclasses, you need to understand the classes well, and classes in Python are a very specific thing (based on ideas from the Smalltalk language).

In most languages, a class is just a piece of code that describes how to create an object. In general, this is also true for Python:

  >>> class ObjectCreator(object):
  ...       pass
  ... 
  >>> my_object = ObjectCreator()
  >>> print my_object
  <__main__.ObjectCreator object at 0x8974f2c>

But in Python, a class is something more - classes are also objects.

As soon as the keyword is used class, Python executes the command and creates the object . Instruction manual

  >>> class ObjectCreator(object):
  ...       pass
  ...

will create an object with a name in memory ObjectCreator.

This object (class) itself can create objects (instances), therefore it is a class.

However, this is an object, and therefore:

it can be assigned to a variable,
it can be copied,
you can add an attribute to it,
it can be passed to the function as an argument,

Dynamic class creation

Since classes are objects, they can be created on the go, just like any object.

For example, you can create a class in a function using the keyword class:

  >>> def choose_class(name):
  ...     if name == 'foo':
  ...         class Foo(object):
  ...             pass
  ...         return Foo # возвращает класс, а не экземпляр
  ...     else:
  ...         class Bar(object):
  ...             pass
  ...         return Bar
  ...     
  >>> MyClass = choose_class('foo') 
  >>> print MyClass # функция возвращает класс, а не экземпляр
  
  >>> print MyClass() # можно создать экземпляр этого класса
  <__main__.Foo object at 0x89c6d4c>

However, this is not very dynamic, because you still need to write the whole class yourself.

Since classes are objects, they must be generated by something.

When a keyword is used class, Python creates this object automatically. But like most things in Python, there is a way to do this manually.

Remember the function type? A good old function that allows you to determine the type of object:

>>> print type(1)

>>> print type("1")

>>> print type(ObjectCreator)

>>> print type(ObjectCreator())

In fact, a function typehas a completely different application: it can also create classes on the go. typetakes a class description as input and calls the class.

(I know, it's stupid that the same function can be used for two completely different things depending on the arguments passed. This is done for backward compatibility)

typeworks as follows:

  type(<имя класса>, 
       <кортеж родительских классов>, # для наследования, может быть пустым
       <словарь, содержащий атрибуты и их значения>)

For instance,

>>> class MyShinyClass(object):
...       pass

can be created manually as follows:

  >>> MyShinyClass = type('MyShinyClass', (), {}) # возвращает объект-класс
  >>> print MyShinyClass
  
  >>> print MyShinyClass() # создаёт экземпляр класса
  <__main__.MyShinyClass object at 0x8997cec>

You may have noticed that we use “MyShinyClass” both as a class name and as a name for a variable containing a reference to the class. They may be different, but why complicate it?

typeaccepts a dictionary defining class attributes:

>>> class Foo(object):
...       bar = True

can be rewritten as

  >>> Foo = type('Foo', (), {'bar':True})

and use as a regular class

  >>> print Foo
  
  >>> print Foo.bar
  True
  >>> f = Foo()
  >>> print f
  <__main__.Foo object at 0x8a9b84c>
  >>> print f.bar
  True

Of course, you can inherit from it:

  >>>   class FooChild(Foo):
  ...         pass

will turn into

  >>> FooChild = type('FooChild', (Foo,), {})
  >>> print FooChild
  
  >>> print FooChild.bar # bar is inherited from Foo
  True

At some point you will want to add methods to your class. To do this, simply define a function with the desired signature and assign it as an attribute:

>>> def echo_bar(self):
...       print self.bar
... 
>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})
>>> hasattr(Foo, 'echo_bar')
>>> hasattr(FooChild, 'echo_bar')
True
>>> my_foo = FooChild()
>>> my_foo.echo_bar()
True

It’s already clear what I'm getting at: classes are objects in Python and you can create classes on the go.

This is exactly what Python does when a keyword is used class, and he does it with metaclasses.

What is a metaclass (finally)

A metaclass is a “thing” that creates classes.

We create a class to create objects, right? And classes are objects. A metaclass is what creates these very objects. They are classes of classes, you can imagine it as follows:

  MyClass = MetaClass()
  MyObject = MyClass()

We have already seen that it typeallows you to do something like this:

  MyClass = type('MyClass', (), {})

This is because a function typeis actually a metaclass. typethis is the metaclass that Python internally uses to create all classes.

The natural question is: why is it that his name is written in lower case, and not Type?

I suppose this is just to match the strclass for creating string objects, and the intclass for creating integer objects. typeit's just a class for creating class objects.

This can be easily verified using the attribute __class__:

In python, everything (everything!) Is an object. Including numbers, strings, functions and classes - they are all objects and they were all created from a class:

  >>> age = 35
  >>> age.__class__
  
  >>> name = 'bob'
  >>> name.__class__
  
  >>> def foo(): pass
  >>> foo.__class__
  
  >>> class Bar(object): pass
  >>> b = Bar()
  >>> b.__class__

And what is __class__each __class__?

  >>> a.__class__.__class__
  
  >>> age.__class__.__class__
  
  >>> foo.__class__.__class__
  
  >>> b.__class__.__class__

So, a metaclass is just a thing that creates class objects.

If you want, you can call it a “class factory” -

typethis is the built-in metaclass that Python uses, but you can of course create your own.

Attribute `metaclass`

When writing a class, you can add an attribute __metaclass__:

class Foo(object):
  __metaclass__ = something...
  [...]

In this case, Python will use the specified metaclass when creating the class Foo.

Caution, there is subtlety!

Although you are writing class Foo(object), a class object is not yet created in memory.

Python will search __metaclass__in the class definition. If he finds it, he will use it to create the class Foo. If not, he will use it type.

That is, when you write

class Foo(Bar):
  pass

Python does the following:

Does the class have Fooan attribute __metaclass__?

If so, it creates in memory a class object with the name Foo, using what is specified in __metaclass__.

If Python does not find it __metaclass__, it searches __metaclass__in the parent class Barand tries to do the same.

If __metaclass__not located in either parent, Python will search __metaclass__at the module level .

And if he cannot find any at all __metaclass__, he uses typeto create a class object.

Now an important question: what can be put in __metaclass__?

Answer: anything that can create classes.

What creates classes?type or any subclass of it, as well as anything that uses them.

Custom metaclasses

The main purpose of metaclasses is to automatically change the class at the time of creation.

This is usually done for the API when you want to create classes in accordance with the current context.

Imagine a stupid example: you decided that all classes in your module should have attribute names in upper case. There are several ways to do this, but one of them is to set it __metaclass__at the module level.

In this case, all classes of this module will be created using the specified meaclass, and we can only force the metaclass to translate the names of all attributes in upper case.

Fortunately, it __metaclass__can be any object called, not necessarily a formal class (I know something with the word “class” in the name does not have to be a class, what kind of nonsense? However, this is useful).

So we will start with a simple example using a function.

# метаклассу автоматически придёт на вход те же аргументы,
# которые обычно используются в `type`
def upper_attr(future_class_name, future_class_parents, future_class_attr):
  """
    Возвращает объект-класс, имена атрибутов которого
    переведены в верхний регистр
  """
  # берём любой атрибут, не начинающийся с '__'
  attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
  # переводим их в верхний регистр
  uppercase_attr = dict((name.upper(), value) for name, value in attrs)
  # создаём класс с помощью `type`
  return type(future_class_name, future_class_parents, uppercase_attr)
__metaclass__ = upper_attr # это сработает для всех классов в модуле
class Foo(object): 
  # или можно определить __metaclass__ здесь, чтобы сработало только для этого класса
  bar = 'bip'
print hasattr(Foo, 'bar')
# Out: False
print hasattr(Foo, 'BAR')
# Out: True
f = Foo()
print f.BAR
# Out: 'bip'

And now the same thing, only using the real class:

# помним, что `type` это на само деле класс, как `str` и `int`,
# так что от него можно наследовать
class UpperAttrMetaclass(type): 
    # Метод __new__ вызывается перед __init__
    # Этот метод создаёт обхект и возвращает его,
    # в то время как __init__ просто инициализирует объект, переданный в качестве аргумента.
    # Обычно вы не используете __new__, если только не хотите проконтролировать,
    # как объект создаётся
    # В данном случае созданный объект это класс, и мы хотим его настроить,
    # поэтому мы перегружаем __new__.
    # Можно также сделать что-нибудь в __init__, если хочется.
    # В некоторых более продвинутых случаях также перегружается __call__,
    # но этого мы сейчас не увидим.
    def __new__(upperattr_metaclass, future_class_name, 
                future_class_parents, future_class_attr):
        attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)
        return type(future_class_name, future_class_parents, uppercase_attr)

But this is not exactly OOP. We directly call typeand do not overload the __new__parent call . Let's do that:

class UpperAttrMetaclass(type): 
    def __new__(upperattr_metaclass, future_class_name, 
                future_class_parents, future_class_attr):
        attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)
        # используем метод type.__new__
        # базовое ООП, никакой магии
        return type.__new__(upperattr_metaclass, future_class_name, 
                            future_class_parents, uppercase_attr)

You may have noticed an additional argument upperattr_metaclass. There is nothing special about it: a method always gets the current instance as its first argument. Just like you use selfin regular methods.

Of course, the names I used here are so long for clarity, but like that self, there is a convention on naming all of these arguments. So the real metaclass looks something like this:

class UpperAttrMetaclass(type): 
    def __new__(cls, name, bases, dct):
        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)
        return type.__new__(cls, name, bases, uppercase_attr)

You can do even better by using superone that will cause inheritance (since, of course, you can create a metaclass inherited from a metaclass inherited from type):

class UpperAttrMetaclass(type): 
    def __new__(cls, name, bases, dct):
        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)
        return super(UpperAttrMetaclass, cls).__new__(cls, name, bases, uppercase_attr)

That's all. There is nothing more to say about metaclasses.

The reason for the complexity of code using metaclasses is not in the metaclasses themselves. It is that metaclasses are usually used for all sorts of sophisticated things based on introspection, manipulation of inheritance, variables like __dict__the like.

Indeed, metaclasses are especially useful for any "black magic", and, therefore, complex pieces. But they themselves are simple:

intercept class creation
change class
return modified

Why use metaclasses instead of functions?

Since it __metaclass__takes any called object, why would you suddenly use a class if this is obviously more complicated?

There are several reasons for this:

The purpose is clearer. When you see UpperAttrMetaclass(type), you immediately know what will happen next.
You can use OOP. Metaclasses can be inherited from metaclasses, overloading parental methods.
Better structured code. You will not use metaclasses for such simple things as in the example above. Usually this is something complicated. The ability to create multiple methods and group them in one class is very useful to make the code more readable.
You can use __new__, __init__and __call__. Of course, you can usually do everything in __new__, but some are more comfortable using__init__
They are called meta classes , damn it! That must mean something!

Why use metaclasses at all?

Finally, the main question. Why would someone use some obscure (and contributing to errors) feature?

Well, usually you should not use:

Metaclasses are deep magic that 99% of users don't even need to think about. If you think whether you need to use them, you do not need (people who really need them know exactly why they need them, and do not need explanations why).
~ Python Guru Tim Peters

The main use of metaclasses is to create an API. A typical example is Django ORM.

It allows you to write something like this:

  class Person(models.Model):
    name = models.CharField(max_length=30)
    age = models.IntegerField()

However, if you execute the following code:

  guy = Person(name='bob', age='35')
  print guy.age

you get not IntegerField, but int, and the value can be obtained directly from the database.

This is possible because it models.Modeldetermines __metaclass__which one will do some magic and turn the class Personthat we just defined with a simple expression into a complex database binding.

Django makes something complicated look simple by exposing a simple API and using metaclasses to recreate the code from the API and quietly do all the work.

In the end

First, you learned that classes are objects that can instantiate.

In fact, classes are also instances. Instances of metaclasses.

  >>> class Foo(object): pass
  >>> id(Foo)
  142630324

Anything is an object in Python: an instance of a class or an instance of a metaclass.

Except type.

typeis its own metaclass. This cannot be reproduced in pure Python and is done with little cheating at the implementation level.

Secondly, metaclasses are complex. You do not need to use them to simply change classes. This can be done in two different ways:

hands
class decorators

In 99% of cases when you need to change a class, it is better to use these two.

But in 99% of cases you don’t need to change classes at all :-)

Tags:

metaclasses