Python from the inside out. Introduction / Buruki's Blog

1. Introduction
2. Objects. Head
3. Objects. Tail
4. Structures of the process

In addition to studying the standard library, it is always interesting, and sometimes useful, to know how the language is arranged from the inside. Andrey Svetlov ( Svetlov ), one of Python developers advise all interested in a series of articles about CPython device. I present to you the translation of the first episode.

My friend once told me: “You know, for some people, the C language is just a set of macros that unfolds in assembler instructions.” It was a long time ago (for know-it-alls: yes, even before the advent of LLVM), but these words I remember well. Maybe when Kernigan and Ritchie look at the C program, they actually see the assembler code? What about Tim Burners-Lee ? Maybe he surfs the Internet in a different way, not like us? And what, after all, did Keanu Reeves see in that creepy green mess? No, really, what the hell did he see there ?! Um ... back to the programs. What does Guido van Rossum see when he reads Python programs?

This post is the first in a series of articles on Python internals. I believe explaining the topic to other people is the best way to understand it. And I really wanted to learn to see and understand the “creepy green mess” behind the Python code. I will mainly write about CPython 3rd version, about bytecode (I'm not a fan of the compilation stage), but maybe I will not ignore much more, which is associated with the execution of Python code of any kind ( Unladen Swallow, Jython, Cython , etc.). For brevity, I write Python , implying CPython , unless otherwise stated. I also mean a POSIX-compatible OS or, if important, Linux, unless otherwise stated. If you are interested in how Python works, I advise you to read this post to the end. You should do this all the more if you want to contribute in CPython. Or you can do this in order to find the mistakes that I made, to laugh at me and leave vicious comments if this is your only way to express your feelings and emotions.

Almost everything that I will write about can be found in the Python source code or in some other good sources (documentation, especially this and this page, separate lectures with PyCon, search in python-devetc.). You can find everything, but I hope that my efforts to combine all the materials into one, which you can subscribe via RSS, will facilitate your adventures. I assume the reader is a little familiar with the C language; with the theory of operating systems; a little with assembler of any architecture; not bad with Python and it feels comfortable on UNIX (for example, it easily installs one of the source codes). Do not worry if you do not have enough experience in all this, but I can not promise easy swimming. If you do not have a customized environment for developing Python, I suggest you go here and follow the necessary steps.

Let's start with what you probably already know. A metaphor of mechanisms seems convenient for me to understand what is happening. In the case of Python, this is not difficult because Python relies on a virtual machine to do what it does (like most interpreted languages). It’s important to correctly understand the term “ virtual machine”": You should think more towards the JVM than VirtualBox (technically, they are essentially the same, but in the real world they are usually shared). It seems easier to understand this term literally - it is a mechanism made up of programs. Your processor is just a complex electronic machine that receives machine code and data as input, has a state (registers), and based on the input data and the current state, it displays new information in memory or on the bus. I see, huh? And CPython is a mechanism assembled from software components that has state and processes instructions (different implementations may use different instructions). This mechanism works in the process where the Python interpreter is located. I like this metaphor with “ mechanisms ”, and I have already describedher in great detail.

Given the above, let's evaluate from a bird's eye what happens when we run this command:

$ python -c 'print("Hello, world!")'

The Python binary is launched, the standard C library is initialized (this happens when almost any process starts), the main function is called (see the sources ./Modules/python.c: mainfrom which it is called ./Modules/main.c:) Py_Main. After some preparatory steps (analysis of arguments, environment variables accounting, assessment of the situation with the standard streams, and so on), is called ./Python/pythonrun.c: Py_Initialize. By and large, this function “creates” and collects the parts necessary to start the CPython machine, and simply the “process” turns into a “process with a Python interpreter inside.” In addition, two very important structures are created: interpreter states and stream states . The built-in module is also initialized.sysand a module that contains all the built-in functions and variables. In the following episodes, these steps will be described in detail.

Having all this, Python crawls in one of several ways, depending on what it was fed to: execute a line (option -c), execute a module (option -m), execute a file (explicitly passed on the command line or passed by the kernel if Python is used as a script interpreter) or REPL starts (this is a special case of executing a file that is an interactive device). In our case, the line will be executed, because we passed the option -c. To execute this line is called ./Python/pythonrun.c: PyRun_SimpleStringFlags. This function creates a namespace.__main__, in which our line of code will be executed (where will it be stored a, if executed $ python -c 'a=1; print(a)'? Correctly, in this space). After creating the space, the string is executed in it (more precisely, it is interpreted). For this to happen, first you need to convert the string to something understandable for the machine.

As I said, I will not focus on the parser and compiler Python. I am not an expert on these areas, I am not very interested in this, and as far as I know, there is no special magic in the Python compiler that goes beyond the university course on compilers. Just go through the tops of these topics a bit and maybe come back a bit later to examine some of the features of CPython behavior (for example, the global operator , which affects the parser). In general, the stagesparsing / compilation PyRun_SimpleStringFlagsare as follows: lexical analysis and the creation of a parse tree , its transformation into an abstract syntax tree (AST), a compilation of AST in the code object using ./Python/ast.c: PyAST_FromNode. Now you can think of the code object as a binary string with which the mechanisms of the Python virtual machine can work - now we are ready for interpretation.

We have almost empty __main__, we have a code object, and we want to execute it. What's next? Everything makes a line from ./Python/pythonrun.c: run_mod:

v = PyEval_EvalCode((PyObject*)co, globals, locals);

The function takes a code object and a namespace globalsand locals(in our case, they are a newly created namespace __main__), creates a frame object and executes it. Back to Py_Initialize, which determines the state of the stream. Each feed stream is represented by a separate state structure, which (among other things) indicates a stack of frames currently executing. After the frame object is created and placed top stack flow condition, it (or more precisely, byte code, to which it points) is satisfied, the operation of operation means is quite long feature ./Python/ceval.c: PyEval_EvalFrameEx.

PyEval_EvalFrameExit accepts a frame, retrieves the operation codes (and operands, if any; we will talk more about this) and executes pieces of C-code corresponding to the operation codes. Let's disassemble a snippet of Python code and see what these “operation codes" look like:

>>> from dis import dis # о! удобная функция для дизассемблирования!
>>> co = compile("spam = eggs - 1", "", "exec")
>>> dis(co)
 1           0 LOAD_NAME                0 (eggs)
             3 LOAD_CONST               0 (1)
             6 BINARY_SUBTRACT     
             7 STORE_NAME               1 (spam)
            10 LOAD_CONST               1 (None)
            13 RETURN_VALUE
>>>

... even without much knowledge, the bytecode is readable enough. We “ eggsload” something with a name (where do we load it from? Where do we load it from?) And load a constant value (1), then “binary subtraction” is performed (what is meant by the word “binary” here? What are operands?), And so Further.

As you might have guessed, the variables are “loaded” from the global and local namespaces that we saw earlier onto the operand stack (do not confuse with the stack of executable frames), exactly where the binary subtraction will pull them out, subtract one from the other and put the result back to the stack. “Binary subtraction” is the subtraction of one operand from another (hence the “binary”, that is, there is no connection with binary numbers).

You can learn the function PyEval_EvalFrameExin the file yourself./Python/ceval.c. It is large enough, and for obvious reasons I will not describe it here in its entirety, but I will show the code that is executed when processing the operation BINARY_SUBTRACT:

TARGET(BINARY_SUBTRACT) {
    PyObject *right = POP();
    PyObject *left = TOP();
    PyObject *diff = PyNumber_Subtract(left, right);
    Py_DECREF(right);
    Py_DECREF(left);
    SET_TOP(diff);
    if (diff == NULL)
        goto error;
    DISPATCH();
}

... pop the first operand, take the second operand from the stack, transfer both operands of the C-function PyNumber_Subtract, make incomprehensible (we will deal with this) to Py_DECREFboth operands, overwrite the top value of the stack with the result of subtraction and make some DISPATCH, if diffnot equal NULL. So. Although we still don’t understand some things, I think that subtracting two numbers in Python at the lowest level is generally understandable. But to reach this point, it took us about one and a half thousand words!

After the frame is completed, it PyRun_SimpleStringFlagsreturns the completion code, the main function performs cleaning (we will pay special attention Py_Finalize), it is de-initialized libc( atexitand so on), and the process ends.

I hope this post has turned out to be quite informative, and in the future we will use it as a foundation when discussing different parts of Python. We still have a lot of terms, which will return: the interpreter, the state of flow, namespace, module, built-in functions and variables of code objects and frames, and those strange words DECREFand DISPATCHfrom the handler BINARY_SUBTRACT. We also have a key “phantom” term, around which we wandered in this article, but which were not named by name - an object . The CPython object system is important for understanding how it all works, and I hope we will discuss it in detail in the next post.

Stay in touch.

Someone was hurt during the translation: meanings, terms, reptiles. Let's make the world a better place together, write about errors in the comments, it’s more reliable.

Join us, come to Buruki ! Together we will learn the wisdom of modern tools and create cool products.