KPHP and engines release

    Quite often, speaking at various conferences, we shared the desire to release KittenPHP under an open license, in accordance with the tradition laid down by large IT companies such as Google and Facebook.

    This event was postponed several times due to the fear that we would not have enough time and energy to interact with the opensource community, but in the end the treasured day came, and the code of KPHP and some other tools used inside the project was made publicly available.

    In this regard, under the cut you will find a more detailed story about the internal structure of VKontakte and those tools that are now available to the open source community.

    Source codes were laid out under the GNU licenses (GPL and LGPL). These licenses are ideologically close to us, since when creating these libraries we often used tools licensed specifically by GNU.


    The source code for VKontakte is being developed in a PHP-like language called KittenPHP or KPHP for short. This code is translated in C ++ by a special translator of the same name. After that, the generated C ++ code is automatically compiled with gcc, resulting in a binary ready to run. This binary is a web server that accepts http requests and generates pages.
    In order to speed up the development process, KPHP compiles various project files separately, and then links. In subsequent compilations, only modified files are processed, or, in the case of large files, only parts of them.

    KPHP is a minimalistic language designed to provide a very high speed of work, without compromising on the convenience and speed of development. In this regard, KPHP does not support all the features of PHP, in particular, it does not have OOP, with the exception of some objects of the standard library. In addition, eval and related things, such as regular expressions with the 'e' modifier, are not supported (instead, the preg_replace_callback function is suggested). Also, functions for working with specific elements of arrays are not supported: first, end, next, prev, current, reset, key; to replace them, getValueByPos and getKeyByPos functions are implemented.
    The rejection of support for a large amount of functionality allowed KPHP to become incredibly fast compared to other tools for web development.
    As an example, we compared it with the Facebook developed HipHop VM and got the following results:
    simple0.000 0.007 0.137
    simplecall 0.000 0.004 0.174
    simpleucall 0.007 0.008 0.178
    simpleudcall 0.007 0.009 0.181
    mandel 0.010 0.066 0.392
    mandel2 0.011 0.074 0.355
    ackermann (7) 0.001 0.011 0.189
    ary (50,000) 0.003 0.008 0.024
    ary2 (50,000) 0.003 0.010 0.022
    ary3 (2000) 0.011 0.077 0.191
    fibo (30) 0.003 0.019 0.481
    hash1 (50,000) 0.018 0.034 0.044
    hash2 (500) 0.011 0.021 0.039
    heapsort (20,000) 0.012 0.040 0.101
    matrix (20) 0.007 0.021 0.121
    nestedloop (12) 0.000 0.012 0.235
    sieve (30) 0.013 0.016 0.114
    strcat (200000) 0.002 0.005 0.014

    The test code is available at:

    From a development point of view, KPHP is sufficiently compatible with PHP so that you can use regular PHP to quickly test the written code, and only compile the code before the final testing and rolling out the project. To support functions implemented in KPHP, but not available in regular PHP, a special library was added that extends the capabilities of PHP.

    In addition, KittenPHP is a good static PHP code analyzer that points out possible errors. For example, in the process of transferring VKontakte to it a year ago, more than 20 serious bugs were found.

    Together with the compiler under an open license, the developers laid out a set of engines that perfectly complement KPHP, but can be used separately from it. For the first time, we announced these libraries to the open source community at Highload 2010 , so we apologize for the rather long waiting period.

    PMemcached (“Persistent Memcached”)

    Reliable key-value storage that allows you to store data without a time limit. According to the MC protocol, the engine works identically with Memcache, except that after a reboot all the data remains.
    In addition to its basic functions, when you enable the corresponding option in the configuration, pmemcached allows you to immediately receive groups of records whose key prefix matches the one specified in the request.


    This engine allows you to store and retrieve various lists of data.
    One copy of the engine can store a set of lists. Each list must have an identifier (int) by which to work with this list.
    Each list can have an unlimited number of items. Each element must also have an identifier (int), a value (int), a flag (int) and can store arbitrary 256 characters of text.
    In addition to receiving lists, it is possible to receive sub-lists by filtering by flags and sorting by values.



    Modification of the Lists engine, which allows the use of keys and record identifiers, consisting not of one number (int), but of the number of numbers (int) predefined in the engine configuration. For example, this allows you to create lists, the key of which is formed from the user ID and the record ID on its wall.



    Designed to search for data on the site. Any textual information can be indexed in the engine with a specific identifier, and subsequently found by the words in the text. The identifiers specified during indexing will be returned in the search results.
    Search supports arbitrary parameters for searching by criteria, and special parameters for various sortings. The engine also allows for complex groupings and intersections.



    The engine is designed to store user data - photos, videos, audio, documents. Thanks to storing different content in one file and indexing in offsets memory, Storage copes with this task better than using the classical approach of storing individual files in a file system.



    The Texts engine allows you to store various text data arrays. Initially, it was developed for VK's personal messaging system, but was later reused for walls and for comments.
    In addition to storing texts, the engine supports various groupings of lists with texts and text search. Thanks to him, an instant search is available in the entire personal correspondence of the user, no matter how big it is.
    Also, an HTTP server is built into this engine, which implements a long poll for receiving updates from the client side. However, later a separate queue engine was created for this purpose, which is described below.



    Hints solves two important tasks:
    1) Designed to search for user objects by word prefixes, used in quick searches on the site.
    2) Allows you to generate ratings of objects with which you can sort the lists of objects by the degree of interest in them with the user. For example, the VKontakte friends list works this way.



    Queue allows you to organize communication between the client and server sides in real time. The client connects to the Queue server assigned to it and receives updates from it, and the server can send the corresponding event to the client at any time. Thanks to the use of channels that the client can subscribe to when connecting to Queue, the engine can be used to transfer data one to many, for example, when the user has a news page open, he subscribes to the queue for events of all his friends, groups and subscriptions. When someone from this list publishes a record, he also writes it to the corresponding queue subscription, and each subscribed user receives information about this on the client, after which the latter can display the received data.

    In addition to the above, you can find in the repository a number of other, not so universal, but no less interesting tools, the documentation for which you will find here .


    By publishing these developments, we are returning the debt to the open-source community, to which we owe a lot. 

    We hope that now they will help the projects currently under development, as MySQL, Memcache, nginx and PHP helped create VKontakte in due time.

    You can see the source code of the engines and KPHP in the repository on github:
    Detailed documentation is located at:

    Also popular now: