uid.me - personal page service based on MongoDB and Mojolicious

Good afternoon, Habr!

We want to do a review post on our new project. The review will affect both the functionality and the technical part, we hope this will make the article interesting to both professional developers and those who read the Habr with the goal of keeping abreast of Technology.

For those who are only interested in the technical side of the project, we recommend that you immediately go to the second part .

PART 1. Lyric

We are the development team of the uid.me personal pages service .
A personal page is, for example, like this:


http://uid.me/dikaya


http://uid.me/pavel_kudinov

Those who are not familiar with the western counterpart of our service should admit: the uid.me projectbegins its history as a clone-localization of the English-language service about.me Creation

History The story

was like this. The uCoz website builder company in which we work has accumulated over 35 million profiles created by webmasters in the depths of its data centers over the 8 years of its existence, as well as numerous visitors to websites, forums and blogs created by webmasters.

All these people are united by the global uID authorization system:



Until today, every person registered in uCoz had a profile of this kind:

http://3707164671.uid.me/ About.me



projectwas chosen as the best existing prototype of an individual page for each uCoz user, which, in our opinion, meets the current trend of self-expression of the inhabitants of the Web at the beginning of the 21st century.

As in the case of about.me , we give the user:

1. The URL of the form uid.me/ surname_name , which can be used for printing on a business card, can be used as the home page in skype, and also be mentioned on any media carrier.

2. The ability to combine a personal photo into a single visual image, a high-resolution background image, basic information about yourself (such as a biography and area of ​​interest).

3. A designer with which you can quickly and excitingly give your personal page a unique look and overall visual consistency.




4. And, finally, the most interesting: today many of us are actively present in social networks. Someone closer to the formats Facebook and Vkontakte, someone limited to microblogging Twitter and Instagram, some have their own popular channel on Youtube.

And here the rule is true - the more social activity a person shows, the more acute the question arises: “which of the social networks should be considered“ main ”?”.

We suggest using uid.meas a kind of personal online business card. Our service allows you to attach the most common social networks to your own profile, and then you don’t have to choose which link to give when you make a new valuable acquaintance, specify skype in your profile or sign it in the forum.

Your posts, tweets and photos will automatically appear in your profile, and the general view will be used for display, and the “stream” function will create a common feed of events, combining in a chronological order everything that happens to you.



By the way, if you want to create a personal page on uid.me , we recommend using automatic registration through a social network. When you click on any of the “ Login via ” buttons”- a personal page will be instantly created without the need to enter registration information!



Looking ahead, we want to say that close integration with social networks in the near future will significantly shift the project development plan.

The second version of uid.me profiles , already under development, will have the main focus precisely on the function of combining information from social networks into a single stream with a customizable data presentation.

In addition, it is planned to develop several interesting infographic widgets in the form of colorful graphs and charts that represent information about your friends, travels, musical preferences and other interesting entertaining facts that the system will be able to extract from your profiles and process automatically.

Perhaps it will look something like this:




PART 2. Technical.

Developing uid.me under the wing of uCoz, we found ourselves in a rather unusual situation: on the one hand, the entire project code was supposed to be written from scratch, on the other hand, on the release day the project automatically became heavily loaded, as it was supposed to be imported into there are more than 20 million profiles, even taking into account the fact that bot registration and absolutely ancient profiles did not pass the competition.

Nevertheless, we decided to do everything beautifully and using fashionable technologies, thus conducting reconnaissance in battle, gaining a lot of experience and potential level-up in the end.

As components of success were selected:

0. Nginx. Where without him.

1. A database that out of the box solves the issue of distributing data to multiple servers + fault tolerance for physical loss of a server from a cluster for any reason. In this capacity, despite the active holivars, MongoDB was chosen.

2. A flexible data scheme that allows without loss to go through the primary and subsequent phases of functional prototyping. Again he helped MongoDB, although there had to pay for the convenience of resources, so that to get the main answer to the question: "BSON - a luxury or a modern vehicle?" - yet to be .

It is worth noting that the original mysql database of user profiles when converted to MongoDB format grew 5 times. However, each profile was enriched with an impressive amount of new data related to the functionality.uid.me ; therefore, it is not only a gluttony of a flexible BSON data scheme.

3. Honestly, given the current trend towards the active use of dynamic JS interfaces (as well as the immense respect for the technological breakthrough made by Google engineers during the development of V8 Javascript, which bypasses all existing scripting languages ​​by an order of magnitude due to dynamic compilation into machine code), crept in crazy idea to use node.js and close the circle of web development in JavaScript, at the same time getting a few fat buns ...

But they decided that “ one project is one new technology, and for now MongoDB is SO good aet ”(c) Alexander Soloviev. By the way, who did not see his report is a hit, we recommend it to the whole team!

As a result, we decided to leave the corporate-familiar Perl as the server technology , however, we managed to gain the second cosmic speed, leave the fast_cgi gravitational field and apply Mojolicious - a modern, stand-alone and adequate (apart from the author) web framework with routes, helpers, bridges, built-in support for asynchronous requests and other sweets set by the modern developer.

4. Total asynchrony and data caching when interacting with social networks.

Speaking of the prototype of the project - it was noticed that the data from social networks received by about.me service, are not updated, loading only once - at the time of connecting the service. The cache update option is probably available to VIP users, but we were not able to get information about.me aboutme . This led us to the idea that it would be best to organize interserver communication and the cache system as much as possible in order to minimize the risk of similar problems in the future.

Almost universally implemented OAuth2 and the similarities in the organization of the API of various social networks made it possible to successfully generalize the interaction.

Of course, at the prototype stage, all work with the API was synchronous, but blocking Hypnotoad workers to implement API requests in a highly loaded project is a definite luxury and waste. Fortunately, Mojolicious is built on a very decent event machine, both in terms of interface and implementation, due to which, by the way, each worker in the pool is able to process not one (but, say, with mod_perl), but dozens of parallel requests Of course, provided that they contain a significant amount of asynchronous code.

By the way, given the fact that one of the main “scary” arguments against using node.jsis its total asynchrony, - Mojolicious can serve as an excellent mental bridge when you start development within the framework of the classical synchronous paradigm, and finish at least with a significant part of the hybrid code (sync + async). Frankly, now we are afraid of node.js much less and hope to apply it in subsequent projects.

In general, uid.me was made on the principle of “no to bicycles”, and Shiva solemnly sacrificed a whole layer of fossilized homemade products, headed by the widely known in narrow circles “ dw ” kilobyte macro , since 2005 faithfully served us and the developers close to us and allowing DBIx :: Class to escape the difficult transcendent horror. Bright memory.

And yet, when developinguid.me one entertaining craft was born - this is a macro

take { … $take->(‘named_callback_slot_1’) ... } process { my $taken = shift; … },

built on Mojo :: IOLoop-> delay and drastically simplifying the entire cycle of operations related to the organization of named cascaded asynchronous API interactions, including cascading exception handling (if you are interested, write in a personal one and share it).

Returning to MongoDB

In practice, I wanted something similar to NoSQL from the time when this was not the mainstream. As part of those highload tasks that we had to deal with at that time, the following understanding gradually emerged:

1. The classic LAMP project starts with the classic SQL database.
2. If the project becomes popular, it gains the status of "highload", otherwise goto 1.
3. The status of "highload" obliges us to think carefully about caching, sharding, replication and
backup of what is stored in the SQL database.
4. The evolution of the data scheme of a living project becomes all the more painful, the more data is accumulated, and the more in demand, the more popular the project is.
5. As a result of all this, the ORM code begins to perform the functions of mutex, serialization / deserialization of data for memcached, primitive sharding, in especially cruel situations - patches to ensure backward compatibility of the data scheme (because it was not always possible to afford a large end-to-end update of data in real conditions) .

However, quite sad, in the yard were harsh 2000s.

The beginning of 2010 was inspired by the emergence of several NoSQL solutions that promised to eliminate most of the problems of the growing highload project out of the box. The advent of open, ready-to-use NoSQL solutions was predicted by many, but, nevertheless, the actual acquisition of a beautiful future pleasantly surprised us.

After consulting with more extreme colleagues in terms of innovations, we decided to try MongoDB .

Studying a new technology for ourselves, we considered it logical to use its capabilities to the maximum, hoping for the best (and, therefore, a silver bullet out of the box), hoping, however, to roll back to more classical techniques in those places where excessive arrogance would confront us with interesting pitfalls.

By using the maximum capabilities, we mean the following:

1. The JSON data storage format allowed us not to bother with the usual parent / child / x-links in the data scheme with or without, limiting ourselves to common sense. As a result, the nested structure of the main user object turned out to be bold, but convenient. A lot of flags, display settings, small linked lists, and all the other things that previously led to the creation of a bunch of near-user SQL tables were boldly put into it.

2. A general purpose code was added to the data model, which at the stage of interface prototyping made it extremely pleasant to increase JS functionality: it became possible to send any JSON that extended the user object with new data via URL / profile / save , for example:

user.save({
    'style.profile.top': '20px',
    'style.caption.tags.color': 'rgba(30, 29, 38, 1)',
    'info.first_name': 'Павел'
});

All operations related to the activities of an authorized user were packaged into a common sending function with a latent collector of 500 ms, combining various atomic edits into common packages.

As a result, client-side developers were able to easily expand the structure of the user object by simply starting to use the new fields.

Of course, after the prototyping phase, the server side / profile / save was equipped with contextual data filters that cut off unknown fields and filtered values ​​for correctness.

There was only one problem - users could be stored in the database for which some fields did not exist at all, since the last time they edited their profile before these fields appeared. Ideally, I would like to have default values ​​for each field that will magically appear in any object retrieved from the database.

At the ORM level, the forced extend of all retrieved data was added with default values ​​for all non-existent fields.

The circle is closed.

We got the opportunity to dynamically expand the structure of the object without resorting to end-to-end updates of the database, to transparently work with it not only from the server, but also on the client-side, while the process of adding a new one was quite pleasant, and the transition from prototype to release was accompanied by exactly two actions:

1. By adding a rule for the data of the new field to extend_rules filters.
2. By adding the expected default value for this field to default_user.

That’s probably all. Thank you for your attention, we are waiting for you to visit!

PS for dessert, lovers of nudity: