The history of one highload project

    How to write a highly loaded, multi-functional project together? What if there is no money and time, and you need to open? Under the cut some interesting information from personal experience.



    The idea to develop a project that makes life easier while maintaining Internet sites came up with a small company from Minsk. Our team is small, but ambitious, so it was decided to develop such a large-scale project together.

    Since the days of the webmaster are difficult and ugly, a lot of functionality was announced for implementation, namely:
    • the project should pull out every week whois information on all existing domains (more than 300 million planned)
    • keep all whois information for all domains together with the change history;
    • notify about the expiration of the domain registration and hosting;
    • monitor sites at least once every 3 minutes and at least from 6 countries (beautifully display time, response codes and graphics with this is all for each country);
    • check the site for blacklisting, as well as for the presence of malicious muck;
    • manage finances with the addition of cost / profit transactions on sites (again, with charts);
    • manage notes, with the ability to add tags and attach files;
    • notify clients via e-mail and sms, according to user settings;
    • fasten an interesting affiliate program with discount coupons and links;

    In general, it was necessary to write "Dominder" (from English domain + reminder).
    A useful, interesting, promising project, but the problem is not enough time and resources. There are two developers (the second one only connected 3 months after the start of development).

    We ordered a cool, import design over the hill. They decided to write in Ruby on Rails, PostgreSQL acted as the main database.

    They began to search for compromises and technical solutions, thanks to which the implementation with the current material shortcomings can be brought to life.

    Solution Number Times

    They decided to score the time spent on the quality of frontend implementation less than we would like. We will finish it on the sly - the main thing is that nothing bothers customers.

    Solution Number Two

    Because there is no time, then one or another implementation method is selected based on the parameters:
    • time costs;
    • scalability;
    • (the current number of users in the system) the implementation must hold.

    Sometimes, grinding his teeth, you have to write one or another piece of code, just because it's faster. Crooked, ugly, but effective and flexible.

    Decision number three

    In parallel with the development, a public relations product was conducted. How did this happen?
    2 “streams of planning” are being built: development and PR. They should be filled evenly based on the current needs of the project and the possibilities for its current implementation. For example, if some functional piece of the project is ready, it was immediately tested, rolled in and popularized. Or, if the current number of users is less than predicted, then first beautiful baubles were developed for customers (for example, a demo account), and optimization of current algorithms was already secondary. And vice versa, if customers began to come in bulk, they would drop everything and begin to severely code. It turned out a kind of swing, but in a different way, in my opinion, in no way.

    Solution Number Four

    Whois information on all domains is available through public pages that constantly, mercilessly index search engine bots. Pages were cached, and domains that were not related to current clients were moved to a separate server (the so-called vertical sharding).

    Solution Number Five

    Due to the fact that the ping history needs to be stored indefinitely - all records are pressed into the “archives” according to the graph display algorithm.

    Decision number six

    Using your own proprietary proxy server to pull whois from registrars. Some of them at first simply banned ip because of very frequent calls. As a result, it was possible to obtain information for half a million domains per day. Not the limit, but there was simply no time for revision.

    Decision number seven

    Everything you need fast, frequent access to ruthlessly shove in NoSql.

    After the launch of the project, the question arose of attracting users.
    The same monitoring of sites in the West has long been popular and millions use it. But to explain to our Soviet man that it’s easier, more profitable and more convenient to pay a few dollars a month and be able to immediately find out that something happened to the site turned out to be a rather difficult task. Many did not understand that monitoring sites (and our other services) is like insurance. Well, if nothing happened. And what if?



    For tests, we added 140 Belarusian sites. As a result, it turned out that about 40% of sites ceases to be accessible at least once a week (for a period of 5 minutes to several hours). And some sites just swing wildly and obviously lose their customers.

    Of course, in Russia there is such a formidable competitor as Yandex.Metrica (and they often ask me what are our differences). But Metric, it happens, doesn’t immediately send notifications (sometimes it’s several minutes late), there is no way to check the availability of sites from different countries, it does not guarantee monitoring in some cases, and you need to install additional code on your sites.

    Having limited resources after the opening of our project, we began to act on our own on all fronts: e-mail marketing, affiliate program, partners in the form of hosting companies and web studios, etc.

    As a result, after all the tweaks and jumps, we opened and began to receive the first profit, while experiencing childish joy from each newly registered user.

    Also popular now: