How we created a high-tech product and fell to the bottom
I want to share with you the story of the good life and the long, slow and painful death of one, once large, real estate portal. Which was not ready to change and make sharp movements in order to adapt to the changing market. A product that for 10 years has gone from TOP-10 to the bottom. This is a story about the importance of the thread that connects developers, with their understanding and intelligence, and managers / directors / managers with their approach to project management.
I came to this job in 2014, a simple junior .net. Mon (hereinafter the real estate portal) was a real estate aggregator, which, in addition to the ad database, included a high-quality thematic news section. News, articles, laws and many different materials on real estate. A large department of cool journalists wrote very cool articles, the interest to which was very high among users.
The site itself was free to advertise, which attracted users (private sellers, real estate agencies). All income was formed due to the usual in this area of advertising placement of text-graphic blocks (TGB) from developers promoting their new buildings and residential complexes.
During its heyday (2008 - 2012), the site had traffic of over 1,000,000 unique visitors per month. And he gave a good conversion for advertisers. These were excellent performance, given that the PN only worked for Moscow and St. Petersburg. There were 2 programmers and a lot of journalists and editors. In essence, only 2 employees carried out all the maintenance, support and revision of the TU. (How many people designed and developed the TU initially were silent).
The positive part of the story ends here.
After the first wave of crisis, the site began to experience traffic problems and very serious ones. He just began to fall. The market began to change quickly enough. Competitors began to crush, because while our PN rested on its laurels and did not develop anywhere, others made their products more modern and responsive to users' expectations.
In addition, search engines have become less and less like our site, which by the way was practically without any CEO. Yes, there were some templates for h1, title, description, a small linking, but in general, and everything (after the beginning of the fall and to the very end, the head of the PN started the SEO battle, which we had been doing for almost 5 years).
With the second wave of the crisis in 2014, the figures fell below 300,000 unique visitors per month. About this time I got a job.
This is where my technical part of the story begins .
The first 2 years I worked as a junior developer. He did everything they gave me. I had to learn a lot from scratch. However, I was lucky with my development manager. He was a little over 40 years old, a programmer of the old school, he knew a lot and knew how. Probably his main credo was - it works and it is good (I am very grateful to him, because I adopted a lot of knowledge and various approaches to the development).
Our technology stack was: MS SQL Server 2012 + ASP.NET MVC 3 . The base kept in itself everything. Even photos in binary form, for 3 sizes set (big, large, small).
Backend included several modules:
- Mon site
- General Purpose Admin
- SEO admin area
- Robot parser KLADR
- XML feed import robot
All this time I was just doing tasks so that the code worked and there were no errors. Especially not delving into and not thinking about further development. But the time has come and it has become necessary.
In the second year of work, my manager left the project. And I was left alone with this ancient colossus, in which there was at most 3% of my code.
It was a wild stress.. The CEO asked me everything, and sometimes I had no idea what was working and how. Naturally, life made me and gradually I penetrated into all the processes and realized that that approach “works — well” is not for me. At this time I read a lot of materials on design, enlightened in the field of popular technology and frameworks. I drove home from work and thought about what I would do tomorrow. And when I went to work, I thought about whether it was right that I decided yesterday. I wanted to understand how to do everything correctly and conveniently. So that you do not need to do crutches, duplicate the code, test everything live, deploy everything manually and discard good ideas because of some design mistakes in the past.
By this time we have done a lot on SEO, having abandoned tasks focused on improving UI. However, traffic fell lower and lower. And at some point the project was frozen. I became engaged only in supporting and correcting bugs ... So it sounded official.
In fact, I began to completely rewrite the system almost from 0. I had to do it secretly from the management. After all, we were never given time. They always said that it was necessary faster, faster. That we are so far behind. And it is necessary to make a competitive product in 4 hands and better than others. Therefore, if I said that I planned to redo everything, then you understand what answer I would hear.
So, rolling up my sleeves, I began to conceive. I chose the classic three-tier architecture. Backend .Net Core + SQL + Mongo, frontend - Bootstrap + JQuery + KnockoutJS .
Arranged data layer. Interfaces, abstractions, repositories are all as expected. The layer on stored procedures worked (the blessing I began to understand quite not bad in SQL). For mapping chose Dapper. It is simple and straightforward. He refused InMemoryCache in favor of Redis in order to bring the cache to a separate server. Next was the level of business logic. All the same interfaces, services, DI. This is how the foundation appeared in the form of a Data-Layer (Stored Procedures + Dapper + Redis) and Logic-Layer. All took about 3 months.
And then, after almost a year of work alone, I achieved that I was given an assistant, insisting that there were a lot of tasks (and there were a lot of them). Together everything went much faster and better, and most importantly even more rationally.
- First of all, we developed an API for photos . It was a simple WebApi, which by Get-request gave the image of the desired quality and size from the disk. We switched to SSD and forgot about the image database as a bad dream. It is difficult to describe how much faster the average page of the site began to load after allocating a separate pool for it.
- We refused KLADRA in favor of FIAS . We wrote a high-quality data parsing service from FIAS to our database, taking into account our features. Screwed to it a service for geocoding houses. Everything worked almost like a clock. Only sometimes there were bugs associated with duplicates of locations or streets in the FIAS database.
- Then they wrote a new personal account for a long time , dressing it from the site. We designed and planned it for a long time to be user-friendly. And also fast and functional. We screwed up the payment for it, and then fiscalized checks (yes, we could not afford to use ready-made integrated solutions). In general, made a good user service. And they were pleased with it.
- Finally got to the robot importing XML feeds . Made a convenient validator and good logging for clients. The new service turned out to be so optimized that if the old one (using EF) worked for about 6-8 hours, then the new one processed the same amounts of data in 2-3 hours.
- After all, they raised the domain with documentation for everything that is there. They put all the points on the shelves for the users and clients of the portal, and also described a part of the documentation, which will be useful for the developers. And this is really important!
- The final step was to optimize the base . We completely reworked it. Cleaned out all unnecessary. Achieved faster search speed for 4-5 seconds to ~ 300 ms . They created indexes, wrote complex queries, used hints, and even made partitions of statistical tables.
Unfortunately, we did not have time to get to the site itself. Because Almost all the tasks with the site were associated with SEO, which took a decent amount of time. New pages, new collections, new rules. More, more, more pages. It was necessary to constantly edit something in the site's engine, which did not allow in parallel to transfer it to the created base.
Here the technical story ends and a sad epilogue begins.
It is worth starting with the fact that the CEO of the site was not connected with the IT sphere. Therefore, a lot of decisions were made by him incorrectly and individually. Very often discussions came to a standstill, because our new ideas were not accepted because of
“It’s not necessary, I’m talking to you as a real estate expert.”or
“So no one is looking, this is a low-frequency request, I’m sure”And after some time, when he himself came to this, our ideas were offered to them with a pretense, why we had not said it before, or with genuine surprise.
"I could not say so, this is absurd"We could not come to a consensus, as a result, we (the developers) simply conceded. And they did as we were asked. Programmer pain is to make useless “features” to no one.
I would like to mention that there were always problems with finance. No investment, except in SEO was not . Even keeping 2 programmers turned out to be expensive. Obviously, it was unrealistic to compete with new portals from the TOP-10 with such a level of financing and management.
As a result, we have a platform portal real estate aggregator. Scalable, expandable, working quickly and technically ready for any cataclysm. With great potential. Good code, minimum crutches and few bottlenecks.
However, it did not give any positive result. At the time of my last working day, despite the presence of 4 million unique pages in search engines, the portal traffic fluctuated around 1,400 unique users per day. And this, it seems to me, states death.
PS: From this whole story I could personally make one main conclusion. You can make a good product from the point of view of the developer, but it will be absolutely not in demand, because does not have proper control. If that thread between employees, which keeps the business afloat, is torn, then your product will certainly go to the bottom.