How to permanently delete 100,000 users and stay alive?

In this post I want to share with you the experience of developing one large project. During its creation, 100,000 users who had to be restored were permanently deleted.



For young developers, this will be a story that will allow me not to make my mistakes in the future, and more experienced dudes can just laugh and comment that they would never have allowed such a situation in their life).

The story began in April 2015. In the beginning, everything was standard. The customer (I will not deliberately say names and company names) wrote on VKontakte, asked why, dropped the approximate technical specifications, met, talked, liked each other, decided to do it.

The idea of ​​the project is as follows:

There are companies that publish on their website discounts from themselves. Well, for example, sneakers with a 20% discount. Users buy a monthly subscription and can enjoy discounts on all the companies represented.

Also, the site has an affiliate system that allows you to invite people. If the person invited by you paid the subscription, then you will receive a percentage. The logic of interest passage is quite complicated, but the customer explained as follows: “Well, there’s nothing complicated, just write small formulas.”

And they have already made a similar site, they just run it in and decided to make it prettier, bigger, better. So, it was necessary to transfer all the data from the old when starting a new site.

It should be clarified that I had not done such projects before, and for me it was something new and unknown. Therefore, I initially did not understand how much objectively I would have to do this. Given that the design has already been drawn, it was only necessary to lay out and put on the back-end, I requested 65t.r. and 2 months of work. Both last numbers turned out to be fatal. There was little money and time) It was necessary to take more. But ... the lack of experience is the lack of experience ...

Now I will tell you the key point. Given that the stage with the “affiliate program” was described as “several formulas”, I decided to do it in a week. Moreover, in terms of TK there really wasn’t a lot of work. Oh how bad I rated this work then ... oh how bad ...

As a result, according to the law of the genre, while I squandered the advance payment, the work did not go and started only after a month) Therefore, the night-day mode of operation was introduced and in three weeks I did everything that was required, except for the logic of the affiliate network. It was necessary to hand over on June 1, and just somewhere in the 20s I took up its implementation.

In a week everything was written exactly according to the TOR. But it wasn’t here ... when meeting and testing it turned out that they did not tell me much! And there are still many nuances in logic that need to be done ... Well, I think, okay ... we will. And the deadline, meanwhile, was already flying into tartarars.

In short ... so a month has passed.

All June, we ruled and completed this damn logic. Either the money goes wrong, the invited person becomes the wrong person ... it was tin. Now I would send them to hell and ask for surcharges ... but then I was younger, snotter and could not afford such luxury.

At the end of the month I was squeezed like a lemon. I did not want to live ... I had to rest. I told the customer that everything ... pause. And he dumped the countryside for two weeks.

When I arrived again an endless series of edits began. In general, I stopped responding to messages and calls. I am gone. Missing and it was impossible to find me. Like this.

I just gave up and went underground. Yes, not manly. Now I wouldn’t do that in my life. But then I was not ready for such a kind of problems. They broke me ...

After some time, they stopped looking for me and calling) I thought that everything was fine. Maybe they found another developer, maybe something else. But no one bothered me anymore. Paradise.

To make you understand, these were not just some obscure types. These are relatively famous personalities in our city. Behind the customers themselves there was a larger leadership, which could really make a very big commotion in my life. But I found out about this only in the middle of the project.

During the lull, I got a job in a web studio. In early August, they started calling me again, and I began not to answer again. As a result, the customer himself came to my work personally) Asked just to finish what was started and go. The format is this: he writes what needs to be completed, I also make a happy end.

Also, we met with the most important guy in the project, who indicated the fact that under the contract I have a percentage for each day of delay. In general, I had to pay another 180 tr plus ... But they were ready to forgive good-naturedly everything, if I did all the changes before the fall and started the project, otherwise the court. I think it’s clear that I didn’t have much choice.

At the end of August there was a launch. He went with a creak. It turned out that several errors flashed in the code that several tens of thousands of rubles were sent to the wrong people ...

Here you need to make a clarification. The project already existed and 100,000 people were registered in it (actually 110 thousand, but I rounded it). Every day about a thousand visited the site. Several hundred people conducted transactions per day. And these few hundred people were very dissatisfied (((

Well, it happens. What to do ... They notified me on the website that it is possible to write to me on VK. I made translations with my hands and at the same time understood where the errors were in the code. Every day I had about hundreds of unread messages. During the month I woke up at 7 a.m., processed user requests, went to work, processed applications again in the evening.

In general, people reacted to the situation with understanding, therefore it wasn’t very smooth, but the situation was taxied out. Gradually, there were fewer errors and site n Chin how to work.

But, as they say, trouble does not come alone!

One evening in the office after work, I decided to file the site. I go in, and he lies! mysql fell off. Before that, mysql sometimes crashed when the server could not cope with the load and I just rebooted it. But there was another case. It didn’t even start ... I spent the whole evening on the forums. Then I believed in Toster! Some guy just all evening answered my messages and helped find a mistake! I was surprised that someone could help a complete stranger for free.

Plus, I didn’t have much experience with servers then, so it was difficult. I only knew that there is a terminal and commands can be entered there. I googled, searched for which command to enter, entered and got the result.

As a result, I came to the conclusion that the database is lost ... no ... Then I found out that there is an ibdata file and if it is empty, then all the data is lost. As you know, it was empty. Completely. 0kb

And the girl and I went to the Stand-up that evening, our friends guys performed. I did not stay until the end and asked to go home. It was there, in the hall, the thought reached me that somehow I had lost 100,000 users. There is not a single backup. There is no hope of recovery. What to do? To call and say that ass boys, I have sowed the labor of several years of your work.

After a call and clarification of the situation, a consultation was convened. At 12 at night. What to do?
I tried the story that the server ran out of space (it was actually so) and therefore mysql was slowed down. In fact, until recently, I was sure that this was so. So, like, my fault is not here.

Thought, thought and decided to go to sleep and on the trail. day to figure out what can be done and what really happened.

I asked for help on freelance. I found one sys-admin there who agreed to help me out. As a result, he confirmed my theory that most likely mysql fell off due to the finished place. And even worse, he confirmed that the base could not be restored.

Customers asked for help from a web studio in which the programmer looked at the history of commands in the terminal and you know what you found? The command that erases the ibdata! File Apparently on the forums, someone was funny that you need to enter this command if mysql crashes, but in a panic I did not read between the lines and entered this command.

Well, I think that's it. The end to me. But the customers realized that it was necessary to solve the problem, and if they "killed" me, then there would be no one to finish it. Fortunately, there was a backup of the old site on which there were 80,000 people. Class. Now it remains to recover 30,000 people) A new phase of messages in VK, recoveries and panic has begun.

Thanks to my girlfriend who supported me. Thank you for the fact that she didn’t leave me at all) For it was then impossible to communicate with me, because every night I whined about what was hard for me and I don’t know what to do ... I don’t like aching people. And I didn’t like myself either.

With grief in half, everything was restored. In the final analysis, I said that let them find a programmer on a permanent basis, a system administrator who will maintain and back up the server and live somehow without me. I no longer wanted to deal with this hellish project.

There was something else to fall off and break, but that was all over. The site is still working. They never found a programmer for themselves (as far as I know), apparently, after all, my hands are not so crooked and now everything works as it should.

By the way, they did not pay me 10 thousand. But I was somehow ashamed to ask them later ... therefore the project cost 50 thousand rubles and 6 months of suffering. After it was over, I thought to myself: "But I got a huge experience." And really ... this project taught me a lot. And it will not be right if I don’t share it:

1) Carefully read the TK and the agreements.

In fact, the weighing of the tusa-jus began because I took lightly on TK. If I read it more carefully, I realized that the stage with the logic is far from complete and that it needs to be completed, the deadlines would move and we would calmly complete the project.

Be sure to read the contracts that you sign. And they must be compiled!

Now I do not draw up a contract only if the customer is my friend or I have already worked with him and I am confident in his good faith. In general, since I signed a bum on that every day the delay is a penny, it turned out to me the situation described above.

2) Evaluate the project in terms of money.

Then I did not know how much to take for the project and for me the figure is 65 tr. was something unattainable. Now for such a project, I would ask for 200-300 thousand. I realized that if a project has a large number of personal accounts, logical relationships, then the project will not be a priori simple. And even if it seems that there is nothing to write there, then this will not happen.

3) Speak out escape routes

You must always understand that a project can go beyond. Or the customer came up with many different ideas that were not in the TOR. Or you just did not calculate and the deadline left yesterday, and the project is only in the middle.

Talk with customers this moment on the beach. At the crossing of horses do not change.
Now, when customers ask how much it will take for the project, I say the minimum and maximum. For example: “A project can really be done in 2 weeks, but, objectively speaking, we will be doing it for a month. Subject to changes, additions and so on. ".

4) Make backups

Make backups, their mother! Do it! Be sure to do it!

Two of my working computers are currently backing up to my hard drive. The server did backups every day, but after one situation I made them to be created every hour and every day. In a nutshell, I did not keep track of and the hard drive ended and mysql broke (the situation is like in this project), but I had a backup and as a whole it saved me).

After this incident, I thought ... But what if actually mysql really broke down due to the small space on the hard, but I deleted ibdata and then there was nothing to prove? If so, then somehow it’s a shame even ...

5) See what you type in the terminal.

See what commands, where and why you enter !!! Now, if I don’t know the command, I google it before entering the terminal).

6) Delineate responsibility

In fact, the situation with the server happened due to the fact that I did not climb into my estate. If you make a site, then make a site, and let those who know how to answer for the server be responsible for it. After this joke, they hired a sys-admin, who then steered a server).

7) Do not whine

In any incomprehensible situation, do not whine! Aching people seem miserable. You can "cry." If it’s hard, go to a person close to you, ask them to listen and pour out everything that has boiled. But do not whine every night as you feel bad and hard.

From time to time I meet people who constantly whine. They cannot be tolerated. With some I stopped talking even because of this.

8) Take responsibility for your shoals

Nakosyachil - catch Lyuli. If something went wrong, then call the customer and explain the situation as is. They are people too and will understand, enter your position and even help solve the problem. Do not disappear) Only you will be worse.

This applies not only to sites, in fact, but also to life.

9) We get high from force majeure

During this mess, a friend told me one interesting thing that all problems come from our heads and if there are problems, then we want them to be. And I realized that, to some extent, I get high on this situation. I am a programmer masochist! I love force majeure and therefore attract them. Moreover, I do not like that, “Ahhhh! I want to break the deadlines! Let's make the site another 5 months for free at night. ” It’s just that inside, a small piece of me sits and quietly enjoys what is happening.

Think about it if you're constantly breaking deadlines.

10) Do not tackle dubious projects

Filter your orders. This project was initially dubious. But I needed money and experience. Now I sift out 70% of orders and say that I do not want to do them, they just are not really interesting to me. Another question is that I can afford it now, I could not then)

On the other hand, I took up this project and got pretty decent experience. Therefore, the devil knows him. Probably, after all, take on dubious projects - this is experience. But only once ...

11) Belief in Toster

I believed in Toster. I was really surprised when I realized that people just respond to other people just like that. It was an amazing template break for me. Now I constantly ask my questions there and they constantly help me out. And as far as possible, I answer to others.

Recently, there was even a situation where I googled a question, a link to a toaster appeared in the search, and there I ask the same question and answered it in a comment. I closed myself.

Completion

Every item now lives with me every day. Each new order I take into account all my mistakes and do not repeat them anymore. After this project, I did not have deadlines and frustrated customers. If force majeure situations arose, then they were immediately led back on track and resolved in peace negotiations.

Do not be afraid to make mistakes. Be careful not to draw conclusions from them. It was a tough lesson. But I do not regret that this happened to me.

Thanks for reading! I hope it was interesting.