How we almost lost 5,000,000 hryvnias per month due to the “wrong” hosting: client history

Clouds are intriguing and frightening for many at the same time, but there are not many “our-real” examples of how people work with cloud providers.

Under the cut, the real story of our client with real numbers and requests, about working with local hosting providers, what problems I encountered, how his entire database was “accidentally” deleted during the transfer, and that it’s important not WHERE, but from WHO to rent capacity, the service provider is important. This case study, an example of good marketing, does not guarantee good service. So, the word to the client:

It all started with the fact that a large pharmacy chain needed a website and hosting for it. Everything would be fine, but our catalog was large, with a huge number of photos. Contrary to expectations, it was only possible to launch our website store from the “third host” :), and then there was a server rental, and after that we went “to the cloud” .... But first things first.

A few words about us: a network of pharmacies " DS ", 111 pharmacies, we serve 25,000 customers daily, we work mainly in western Ukraine - in Lviv, Volyn, Ternopol, Chernivtsi, Ivano-Frankivsk, Khmelnitsky and Transcarpathian regions. The figures tell us more about the scale of the business:



After working for several years, at some point we decided to expand. Among other things, we decided to create a website that will present the entire catalog of goods offered by our pharmacy network. At that time (2012) it was about 6,000 items. As an online presentation tool, the site was supposed to tell customers about drugs in our pharmacies. Each item of the assortment was supplemented with instructions and photographs.

Since the sale of drugs on the Internet is prohibited in Ukraine, we have selected goods for online sale in a separate category. This list includes cosmetics, non-medications (vitamins, for example), medical equipment, etc. Thus, the first task of the site was informative - instructions for use + photos, the second - the sale of permitted online assortment of goods.

The site was implemented on Bitrix and integrated with 1C. For hosting for the site, we turned to one of the Ukrainian hosting companies. During testing, we have never been able to unload the entire directory. After the failure, we thought that the problem was that “we came across such a hosting”. Ok, we tried another vendor - and again the catalog unloading did not work even once. Suspicions that something was wrong with our software quickly vanished (tested on our virtual server - it worked).

I had to search for a hosting provider whose capacity could withstand the load of our voluminous product catalog. After 2 unsuccessful attempts with Ukrainian hosting, it was decided to try hosting from a European provider. Having made this decision, we have never regretted!
When downloading the catalog already from the SIM-Networks hoster , we did everything the first time without problems and delays. Since then, four years of working with this hosting have been successful.



Leasing out other parts of our IT structure

In addition to the site, in mind - let's say politically correct - of “some actions” of regulatory authorities (remember 2013), we wanted to secure our infrastructure, our ERP network, our data. To do this, we decided to try the removal of infrastructure in the "cloud". Our servers could not cope with the peak load and it was necessary to increase capacity. In particular, it was a matter of streaming downloading documents (more on this below). Initially, the idea was to switch from SAS disks to SSDs on our own servers, but, having calculated the cost of such a transition, we realized that it was much more profitable to rent the required capacities. At the same growth rate as ours, it will be necessary to purchase expensive equipment every year. Naturally, such a strategy is not at all profitable.

Having monitored the prices, initially we tried to "go to the cloud" to one of the domestic hosting providers - due to the lower price. We took the power to the test, downloaded a copy of our system, carried out the most necessary, in our opinion, tests - everything was fine. But, having launched the system in a productive way, we were very disappointed. Throughout 2014, we observed failures in access to data, which led to huge delays in work.

Access failures from Ukrainian hosting. How It Was

It was "impressive." For example, infrastructure performance dropped to zero by about 1-3 minutes, and then quickly “rose”. And this seemingly small delay created a snowball of even greater time delays and became the cause of serious disruptions to our business processes.



In a specific situation, the problem looked like this: a drop in the hosting “for only 3 minutes” built up a large queue for our database (all users hung up), the 1C system could knock out a lock in these 3 minutes and all packages of documents of outlets and branches were reset. Did it bring any inconvenience?

Yes, everyone was just “happy” - stamped their feet and cursed at what the light was on! At the level of a particular employee, everything developed as a solid thriller. Suppose a certain manager downloaded several invoices, and if he failed, he didn’t know whether the wiring had passed or not? Like many other colleagues at the same time, this manager had to delete all downloaded postings and manually drive in unloaded invoices. It was such an eternal, boring, annoying and useless process. The monthly turnover of our network was approximately 50,000,000 hryvnias, and due to such failures the company could lose up to 10%, i.e. up to 5 000 000 UAH per month!

Not wanting to do anti-advertising, I will not name these hosting companies.

Our peak load is approximately 700 batch-loaded documents in the morning, from 9.30 to 11.00, and the same in the afternoon - from 15.00 to 17.00. Every day (every working day) about 15,000 documents were published. The entire volume of documents was loaded at the same time and was subjected to various processing - verification of 1 waybill could be carried out on 20 (!) Positions.

I must say about the peculiar seasonality of such failures on the Ukrainian hosting: as a rule, they happened at the beginning or end of the month (about 10 days), the middle of the month (obviously, when everyone submitted the documents to the pension fund), the 20th day (delivery time tax documents).

Comment from SIM-Networks
Such delays most often occur when a provider rated for, say, 5 clients, places 10 clients on a server, in the (naive) hope that they will never use all their (already purchased by them!) Capacities. We believe that if a client bought a place, bought a reserve of power - whether it is empty or not - it is already a client, and he can do whatever he wants with resources.

It was noticeable that the hoster did not count on the simultaneous use by all customers of all leased capacities. As a result, the resource guaranteed by the vendor, ordered and paid by us, was not provided! There were no failures, obviously, only when our hosting neighbors did not use servers.

So, for a very long time we proved to the hoster that this problem lives on their side - we started monitoring, wrote letters about failures. And only after a year of such torment, the hoster company agreed to buy an SSD shelf “specially for us” in its data center. But it was too early to relax ...

The migration of our data to this shelf was done horribly. The provider promised to transfer the base in just a day, on the weekend. On Sunday afternoon, the restructuring of the new shelf was not yet completed and we asked for everything to be returned, to which we received an answer - but we moved it, we look at the system, and so we deleted the old database. The hoster decided to delete this data, since he believed that the migration had already occurred and it was no longer necessary to store them! - we were shocked ... i.e. even before the new SSD regiment was completely restructured, they killed the entire source database (!), and we already need to launch our entire retail network.

Due to such a state of emergency, we deployed our data from backups. A complete restructuring of our “new” SSD shelf at the hosting provider took a week! And all this time we had to work much slower on our spare infrastructure. Can you imagine what it means for a retail chain to stop for a whole week ?!

We were ready to rent physical equipment from this hosting provider, for which the hoster told us that we needed about $ 85,000 and offered to buy this equipment for installation in their data center. This option, of course, did not interest us.

In search of a solution, we tested different hosting services - both ours and European ones - and saw the difference.

On our own servers, 100 invoices loaded in 16 minutes. If you put 5 people who will load 100 invoices, it stretched to 2 hours and these 16 minutes turned into 40 minutes - 1 hour.

On the SSD hosting of a domestic provider, the same package of documents was downloaded in 9 minutes.



With SSD hosting in a European data center, everything was loaded in about 4 minutes. The main task that we solved was the elimination of queues for the database, and at the capacities leased from the provider SIM-Networks, we managed to achieve the desired result.

After the accumulated unsuccessful, but extremely useful experience, we decided to rent from the European hosting provider not the virtual, but the physical infrastructure. The company transferred not only the main 1C server to it. After a successful experience of use, we additionally rented 2 backup servers, plus we moved the cluster, including IT infrastructure, file server, Microsoft exchange, etc. As a whole, we rented several servers and a shelf in the data center.

On the economic feasibility of leasing capacity for IT infrastructure

At the time when we were thinking about switching to SSD, our servers had already served us for 3 years, the warranty from the manufacturer was just ending (next business day) and it needed to be renewed. This guarantee annually demanded about 300,000 hryvnias (about $ 25,000 at that time) plus the cost of electricity.

Calculations showed that with the independent purchase of all the necessary equipment, we will reach the 0 capital cost coverage only after 3.5 years. Given the constant development of the company, the amount of the cost of increasing the capacity of our IT infrastructure could increase exponentially.

As a result, renting the same equipment from the hosting provider was not much more expensive, but this was without worries about:
- extending and buying a guarantee;
- increase in physical capacities (equipment upgrade);
- electricity costs.

We are going to the clouds!

We want to expand again, since we are already short of power, and are seriously considering the option with the cloud. We have no fears of moving to the cloud, as we have seen from our own experience that it’s not so important WHERE to rent, it is important to WHO to rent from, it’s important to choose a reliable service provider!

Andrey Semenyuk
IT Director
DS Chain of Pharmacies (Market Universal LTD)

History of our adventures with hosting and moving to the clouds:

2012 - hosting of dska.com.ua, unsuccessful attempts to upload the entire catalog to Ukrainian hosting, hosting of the site in SIM-Networks

2013 - plans to switch from own SAS to own SSDs (economically unprofitable), transfer of the company database to the Ukrainian data center, failures during operation - disruption of business processes - during the year

2014 - allocation of the SSD shelf in the Ukrainian data center, fakap to a week during the database transfer, transition to SSD hosting in SIM-Networks - database transfer, 1C-servers

2015 - after successful use, we rented several backup servers, moved the cluster (IT infrastructure), file server, Microsoft exchange and etc

2016 - plans to fully switch to the region Private / rental infrastructure at SIM-Networks

---

SIM-CLOUD - Failover Cloud in Germany

Dedicated Servers in Reliable Germany Data Centers!
Any configuration, quick assembly and free installation.

Our fast SSD hosting for sites on CMS 1C-Bitrix . We are also ready to offer dedicated servers and VPS optimally configured for working with 1C-Bitrix products.