Experience of real import substitution using Russian storage AERODISK

Now a lot of talk on TV about the success of Russian companies in the field of import substitution. Road maps are drawn, registries of domestic developers are created, forums are organized, and government funds are invested. money (as usual, it is not clear where).

The TV has already told everyone that the bright future will soon come and all of us are waiting for the Russian operating systems, databases and other necessary things.



In reality, everything, as usual, is somewhat different ...



A few years ago, Softline began actively monitoring the market of Russian IT solutions in order to meet the demand from state (and not only) companies for import-substituting products in IT. Well, there must be something, right?

Since today we are writing about storage systems, it means that we will tell about our experience as we searched, tested and implemented storage systems for a Russian manufacturer of the company AERODISK.

But before we got acquainted with this vendor, we went through seven circles of hell a long way of getting acquainted with OEMs, who in all seriousness pretend to be domestic producers.

And now just a minute of attention:

Firstly, we are not at all against the OEM business, this is normal and practiced all over the world. For example, the same HP has long and successfully OEM-IT DotHill storage systems, selling it around the world as its own and everyone likes everything.

Secondly, we are against only frank deception (I think everyone will agree with us here).

We will not give here the names of organizations that behave badly, deceiving themselves as domestic producers. We do not want to create anti-advertising to anyone, but, unfortunately, there are such companies, and they are widely known.

Attempt number one


Having started the search, we initially stumbled upon a whole bunch of so-called “Russian storage systems”, which are sold for “import substitution” in all papers and pictures, and in fact both equipment and intelligence are either Chinese or American.

Well, damn it with the equipment, of course, that even American manufacturers make their hardware in Asia, now this is the norm, but who's stopping to write?



Attempt number two


Of course, we were upset, but continued to search. Soon we found several Russian solutions that we actually had developed, but tests showed that they suffer from one of two (or two) diseases.

  1. Raw solutions
  2. Solutions not for serious (or for niche) tasks

Obviously, we could not work seriously with such decisions. Again, uncomfortable.



Attempt number three


Frustrated even more, we continued to eat cactus. At this moment (it was the end of 2017) we had a large federal project, where it was necessary to make maximum use of Russian technologies. There was still a design stage: the main technical solutions were laid. This was part of the “Safe City” federal project in one of the cities hosting the World Cup 2018.

The concept of the “Safe City” implies the unification of all the responsible security services in a single direction with the close integration of IT systems. This helps to respond to incidents much faster, and in some cases even prevent them.
Technically, the essence of the project is that everything in the city is hung with cameras (several thousand cameras), and these cameras, using a smart video surveillance system, automatically record dangerous or potentially dangerous events and in good resolution continuously write data to the data center. In-line analytics of events from cameras is displayed on emergency services consoles, and, moreover, records from cameras are stored in the data center for at least one month.

At any time, a law enforcement officer can contact the service operator and as quickly as possible get the video from anywhere in the last month for further analysis. The requirements for availability and performance (several thousand cameras of good resolution is not a pound of raisins, as you understood) in such a project are maximum. If the desired video is lost or not recorded, the operator of the service may, in all seriousness, go on holiday to the resorts of sunny Magadan.

The end customer (service operator) requested, where possible (without detriment to quality), to use Russian solutions where possible, since they were then going to ask him: “what did you do for import substitution?”. He didn’t want to blush in front of the big bosses.



There were no problems with video surveillance systems, because There are many Russian decisions, the choice is large, and in this case a proven solution was used. But with the storage system (since our searches were not successful), we were determined to use a long-known foreign solution. And then one of our project partners suggested using Russian storage AERODISK for the storage level.

We (Softline) at that time, of course, knew that there was such a manufacturer, and that it was not an OEM. We heard different reviews about him: both good and not so much, so we did not have an unequivocal impression. We did not reach its testing, because testing the solutions of other Russian developers (see attempt number two) failed, and we temporarily stopped activity due to constant failures.

But the offer was made, the customer took the idea with a bang. And we went to find out what AERODISK was doing for the storage system, and decided to visit them.

We were quite pleased with the visit to AERODISK. They showed us the system in work, the demo center, and also gave us a chat with the developers, who "make these things with their hands" that make the future.

We asked AERODISK to organize a test lab specifically for this project and, together with the selected video surveillance system, to emulate a productive load. The specificity of the task consisted in the fact that, in addition to the usual streaming video recording, the tasks of reading and rewriting data based on the analysis performed are added to the constant recording stream. Knowing this load profile, we drove the AERODISK data storage system both in the tail and in the mane for several weeks. In general, we were satisfied with the results, the system in some cases even surpassed foreign manufacturers, but there were also shortcomings. But they all boiled down mainly to minor bugs in the interface, which were promptly corrected by those. manufacturer support.

The result of the test was:

  1. Nothing broke, although we broke
  2. In the course of the test, the support worked at a good level.
  3. Performance for our task was sufficient
  4. We realized that we, as a system integrator, can fully support the system ourselves (this is an important criterion for us)

We decided to go to this project with AERODISK storage systems and standard x-86 servers connected to the storage system via Fiber Channel and 10Gbit Ethernet. It was necessary to assemble two fault-tolerant clusters that will be simultaneously serviced by urban video cameras.

Implementation


The design solution was developed from scratch, and really neither we, nor our partner, nor the operator had such experience. It is clear that various best practices and other theory were used, but, as they say in the military, “any plan is good before the first shot”. The project on paper looked perfect and was approved. It was embarrassing that AERODISK did not participate in the design, due to the fact that it appeared at the last moment in the project and it was already impossible to reaffirm something without postponing the deadlines (we cannot transfer the 2018 World Cup deadline))))).

A set of solutions was delivered, set up according to project documents and put into trial operation, which was supposed to take several months. It all started very well and both storage systems for the first month were filled with data from video cameras. The flight was normal, everything is like on paper. We were very pleased, but did not yet know that the real problems were still waiting for us ahead.

We were already relaxed, but at the end of the second month of trial operation, the operator began receiving complaints about the performance and instability of the system as a whole.



We urgently engaged a video surveillance solution provider and AERODISK to solve the problem. At that time, we waited for the ping pong to start, like this:



To our surprise, this did not happen, and both vendors went head over to diagnosing the problem. The next day a diagnosis was issued. The cause of performance problems was in the incorrect configuration of the storage system. It worked that embarrassed us from the very beginning: physically it was not possible to involve the storage system vendor in the design, and it was this section that was designed incorrectly, without taking into account the features of the AERODISK storage system. We were even glad at that moment, because “well, if it’s crookedly adjusted, so let's reconfigure, what's the problem then :)?”

But it was not there. The problem was that the video from the cameras was written mainly on SMB file balls, which were presented from the storage systems to the video surveillance servers, and this was the root of evil, and for the right video surveillance, you need to present the block devices and format them already at the server level local file systems. It would seem, what the problem is, we create LUNs and give them to servers, but no. Since for the first month of operation the entire usable storage capacity of both storage systems was already taken, the LUNs simply have nowhere to create. There is no place, and you cannot delete old videos in order to free up space, for “they will do it”. Well, backups will not help here, which is obvious, and replication was not incorporated into the project.



There was no more budget for buying the same amount (half a petabyte), and the current disk space could not be cleared. The option to use temporary space in our cloud did not fit, since there will be too much delay with such recording volumes. What began so well was coming to a terrible end.



Decoupling


But help still came. AERODISK suggested putting another array of similar capacity and performance next to the storage reconfiguration next to it, redirecting the entire record to it, and waiting for one month, when the data from incorrectly configured storage systems are automatically deleted. After that, while the video data is being written on the temporary storage system, and the permanent storage systems are free, you should perform the correct setup of block access on the free storage systems. Well, and then do everything in reverse order. As you understand, all these operations should be performed “without a single break)))”, that is, without stopping recording from video cameras.

Independently crank such a special. we were a little afraid of the operation. Therefore, AERODISK (for which many thanks to him), apart from the iron, several times sent two engineers to carry out these works.

Switching to temporary storage, reconfiguration and reverse switching was performed without any stops. The performance of storage and complex as a whole has returned to normal. The operator of the service was beaming with happiness, since he was almost resigned to the failure of the project.
To date, the Safe City complex has been commissioned, we have gained invaluable experience, and the software and hardware complex used in this project has been taken as the standard for future applications in other cities of our country.

findings


So, we studied the market of Russian data storage systems, walked through OEM fakes, raw solutions, could still find a serious player in this market - AERODISK, whose product SHD ENGINE N-series can safely compete with ours more eminent foreign solutions. Yes, the implementation of a large and complex project did not all go smoothly (and it can be different?), But the result was, in our opinion, excellent. It is safe to say that Homeland may not be worried about import substitution in the direction of storage systems.


Also popular now: