About the data center in all honesty: how we solved the problem of dust in the server rooms of the data center
Hello, Habr! I’m Taras Chirkov, Director of Linxdatacenter Data Center in St. Petersburg. And today in our blog I’ll talk about the role in maintaining the cleanliness of the room in the normal operation of a modern data center, how to properly measure, achieve and maintain it at the right level.
Purity trigger
Once a client of a data center in St. Petersburg approached us about a layer of dust in the lower part of the equipment rack. This was the starting point of the investigation, the first hypotheses of which suggested the following:
- dust enters the server rooms from the soles of the shoes of employees and customers of the data center,
- entered through the ventilation system,
- both.
Blue shoe covers - to the dustbin of history
Started with shoes. At that time, the problem of cleanliness was solved in the traditional way: a container with shoe covers at the entrance. The effectiveness of the approach did not reach the desired level: it was difficult for guests to control their use by the data center guests, and the format itself was inconvenient. They quickly abandoned in favor of more advanced technologies in the form of a shoe car. The first model we installed for such a device failed: the machine too often tore the shoe covers when trying to put them on shoes, its use was more annoying than making life easier.
Turning to the experience of colleagues in Warsaw and Moscow did not solve the problem, and as a result, the choice was made in favor of the technology of fusing thermal film on shoes. With the help of thermal film, you can put on “shoe covers” on shoes with any sole - up to a thin female heel. Yes, the film also sometimes slides, but much less often than the classic blue shoe covers, and the technology itself is many times more convenient for the visitor and more modern. Another important (for me) plus - the film easily covers the largest sizes of shoes, unlike traditional shoe covers, torn when trying to put them on the 45th size. For greater modernity of the process, ballot boxes were installed with an automated opening of the lid by a motion sensor.
This process looks like this:
Guests immediately appreciated the innovation.
Dust in the wind
Having put in order the most obvious channel of possible pollution of space, we took up the finer matter - air. It is likely that a significant portion of the dust enters the server rooms through ventilation due to insufficient filtering, or is brought in from the street. Or is it all about poor quality cleaning? The investigation continued.
We decided to take measurements of the content of particles in the air inside the data center and invited a laboratory specializing in air quality control in cleanrooms for special purposes to carry out these works.
The laboratory staff measured the number of control points (20), compiled a sampling schedule to track the dynamics and create the most accurate picture. The cost of the full process of measurements by the laboratory amounted to about 1 million rubles, which seemed completely inexpedient to us, but it gave a number of ideas for independent implementation. Along the way, it became clear that the laboratory is good, but analyzes need to be carried out in dynamics and constantly accessing their services is extremely inconvenient.
Having looked at the planned actions of the laboratory, we decided to look at more utilitarian devices for independent work. As a result, we managed to find the instrument necessary for this task - an air quality analyzer. Here's one:
The device shows the content of particles of different diameters (in micrometers).
Revising standards
This device analyzes the number of particles, temperature, humidity and shows the results in units of measurement, respectively, with ISO standards for this parameter. The display shows the levels of particles with different diameters in the air sample.
At the same time, they sinned on filters: at that time, G4 filter models were used inside server rooms. This model provides a rough air purification, therefore, the probability of the passage of particles leading to contamination was assumed. We decided to purchase F5 fine filters used in air conditioning and ventilation systems as second-stage filters (after-treatment) for testing.
The investigation was carried out - you can proceed to the control measurements. They decided to use the requirements of the ISO 14644-1 standard for the number of suspended particles as a guide.
Classification of clean rooms by the number of suspended particles.
It would seem - measure and compare according to the table. But not everything is so simple: in practice, it turned out to be rather difficult to find air purity standards for server rooms of the data center. This is not explicitly stated anywhere, by any organization or industry institute. And only at the internal forum Uptime Inside Track (access to it have people who have been trained under the programs of the Uptime Institute) there was a separate discussion on this topic. Based on the results of its study, they were inclined toward focusing on the ISO 8 standard - the penultimate one in the classification.
The very first measurements showed that we underestimated ourselves - the results of internal air inspections yielded compliance with the requirements of ISO 5 in indoor premises, which significantly exceeded the standards required by the participants of Uptime Inside Track. At the same time - with a large margin. We have a data center, and not a biological laboratory, of course, but in order for the concentration of particles in the air to be equal to ISO 8, this should be an object of the class “minimum cement plant”. And how you can apply the same standard to a data center is not very clear. At the same time, we got the result at ISO 5, making measurements when filtering the air with G4 filters. That is, dust cannot get into the racks through the air, the F5 filters turned out to be excessive, and they were not even used.
A negative result is also a result: we continued to search for the causes of pollution in other directions, and included air quality control in quarterly inspections, combining them with inspections of BMS sensors by verified devices (ISO 9000 requirements and customer audits).
Below is an example of a report that is filled out on the basis of data obtained during measurement. For greater accuracy, measurements are made by two instruments - Testo 610 and BMS sensor. The table heading indicates the limit values for the devices. Deviation of these parameters is automatically highlighted to facilitate identification of problem areas or time periods.
Everything is clear with us: the difference in instrument performance is minimal, and the particle concentration is much lower than the limit.
Through the back door
Since there were other entrances to the clean rooms besides the main entrance for customers, where we installed a shoe machine, there remained the need to exclude dirt from entering the data center through them.
It is inconvenient to wear / remove shoe covers during procedures for unloading equipment, so we found a machine for cleaning soles. Conveniently, functionally, but the human factor affects in the form of an optional approach to this device. In fact, the same as with shoe covers at the main entrance.
To solve the problem, they began to look for cleaning options that cannot be avoided: sticky carpets with separable layers handled this best. In the process of authorization at the front door, the visitor must stand on such a rug, removing excess dust from the soles of his shoes.
Cleaning ladies tear off the top layer of such a rug every day, there are 60 layers in it - enough for about 2 months.
Having visited Ericsson's data center in Stockholm, among other things, I paid attention to how they solve these issues there: along with tear-off layers, Sweden uses reusable Dycem antibacterial carpets. I liked the idea because of the principle of reusability and the ability to provide a large coverage area.
Magic antibacterial carpet. It’s a pity, not a plane, but it could - at such a price!
We hardly found company representatives in Russia and estimated the cost of the solution for our data center. As a result, we got a figure almost 100 times more expensive than solutions with multi-layer carpets - about the same 1 million rubles as in the project with measurements of air purity. In addition, it became clear the need to use special cleaning products, naturally available only from this manufacturer. The solution also fell away by itself, settled on a multilayer version.
Manual labor
I would especially like to draw attention to the fact that all these measures did not cancel the use of the work of cleaners. In preparation for the certification of the Linxdatacenter data center according to the Uptime Institute Management & Operations standard, it was necessary to clearly regulate the actions of the cleaning service employees in the data center. Detailed instructions were drawn up prescribing where, what, and how they needed to be done.
A couple of excerpts from the instructions:
As you can see, everything is registered, literally every aspect of work in a particular room, cleaning agents, materials, etc. that are acceptable for use. Not a single detail is left without attention, even the smallest. Briefing - under the signature of each employee of the service. In server rooms, electrical rooms, etc. they are cleaned only in the presence of authorized data center employees, for example, an engineer on duty.
But that's not all
Also in the list of measures to guarantee cleanliness in the data center: rounds with visual inspection of the premises, including weekly inspections of the racks to detect scraps of wires left inside them, packaging residues from equipment and components. For each such episode, an incident is initiated, the client receives a notification about the need to eliminate violations as soon as possible.
Also, for unpacking and setting up equipment, we created a separate room - this is also part of the company's cleaning policy.
Another measure we have learned from Ericsson's practice is to maintain a constant air supply in server rooms: there is more pressure inside the rooms than outside so that there is no traction inside - this solution will be described in more detail in a separate article.
Finally, we got ourselves the assistant robots for rooms, which are excluded from the list of cleaning staff available for visiting.
The grille on top not only gives +10 protection to the robot, but also allows it not to get stuck under the vertical cable trays of the racks.
An unexpected find as a conclusion
Cleanliness in the data center is important for the operation of server and network equipment that draws air through itself. Exceeding the permissible dusting levels will lead to the accumulation of dust on the components and the total temperature increase up to 1 degree Celsius. Dust reduces the cooling efficiency, which in terms of the year can turn into noticeable indirect costs, as well as affect the fault tolerance of the facility as a whole.
We can say that this is a speculative assumption, but the experts at Uptime Institute, who certified the Linxdatacenter data center according to the Management & Operations Quality Standard, pay serious attention to cleanliness. And it was all the more pleasant to receive the most flattering assessments in this area: our data center in St. Petersburg seriously exceeds the certification requirements. The institute’s expert called us “the cleanest data center he has seen,” moreover, our data center is used by Uptime as an example of resolving the issue of clean server rooms. Also, we easily pass any client audit on this parameter - the most serious requirements of the most capricious customers are met with excess.
Let's get back to the beginning of the story. Where did the pollution come from in the very complaint from the beginning of the article? The part of the customer’s counter that caused the launch of the entire project “Clean in the Data Center” was contaminated from the moment the counter was imported and installed in the data center. The client did not clean the rack by the time of entry into the server room - when checking neighboring racks installed at the same time, it turned out that the situation was the same with dust. This situation was the reason for adding a cleaning control point to the client checklist for installing the rack. The likelihood of such things must also never be forgotten = forewarned, therefore armed. It's all about “purity and dictatorship” in our data center, in the next article I will talk about pressure sensors, but for now ask questions in the comments.