10 most unexpected data center outages

Published on July 25, 2014

10 most unexpected data center outages

Once, one way or another, the electrical network fails along with the backup systems, or the engineer makes a mistake and the data center “lies down”. And while technology in the industry is constantly evolving, reducing the frequency of malfunctions, sometimes something suddenly goes wrong. But there are times when terrible malfunctions are caused by completely unexpected and surprising reasons.



And here is a list of the 10 most incredible reasons for such failures:

Second error


An extra second is sometimes added to the World Time settings to take into account the change in the Earth's rotation speed. The addition of one second to the global atomic clock in 2012 brought a lot of trouble. Then there were interruptions in the work of popular web resources, including LinkedIn, Reddit, Mozilla and The Pirate Bay. In Australia, an average of 400 hours was delayed for two hours due to the fact that employees had to switch to manual check-in of passengers.



Squirrel at Yahoo


In fact, proteins that cause malfunctions in the data center, not to say so completely rare. They bite everything in a row, including the wires that are used to transfer data. In a 2010 report at the O'Reilly Velocity conference, Mike Christian, who runs Yahoo’s business continuity service, said: “Toasted squirrel has disabled half of our Santa Clara data center.”



Relocation


Transporting servers can be quite troublesome. NaviSite, which is now owned by Time Warner, acquired the Alabanza hosting provider in 2007 and had to move the servers from their main data center in Baltimore to its site in Andover, Massachusetts.

In fact, they disconnected the servers, loaded them into trailers, and transported the servers over 420 miles. Many sites hosted by Alabanza did not respond to requests while the physical movement and commissioning in the new location lasted.

Another relocation issue occurred a few months earlier when Hostway moved ValueWeb servers from Miami to Tampa. Hostway after noted that more than 500 servers experienced hardware problems after installation on a new site.

Anchor against the Internet


Between the continents, the signal is transmitted through powerful submarine cables . These cables are quite durable, given their location. However, there was one case where a ship simply dropped anchor on one of the cables. In 2008 there was a whole epidemic of damaged submarine cables, which led to interruptions not only in the operation of the data center, but also left whole regions without communication.



"Every wall is a door." - Ralph Emerson


Nianet, a Danish Internet service provider, disconnected when thieves punched a hole in the wall of the data center in Tostrup to get inside and steal something. According to media reports, they left, capturing a bunch of network cards. How thieves were able to break through the wall and why they took only network cards is still a mystery.

Be careful when throwing gobies


There was at least one case where the work of a data center was disrupted due to a smoldering mulch. The Perth iX data center in western Australia stopped operating for an hour after the VESDA (Very Early Smoke Detection Apparatus, very early smoke detection) fires. It turned out that the system caught the smoke of smoldering mulch (various biomaterials such as foliage, compost, etc.) from a garden bed adjacent to the outer wall of the object, which caught fire, most likely due to an unquenched cigarette butt.

Keep driving




In 2007, Rackspace, a company known for its phenomenal uptime, experienced several hours of downtime due to a truck driving into one of the power transformers, followed by an explosion.

The backup power system tried to support the operation, but the two coolers failed to start. Several large sites did not work on the network at the same time.

Czech and mate, Internet


In 2009, the carelessness of a little-known Czech Internet provider in BGP settings caused a number of small interruptions in the work of several large hosting companies. Czech provider Supronet “has single-handedly set up a global crisis on the Internet for several hours,” Renesys said.

Servers at the pawnshop


In 2007, two masked men broke into the data center in Chicago and stole a bunch of computer equipment. This data center was owned by the old hosting company CI Host, which currently does not exist on its own.

It is reported that an employee who worked alone on the night of the robbery was hit by a stun gun, while others say he was beaten. More than 20 servers were handed down, which in fact destroyed many sites.

After the robbery, there were many rumors about what had happened. It was rumored that the robbers got inside with a saw (which would happen to Nianet a few years later). CI Host said that two robbers hid in a mechanical cabinet (elevator). It was believed that the company itself arranged this crime with the aim of committing fraud, but these rumors were not confirmed.

The most amazing thing in this whole thing was that this was not the first time.

“People are talking about four robberies, which is a mistake!” The company’s CEO said to the WHIR magazine. “Robbery is when property is taken away by force or threats. Technically, CI Host was robbed just twice in two years. Twice more were hacks, things were stolen, but these were not robberies. ”

Super Hurricane Sandy




It's hard to talk about Hurricane Sandy. We hope that this out of the ordinary phenomenon will no longer suit New York’s chaos. But after the disruption of the data center caused by this storm, we heard some amazing stories, such as the story of the Team with buckets (Peer 1 Hosting carried buckets of diesel fuel at a height of 17 floors to start the emergency generator of the data center of the company).

Usually, reaching so far north, the storms almost disappear. However, Sandy moved north, taking on some of the features of tropical storms and gaining more and more strength. A strong current directed the storm to the coast, there was a full moon, the tide was stronger than usual, the waves were just huge.

A storm of this type has never been observed in the Northeast. This is even more surprising than a squirrel roasted in power cables, a smoldering mulch, a truck driving off the track. Some things in the world we can never foresee.