January 26, 2018 at 09:34

Personal experience: how we transferred infrastructure from one data center in the USA to another



    We at King Servers are faced with the need to change one of our foreign data centers. The data center was located in the US state of California, and it was necessary to look for a new one in the same region. Today we will talk about how the move went, share our impressions of it and give a couple of tips to those who have yet to face the infrastructure sites in the United States.

    Why change the data center


    The development of the company's infrastructure gradually led to the fact that we were no longer satisfied with the conditions of our current data center. Despite the fact that the data center had the Tier2 standard, literally in a year we encountered a number of problems, such as power failure in the rack part or the complete loss of the Internet channel.

    In addition, the site was not Carrier-neutral - in practice, this meant that communication services were provided only by one company, which owned the data center. Therefore, at the time of failures on its channels, the connection disappeared completely, which could not suit either us or our customers.

    After weighing the pros and cons, we decided to migrate the infrastructure to a new data center, in which it would be possible to use two independent power lines at the same time and connect to several providers.

    Relocation process


    Both data centers were not far from each other, at a distance of 20 km. Old in Fremont, and new in Santa Clara.



    We thoroughly prepared for the move - we had previously purchased supplies necessary for mounting the servers in racks, new power cables and patch cords in order to reduce the time for dismantling in the old data center.



    Reserves for relocation

    We also installed a second router in the new data center and organized the simultaneous operation of our networks in both data centers.

    As a result, a plan was developed, involving a phased transfer of infrastructure to a new location - so we could continue to work and not cause unnecessary inconvenience to users.

    Initially, we wanted to use the services of a specialized company for transporting equipment, but in the process of negotiations with several companies we found out that they could transport our equipment either in large batches with high downtime for customers, or in small parts with minimal downtime, but the price under these conditions increased significantly. As a result, it was decided to send several of our engineers across half the globe and do the work on our own. Even taking into account flights, accommodation and rent of transport, such a step came out 2 times cheaper than the minimum price from California moving companies.

    The first stage of the work included going to the old data center to assess the infrastructure - not all the engineers of our team had been there before, but some had come for the last time a couple of years ago, and during this time a lot of different work was done by the data center staff. It was also necessary to check the installation of the equipment - as it turned out, not everything was as good as on our last visit.

    Excessive cables were used in the racks; some servers were mounted without using a sled and tightly bolted to the rack. It became clear that it would take much more time to dismantle than we planned. Fortunately, there were enough spare materials to do without completely dismantling all the iron, and we still fit into the migration schedule.



    This is not to say that we were satisfied with what we saw in the old

    data center. Then an exploratory trip to the new data center followed in order to look around and explore the new racks.

    During the first visit to the new data center, our engineers first of all appreciated its manufacturability. Everything was very strict in the old data center - the data center employees not only meticulously checked the documents of the engineers, but also accompanied them when moving to and around their racks. This entailed temporary delays, as often it was necessary to wait until such an accompanying person was free. The new site turned out to be much more democratic in this regard, which allowed us to work faster.

    According to the plan, the team of engineers had to come to the old data center, pick up 15-20 servers, then install them in racks in a new place, turn them on, and after that everything should work without requiring additional intervention on our part.

    It was not without difficulties - for example, it suddenly became clear that the Cisco router that we planned to use to organize the tunnel did not work as planned, and after transporting the test part of the servers that were not used by clients and testing, it became clear that it could not withstand client traffic.

    I had to quickly change the network diagram and use additional routers to share the load from clients and allow the network to last a couple of days during the migration. The setup time was no longer part of any of our plans, which nevertheless caused a deviation from the schedule.

    Also, not all servers started immediately after transportation. Some of our clients administer dedicated servers on their own and close all access to us, including monitoring. On some servers, the disks were not in the best condition, and even a careful shutdown with the correct shutdown of the system did not save from boot failures due to damaged file systems and broken raids.

    In each new batch of transported servers, a couple of similar problems were encountered, and the technical support engineers after the migration of each batch of servers carried out work to restore their performance.



    Total: general impressions


    Everything is very strict in American data centers, getting even minimal access to your own equipment is not easy - you need to get approval for every action that can only be given by a manager who is not necessarily physically located in the DC. Then you need to create a ticket in the helpdesk system and wait for its permission.

    You can also meet fairly stringent conditions, for example, on the use of data center resources - so in our new data center there is a restriction prohibiting a long time (more than 1-2 days) from leaving your hardware in a warehouse. We needed more time, and in order to get permission for a longer storage, we had to get the approval of the manager according to the scheme described above. ID (passport or rights), as in principle in the USA, is constantly asked in local data centers, you also need to be prepared for this.

    On the other hand, from an organizational point of view, the use of DC in California was very pleasant. All conditions stipulated in the contracts have always been fulfilled, which we have not always been able to achieve in Russia. For example, once our move to a new data center failed because its engineers did not have time to prepare the racks - although the preparation dates were clearly stated in the contract (we talked about our adventures in this material ).

    In addition, local engineers turned out to be very friendly and always came to meet us - for example, when we had a shortage of cables so that we would not interrupt and go to the store for new ones, colleagues allocated us resources from their stocks.

    Conclusion: what you need to know when working with American data centers


    In conclusion, once again we list a few basically

    • In the USA, not all data centers are very technically advanced - therefore, you should not choose the first option that you get, but devote more time to searching. Perhaps the cost of services of a more innovative data center will be more expensive, but with certain sizes of infrastructure, better conditions can bring savings in repair, maintenance and replacement of iron.
    • It is necessary to strictly follow the rules - the USA is such a country where laws are enforced, even if it does not always seem logical. But if the rules say that the client cannot move away from his desk in two steps, then to do this, you will need to obtain the permission of the manager - and simultaneously sign more than one paper and show the ID several times.
    • Everyone is very friendly, but in any case you need to rely only on yourself - the famous American positivity is not a myth and extends to data center engineers who are always willing to help. However, this does not mean that you need to relax - for example, no one will correct the mistakes in preparing the migration for you.

    More useful links and materials from King Servers :


    Tags:

    Also popular now: