As I flew across the country, implementing a project for several thousand jobs
The biggest head of Lenin in the world. Soviet Square in Ulan-Ude
When you take on a two-year project with geography across the country, you expect problems with coordination, strange relationships with contractors, movers with servers in hand, unrepresented ports for TK and much more. My life was preparing for this: everything is perfectly described in the methodology of preparing project managers.
Everything was not in vain.
Such stories are very good to tell after a couple of years in a smoking room, but not to meet on projects.
What should have been done
In the autumn of 2016, one large organization finally decided to switch to VDI access in the workplace. Their application software has changed, but computers have not. Therefore, in the future, it all began to inevitably slow down. Having estimated the cost of updating the PC park, we decided to immediately take the next step. And they went to VDI.
In 2017, we completed 5,000 jobs in several regions.
The solution architecture was defined at the time of the competition. It is as follows:
As you can see, there are 5 server nodes (this is located in data centers and on separate platforms), one for each group of regions. Among themselves, these five sites are connected via VPN through the corporate network of the organization (there is a certain node with production servers in the center). At the bottom of the scheme - the types of jobs.
Soft:
Kernel - hyperconvergent architecture with VSan, VMWare - hypervisor up to version 7. Eleven servers: 3 management servers, 8 compute, switches. Physically, this is one rack: crypto tools, control network, controllers, vSAN, production. Each rack can hold up to 5 thousand users. Plans for the next year - a doubling.
Remote maintenance in the contract, but it is necessary to go down: you can not agree with a number of things safe. We keep five employees who are escorted from the center. Plus, we have subcontracts in all regions to come, to replace the piece of iron. Spare parts do not store, everything is duplicated. We buy from manufacturers of iron the next working day, vendors store spare parts.
From the user's point of view, we don’t touch anything on the final workplaces, except that we put another icon on the desktop. From it they connect. At some point, their administrators will decide that it is enough to update the software on local disks and, when users get used, leave only this icon.
Start
At the beginning of the work, we knew that organizations are quite expensive to put their own optics (which is logical), so she buys channels from long-distance providers. In most cases, this is not a problem, but in Kamchatka, for example, they have a satellite gate. This creates certain surprises for VDI. Running a little ahead, the project has not yet been distributed there, but in a number of cities we have very narrow channels.
Regarding the work with the application, users are quite undemanding to the strip - there are static screens, and only the cursor moves. However, they often send scans of documents, for example, passports for a client profile. This creates an additional load on the channel.
The three base sites are their own server nodes in their own downtown offices. Two more are external data centers where fences for this company are made.
Typical difficulties
The first stage is a survey. You go on points and look with your eyes, where that will stand. You pretend how and what: you communicate with local IT specialists, check the rules of work at the sites, look where to put the boxes, who will lift things to the 4th floor, do you need movers, as with passes for them. We traveled together: I, PM for the organization, and the architect (he is the technical manager of the project). Some places needed additional routers. At two sites there were no necessary providers, the customer went to negotiate with them, so that they stretched the link there.
It was necessary to solve the above problem with the channel loading scans. This is what happens: the entire stream from the scanner is sent to the server node, and already there is processing. In fact, it was necessary to do preprocessing in the final subnets before leaving the corporate network. There are rules for scanning so that the document is distinguishable. There are VMware tools - we have attracted their experts for optimization, testing and acceleration. They poshamanil, made pre-tuning and rendered a verdict: you can do it faster than you can, only by expanding the channels.
The same job was on printing.
There were problems with the drivers of old models of multifunction devices. Each type image was made together with people from the region. In each region we were accompanied by our own team. We gave out a test image, they stuffed with firewood, software, tested that it all worked, further into the combat pool. For each of the regions in the workplace was created its own basic image. They differ in a set of drivers for printer scanners and a bit of software. Users of one region work with one way. A person connects to and from this virtual machine and goes to his applications. Users do not go to "foreign" regions.
The main difficulties that arose when setting up VDI. The security men for a very long time and stubbornly did not want to coordinate in principle the connection. The reason is simple: the current regulatory framework for the registration of signatures is lagging behind reality. We had to clarify, redo and harmonize business processes. That is, we took and started writing their own work regulations.
After the survey and imaging it was necessary to just ride and place the equipment. We knew that surprises awaited us.
Atypical problems
So, the capital of one of the Siberian regions ... here is the data center, this is your place. Everything is ready, except that there is no power inside the fence. People who are responsible for the data center, we were assured that in order to make the outlet inside the fence, you must turn off the entire data center. It is necessary to coordinate with the tax and other comrades so that on this day there would be no wages, taxes and nothing at all critical. It is better to do it in six months.
We started planning. Along the way, we found a contractor who built this data center. In general, for me it is very strange that the power points did not immediately do, one costs up to 5 thousand rubles with automatic guns. The room is well done, everything is on Schneider. It turned out that APC technological sockets are supplied on request, because few people need them. Order time 11-14 weeks. They are not anywhere. But we, as in that joke, were rescued by a Russian ravage: at some point the contractor became more friendly and remembered that they had one unrecorded in the bins. And it was not necessary to disconnect the data center. This year we are expanding again, there will be no outlet again - the customer promised to make a decision whether to stay on the site or change it 2 weeks before the equipment starts.
But what happened in the million-plus city was an industrial data center, positioning itself as very cool. There was no place to throw out the trash last year. The box from the server remained - take out yourself. And you're on a business trip. You order a container, take it out. Well, at least we, taught by bitter experience, have cash in advance in case of an emergency. On the spot, it turns out to be done quickly, but with accounting then closing all these costs is a whole adventure. Try to justify the order for the removal of the garbage truck in travel expenses. Before that, there was a problem a few years ago: they ordered aerial platforms for maintenance of air conditioners. The same trouble.
Evening departures. At 4 am Moscow time, the start, the flight across the whole country, at 8 am, flew in and immediately went to the meeting, the customer meets you. From there to the object. And so the second working day went right after the first. And let's see, but let's do it - you can't seem to refuse. You need to plan an extra day, always with such flights.
Muscovites who come to you to tune everything belong in two ways. On the one hand, they don’t trust them completely, because they are a stranger. On the other hand, they very actively want to help, because they know that work is going on in different regions. Therefore, all that can be solved on the spot - decide. And still try to show the city. While you do not know anyone - cautious, benevolent attitude. For them, such projects are a small holiday, because IT people can ask for something in the federal center. All purchases from them are centralized: they write a justification in the center, and there they buy. Under our project, all the rationales "make it wider and thicker than Inet", "make more server room" and so on have passed.
There was a lot of work on communication between their departments. On the part of the customer, many departments and departments participated in the project, they are all independent. Each of the related departments will work only on internal orders despite all this pilot project. You must, in spite of the fact that they do not communicate or communicate badly - to communicate with everyone, to get acquainted, to annoy, to write. As a result, it turns out that to engage in the internal organization of their work. In a couple of cities we wrote letters for them, which they sent to other people within the organization.