August 24, 2010 at 15:26How to make cloud (cluster) hosting for a couple of cents *
Three years ago I had an interesting task. It was necessary to assemble a platform that combined several racks with servers into a single whole, for the dynamic distribution of resources between sites written for the LAMP platform. Moreover, so that interference in the code of sites is minimal, and even better - is generally absent.
At the same time, no expensive solutions like the Cisco Content Switch or the optical fiber disk shelf can be used - there was not enough budget.
And besides, of course, in the event of a failure of one of the servers, this should not have affected the operation of the platform.
First of all, you need to divide the creation of the platform into subtasks. It’s immediately clear that you will have to do something to synchronize the data, since the shared disk is not available. In addition, you need to balance traffic and have some statistics on it. And finally, automation of the provision of the necessary resources is also a rather serious task.
I had a choice of how to organize the platform. These are OpenVZ and XEN. Each has its pros and cons. OpenVZ has a smaller overhead, works with files and not block devices, but does not know how to run anything other than Linux distributions. XEN allows you to start Windows, but it is more difficult to work with. I used OpenVZ, since it was more suitable for solving the problem, but nobody limits you in the choice.
Then I divided the server into places under VDS, one for each core. The servers were different, so I had a set of 2 to 16 virtual machines on each of the servers. In the “average ward” ~ 150 virtual machines were released per rack.
The next item is the on-line creation of VDS on demand + protection against breakdown of any server. The solution was simple and beautiful.
For each VDS, an initial image is created in the form of files on the LVM section. This image "spreads" across all platform servers. As a result, we have backup of all projects on each server (paranoid cries of emotion), and creating a new VDS “on demand” was simplified to snapshot from the image and start it in the form of VDS (it takes just a few seconds).
If the integrity of the files was simple, then the situation with the synchronization of the database was worse. From the beginning, I tried the classic example - master-slave, and ran into the classic problem: slave lagged behind master (yes, and thanks for replicating in one thread, thank you very much).
The next step was Mysql-Proxy. As a system administrator, this solution was very convenient for me - I set it and forgot it, only the config needs to be updated when adding / removing new VDS. But the developers had their own opinion. In particular, it is easier to write a certain PHP class for synchronizing INSERT / UPDATE / DELETE queries than to learn Lua without which Mysql-Proxy is useless.
The result of their work was the so-called API, which was able to find neighbors with a broadcast request, synchronize to the current state and inform the neighbors about all changes to the database.
But still, it's worth exploring Lua and making a native mode of operation when all requests are synchronized with neighbors.
Balancer is a key aspect of the platform. If the balancing server crashes, then all the work will not make any sense.
That's why I used CARP to create a fail-safe balancer, choosing FreeBSD as the OS and Nginx as the balancer.
Yes, the expensive NLB has been replaced by two weak machines with FreeBSD (marketers are furious).
When the platform started, one copy was launched for each site and monit monitored on the balancer to ensure that the primary copy always worked.
In addition, the Awstats statistics analyzer was installed on the balancer, which provided all the logs in a convenient format, and most importantly - there was a script that polls each VDS via SNMP for its load.
As we remember, I allocated one core for each VDS, therefore Load Average of 1 - this will be a normal load for VDS. If LA was 2 or higher, a script was run that created a copy of VDS on a random server and registered it in the upstream of nginx. And when the load on the additional VDS fell less than 1 - accordingly, everything was deleted.
If you take a rack with servers and a switch supporting the CARP protocol, then to create a cloud hosting you will need:
* Amounts with four zeros are enough to fill the rack. Compared with solutions from brands where the price of one rack is the sum with six zeros, this is really a couple of kopecks.
At the same time, no expensive solutions like the Cisco Content Switch or the optical fiber disk shelf can be used - there was not enough budget.
And besides, of course, in the event of a failure of one of the servers, this should not have affected the operation of the platform.
Goal for inventions of cunning
First of all, you need to divide the creation of the platform into subtasks. It’s immediately clear that you will have to do something to synchronize the data, since the shared disk is not available. In addition, you need to balance traffic and have some statistics on it. And finally, automation of the provision of the necessary resources is also a rather serious task.
Let's start from the beginning, let KO come with me
I had a choice of how to organize the platform. These are OpenVZ and XEN. Each has its pros and cons. OpenVZ has a smaller overhead, works with files and not block devices, but does not know how to run anything other than Linux distributions. XEN allows you to start Windows, but it is more difficult to work with. I used OpenVZ, since it was more suitable for solving the problem, but nobody limits you in the choice.
Then I divided the server into places under VDS, one for each core. The servers were different, so I had a set of 2 to 16 virtual machines on each of the servers. In the “average ward” ~ 150 virtual machines were released per rack.
How to sync data
The next item is the on-line creation of VDS on demand + protection against breakdown of any server. The solution was simple and beautiful.
For each VDS, an initial image is created in the form of files on the LVM section. This image "spreads" across all platform servers. As a result, we have backup of all projects on each server (paranoid cries of emotion), and creating a new VDS “on demand” was simplified to snapshot from the image and start it in the form of VDS (it takes just a few seconds).
Base and API
If the integrity of the files was simple, then the situation with the synchronization of the database was worse. From the beginning, I tried the classic example - master-slave, and ran into the classic problem: slave lagged behind master (yes, and thanks for replicating in one thread, thank you very much).
The next step was Mysql-Proxy. As a system administrator, this solution was very convenient for me - I set it and forgot it, only the config needs to be updated when adding / removing new VDS. But the developers had their own opinion. In particular, it is easier to write a certain PHP class for synchronizing INSERT / UPDATE / DELETE queries than to learn Lua without which Mysql-Proxy is useless.
The result of their work was the so-called API, which was able to find neighbors with a broadcast request, synchronize to the current state and inform the neighbors about all changes to the database.
But still, it's worth exploring Lua and making a native mode of operation when all requests are synchronized with neighbors.
Glory to FreeBSD
Balancer is a key aspect of the platform. If the balancing server crashes, then all the work will not make any sense.
That's why I used CARP to create a fail-safe balancer, choosing FreeBSD as the OS and Nginx as the balancer.
Yes, the expensive NLB has been replaced by two weak machines with FreeBSD (marketers are furious).
And most importantly, how it worked
When the platform started, one copy was launched for each site and monit monitored on the balancer to ensure that the primary copy always worked.
In addition, the Awstats statistics analyzer was installed on the balancer, which provided all the logs in a convenient format, and most importantly - there was a script that polls each VDS via SNMP for its load.
As we remember, I allocated one core for each VDS, therefore Load Average of 1 - this will be a normal load for VDS. If LA was 2 or higher, a script was run that created a copy of VDS on a random server and registered it in the upstream of nginx. And when the load on the additional VDS fell less than 1 - accordingly, everything was deleted.
I summarize
If you take a rack with servers and a switch supporting the CARP protocol, then to create a cloud hosting you will need:
- Explore Lua and configure transparent synchronization through Mysql-Proxy
- Screw billing to account for additional copies of VDS and traffic
- Write a web interface for managing VDS
* Amounts with four zeros are enough to fill the rack. Compared with solutions from brands where the price of one rack is the sum with six zeros, this is really a couple of kopecks.