September 3, 2014 at 13:46VPS Search - Autumn Update
Autumn has come, so I had a desire to report on the work done over the summer on the project poiskvps.ru
I think that almost every webmaster or server owner encountered a problem when a backup copy of the data was needed, but for some reason it did not happen. On a common shared hosting, it is accepted that the hosts themselves make backups and in which case they are ready to provide them to users. However, when renting a virtual or dedicated server, the situation is exactly the opposite: the owner needs to independently monitor the safety of data and perform backups on their own. Sometimes hosters offer “boxed” solutions, when together in the server they give the user the opportunity to rent a place to store backups. In my opinion, this is not the best idea, since backups need to be stored at least in another data center, and preferably in several. For example, you can recall the fire in the hosting data center. ua - no one is safe from this set of events, but such a case shows that storing backups in the same data center is not the best idea. Also, in my opinion, storing backups with the host is also not the best idea, even if backups are stored far from the main server. Recently, it is not uncommon for a hoster to simply disappear or seize all equipment.
The backup space rental service has been operating for less than half a year, but there have been significant changes in it recently. Today I want to talk in detail about them and about the service. Initially, the idea was to make a simple and convenient server for storing backup copies, at the same time I wanted the price per gigabyte to be sky-high. For this, a rather well-known data center was chosen - hetzner.de - and until recently it was possible to buy a place only in this data center in Germany at a very “tasty” price: 10 Gb costs 30 rubles, 50 Gb - 100 rubles and more proportionally.
However, a few days ago it became possible to rent storage in addition to Germany on servers in Russia, in St. Petersburg, Selectel DC. This may be relevant for those users who want to make backups at a very good speed. The traffic is not limited, the connection speed is 100 Mbit / s, there are no artificial speed limits. The price is similar to accommodation in Germany. Plus, until the end of September there is a constant 10% discount on any volume.
For the user, the repository is a login-password pair with which you can manipulate files using the FTP protocol. The service does not offer any other ways of exchanging information, since the most common control panels support FTP downloads, and the service is primarily focused on simplicity.
On the technical side, a Debian OS and an ISPmanager control panel are installed on each server. Using the API of this control panel, all actions are performed with the user account: activation, password change, receiving information about the occupied space and receiving a list of files, setting restrictions on IP addresses.
The activation of the service can be divided into two logical parts:
1. Generation of a username and password and creation of a user with the necessary restrictions on the server.
A small explanation of why the username and password are generated, instead of the usual suggestion, come up with them yourself.
The result of the php function uniqid () is used as a login; accordingly, at any moment of time this value is unique, it is valid for creating a new user in the system, and it is difficult to pick it up. Also, a construction of the form login.backup.poiskvps.ru is always used as the server address; accordingly, the login must also be a valid subdomain. In order not to “load” the user with unnecessary information, it was decided to use just such an approach, otherwise the user would have to select the correct login for a long time so that it would satisfy all the requirements. All the same, these data are used only during configuration and forget about them.
The password is generated by the service and is not requested from the user due to the fact that the activation of the service may be delayed: that is, the user ordered the service, but paid for it in a day. And in order not to store the password in clear form for subsequent activation of the store, the password is automatically generated immediately after payment and sent to the mail. Naturally, nothing prevents the user from immediately changing the password. Using AJAX on the site makes this pretty easy.
2. Creating the right subdomain to activate the server address of the form login.backup.poiskvps.ru. If you need to move a user’s account from one server to another (for example, to better manage the occupied space), the best way is to give each user a subdomain. If, for some reason, the user account moves to another server, it will be enough to change one entry in the CSN, and the user will not notice anything. The time for updating DNS records has been set quite short, so that by the time you receive an email with a login and password, DNS records already have time to update, and the user successfully logs on to the server.
Immediately after activating the service and receiving the generated password, any paranoid will have a desire to change this password. Everything is pretty thoughtful here: you can change the password directly from the control panel, and changing the password is deprived of annoying functions, for example, you do not need to type the password twice, and there is only one “quality” limit: the password must be 8 characters or more. That is, the user will never see any notifications “There must be a large vowel letter in any even position in the password”.
Well, and as a bonus - the password is changed without reloading the page, so a pretty picture will indicate that data is being exchanged for about a second, and as soon as the response is received from the destination server that the password has been changed, the user will be notified immediately.
Most often, backups are set up once and forget about it, until the time comes to use backup. Agree, it will be very sad to know that there are no backups for the last three months, since all available space has been taken up. Of course, it is not in our power to foresee, and even more prevent all cases of “corruption” of backups, but the issue of occupied space can be brought to perfection.
The principle of operation is as follows: the user indicates the percentage of occupied space, after filling in which he needs to send a notification that there is not enough space left.
The service has one of the “background” processes, which once a minute receives information from all servers and updates statistics on free and occupied space. With this function, you can just see how much was occupied before the update and how much was found after, and if the threshold specified by the user is overcome - you need to send a notification to the user.
In theory, this is a good plan: a clear notification without delays, and the user immediately learns that there is not enough free space left. However, realities add a bit of negativity: most often control panels make backups as follows: first they download the “fresh” version of the backup, and only after that they delete the oldest one. Because of this, at the time of uploading to the storage, free space may run out, and the user will receive a notification each time, although after the file has been downloaded, the free space will reappear. To avoid such cases, the following mechanism is used: as soon as the threshold specified by the user is over, a checkbox is put in the database, and after an hour the occupied space is checked again: if there is still not enough space, the user receives a notification,
In order to increase security on the service, it is possible to set a restriction on IP addresses. If the server supports IPv6 protocol, then you can set a restriction on addresses of this type.
Initially, the restriction was planned to be done through a common server firewall, but this would bring a lot of inconvenience, for example, a mandatory access restriction for all users. However, at the moment, the restriction applies to a separate account and does not affect the "neighbors", so if the user does not want to (for example, he does not have a fixed IP address), then the restrictions can not be set. Like the password, updating occurs through AJAX without reloading the page.
Another almost unique feature of the service is viewing a list of files and folders directly from the storage control panel. In order not to connect to the server every time through the FTP client, users have the opportunity to view the list of files and folders in the form of a tree directly through the control panel. The principle of operation is as follows: every 15 minutes all servers are polled, and for each account a file and folder map is created, which shows the name, date of change and size. Since different users may have a different number of folders, crawling all the storages and compiling their maps can take several minutes, so the information is updated 4 times per hour. However, for the future there are plans to make a button for a “manual” update.
I will be glad to answer questions and listen to comments. Upon request in PM or at leonid@poiskvps.ru I can provide a test service to check the data transfer rate.
PS I thank the hosting company VDSina.ru for supporting the project.
I think that almost every webmaster or server owner encountered a problem when a backup copy of the data was needed, but for some reason it did not happen. On a common shared hosting, it is accepted that the hosts themselves make backups and in which case they are ready to provide them to users. However, when renting a virtual or dedicated server, the situation is exactly the opposite: the owner needs to independently monitor the safety of data and perform backups on their own. Sometimes hosters offer “boxed” solutions, when together in the server they give the user the opportunity to rent a place to store backups. In my opinion, this is not the best idea, since backups need to be stored at least in another data center, and preferably in several. For example, you can recall the fire in the hosting data center. ua - no one is safe from this set of events, but such a case shows that storing backups in the same data center is not the best idea. Also, in my opinion, storing backups with the host is also not the best idea, even if backups are stored far from the main server. Recently, it is not uncommon for a hoster to simply disappear or seize all equipment.
Backup Lease Service
- Service activation
- password change
- Occupancy Threshold Notifications
- Access restriction by IP addresses
- View a list of files and folders
The backup space rental service has been operating for less than half a year, but there have been significant changes in it recently. Today I want to talk in detail about them and about the service. Initially, the idea was to make a simple and convenient server for storing backup copies, at the same time I wanted the price per gigabyte to be sky-high. For this, a rather well-known data center was chosen - hetzner.de - and until recently it was possible to buy a place only in this data center in Germany at a very “tasty” price: 10 Gb costs 30 rubles, 50 Gb - 100 rubles and more proportionally.
However, a few days ago it became possible to rent storage in addition to Germany on servers in Russia, in St. Petersburg, Selectel DC. This may be relevant for those users who want to make backups at a very good speed. The traffic is not limited, the connection speed is 100 Mbit / s, there are no artificial speed limits. The price is similar to accommodation in Germany. Plus, until the end of September there is a constant 10% discount on any volume.
For the user, the repository is a login-password pair with which you can manipulate files using the FTP protocol. The service does not offer any other ways of exchanging information, since the most common control panels support FTP downloads, and the service is primarily focused on simplicity.
On the technical side, a Debian OS and an ISPmanager control panel are installed on each server. Using the API of this control panel, all actions are performed with the user account: activation, password change, receiving information about the occupied space and receiving a list of files, setting restrictions on IP addresses.
Service activation
The activation of the service can be divided into two logical parts:
1. Generation of a username and password and creation of a user with the necessary restrictions on the server.
A small explanation of why the username and password are generated, instead of the usual suggestion, come up with them yourself.
The result of the php function uniqid () is used as a login; accordingly, at any moment of time this value is unique, it is valid for creating a new user in the system, and it is difficult to pick it up. Also, a construction of the form login.backup.poiskvps.ru is always used as the server address; accordingly, the login must also be a valid subdomain. In order not to “load” the user with unnecessary information, it was decided to use just such an approach, otherwise the user would have to select the correct login for a long time so that it would satisfy all the requirements. All the same, these data are used only during configuration and forget about them.
The password is generated by the service and is not requested from the user due to the fact that the activation of the service may be delayed: that is, the user ordered the service, but paid for it in a day. And in order not to store the password in clear form for subsequent activation of the store, the password is automatically generated immediately after payment and sent to the mail. Naturally, nothing prevents the user from immediately changing the password. Using AJAX on the site makes this pretty easy.
2. Creating the right subdomain to activate the server address of the form login.backup.poiskvps.ru. If you need to move a user’s account from one server to another (for example, to better manage the occupied space), the best way is to give each user a subdomain. If, for some reason, the user account moves to another server, it will be enough to change one entry in the CSN, and the user will not notice anything. The time for updating DNS records has been set quite short, so that by the time you receive an email with a login and password, DNS records already have time to update, and the user successfully logs on to the server.
password change
Immediately after activating the service and receiving the generated password, any paranoid will have a desire to change this password. Everything is pretty thoughtful here: you can change the password directly from the control panel, and changing the password is deprived of annoying functions, for example, you do not need to type the password twice, and there is only one “quality” limit: the password must be 8 characters or more. That is, the user will never see any notifications “There must be a large vowel letter in any even position in the password”.
Well, and as a bonus - the password is changed without reloading the page, so a pretty picture will indicate that data is being exchanged for about a second, and as soon as the response is received from the destination server that the password has been changed, the user will be notified immediately.
Occupancy Threshold Notifications
Most often, backups are set up once and forget about it, until the time comes to use backup. Agree, it will be very sad to know that there are no backups for the last three months, since all available space has been taken up. Of course, it is not in our power to foresee, and even more prevent all cases of “corruption” of backups, but the issue of occupied space can be brought to perfection.
The principle of operation is as follows: the user indicates the percentage of occupied space, after filling in which he needs to send a notification that there is not enough space left.
The service has one of the “background” processes, which once a minute receives information from all servers and updates statistics on free and occupied space. With this function, you can just see how much was occupied before the update and how much was found after, and if the threshold specified by the user is overcome - you need to send a notification to the user.
In theory, this is a good plan: a clear notification without delays, and the user immediately learns that there is not enough free space left. However, realities add a bit of negativity: most often control panels make backups as follows: first they download the “fresh” version of the backup, and only after that they delete the oldest one. Because of this, at the time of uploading to the storage, free space may run out, and the user will receive a notification each time, although after the file has been downloaded, the free space will reappear. To avoid such cases, the following mechanism is used: as soon as the threshold specified by the user is over, a checkbox is put in the database, and after an hour the occupied space is checked again: if there is still not enough space, the user receives a notification,
Access restriction by IP addresses
In order to increase security on the service, it is possible to set a restriction on IP addresses. If the server supports IPv6 protocol, then you can set a restriction on addresses of this type.
Initially, the restriction was planned to be done through a common server firewall, but this would bring a lot of inconvenience, for example, a mandatory access restriction for all users. However, at the moment, the restriction applies to a separate account and does not affect the "neighbors", so if the user does not want to (for example, he does not have a fixed IP address), then the restrictions can not be set. Like the password, updating occurs through AJAX without reloading the page.
View a list of files and folders
Another almost unique feature of the service is viewing a list of files and folders directly from the storage control panel. In order not to connect to the server every time through the FTP client, users have the opportunity to view the list of files and folders in the form of a tree directly through the control panel. The principle of operation is as follows: every 15 minutes all servers are polled, and for each account a file and folder map is created, which shows the name, date of change and size. Since different users may have a different number of folders, crawling all the storages and compiling their maps can take several minutes, so the information is updated 4 times per hour. However, for the future there are plans to make a button for a “manual” update.
I will be glad to answer questions and listen to comments. Upon request in PM or at leonid@poiskvps.ru I can provide a test service to check the data transfer rate.
PS I thank the hosting company VDSina.ru for supporting the project.