How we stopped spending a week issuing a dev-stand
Every developer wants his dev-stand. Every tester wants his test bench. And every specialist in preproduction wants his own booth - in order to finally check everything and rehearse the launch in the prod. When all these Wishlist converge in the processing - one of the largest and most active systems of the bank - the cost of infrastructure makes you scratch your head and look for "options". That we have found, we will tell in this post.
The amount of processing databases we have is about 6 TB. On one copy of the database, the developers interfere with each other, so the actual amount of space occupied by the bases grows quickly and proportionally. Although how fast ... too fast for the escort service and not fast enough for those who need copies of the databases. And that's why.
For testing, it is necessary that the test stand is fully consistent with the current version of production (the same applies to the stand of production). The main backup of the system is copied for a full day, then deployed on the stand. During these lengthy operations, the stands are unavailable, so copying is transferred to weekends and holidays, when no one works with the stands. We get a delay of 1 to 5 days. To preliminary coordinate the copying process itself, it also takes time - after all, we have several test benches, usually from three to six. Add 2-3 days to coordinate stand idle time. Before you get to the approval of the administrator, the application is still in the queue. In sum, we get a very large delay.
What can speed up the process and save space? Database Virtualization. We looked at various options: Oracle SnapClone, NetApp + Oracle Cloud, just snapshots on arrays. Everything requires complex setup. Oracle solutions, moreover, work only with Oracle databases.
Then looked narrowly at Delphix. It is easy to implement it, it supports different databases - Oracle, SQL Server, DB2, Sybase ASE. An interface is provided for all operations. Through it, developers and testers can independently manage their copies - update, save / restore state, stop / start, etc. There is also an API and CLI for integration with CI / CD systems or their processes.
The deployment of Delphix takes not a lot of time - a few hours. The source can be connected much longer, here time is proportional to size. In our case, the source was the selling copy of the database, and its connection took almost a day. Preparing the source and servers for database clones does not require much effort. On the target server, you need to install the appropriate ORACLE_HOME, and also create a special user for the connection. For virtual test copies, we use the same servers that previously had physical copies.
Delphix allows you to create test benches almost in real time, without any coordination, because the stands are completely isolated from each other. Some time is spent only on updating the database to the current state - from 20 minutes to several hours, there is no question about any days.
We try to conduct load testing in conditions as close as possible to the product. If it is sold on physical disks, then the load stand, too. In this case, the presence of the V2P button in Delphix, which allows you to make an "honest" base of the virtual one, helps.
As for saving disk space, the readings of our Delphix dashboard, of course, are coined - reducing the volume 73 times is too fabulous. In our processing, a copy of a sale with daily snapshots and archived transaction logs for 2 weeks (200 GB per day) takes up 4.5 TB of disk space. Another 1.5 TB - nine clones sizes from 50 to 500 GB, each of which also stores the history of daily snepshots. In total it turns out 6 TB.
We add another 15% free space (900 GB) so that Delphix can work normally. Thus, spending only about 7 TB, we can get a test copy with data that is relevant at any point in time during the last two weeks.
Previously, in order to store nine physical copies of the database in 6 TB, we needed 54 TB (or ~ 20 TB, taking into account the compression 2-3 times on the storage system). And, unlike the new system, here the developers could not manage these copies and restore the previous states - when data was destroyed, it was only possible to perezalit from the copy of the sale.
Also Delphix allows you to quickly make for different projects different branches of the same container - and all this in the shortest possible time. Developers are not afraid to damage the data, they can roll back and restore the previous state. This gives a performance boost.
Impressions of Delphix are mostly positive, although not everything is perfect. The biggest problem is the use of disks. Each data block is stored only once for all virtual copies, and until all virtual copies stop using the block, it cannot be deleted. Organizational problems may arise here - not all users are ready to support the short life cycle of their stands. If the test stand lives on an old copy of the sale, then the space occupied increases. We solve this issue extensively, by expanding the disks, which can be done without interrupting the service. We make sure that 15% of free space is always saved. If it is less, the system simply stops performing any operations with virtual copies. Although the bases themselves will remain available.
For systems with intensive disk I / O, network bandwidth is likely to be the limiting factor. Depending on the specific load profile, the system may start working better with virtualization. Depending on the load, the average db file sequential read time is 5-10 ms, which is pretty good even for industrial systems.
To Delphix, “classic” drives are connected in any way that supports ESX, and the vendor has a list of recommendations on how to do this with maximum performance. We use SAN. The system itself presents the disks on which the virtual database files are located, only using the NFS protocol. For this reason, you need to be attentive to the bandwidth of the channel and its workload. In our case, it makes sense to place only data files for Delphix on disk arrays, so that no activity in the bank would affect the I / O speed for virtual databases.
Now we are working on Delphix 5.1.9, but we are looking at version 5.2 - in it the user interface is gone from flash and, according to the vendor, it has become much more convenient. Delphix made an impression on our colleagues, and after processing we are thinking of transferring profile, CRM, Internet bank to this system.
The amount of processing databases we have is about 6 TB. On one copy of the database, the developers interfere with each other, so the actual amount of space occupied by the bases grows quickly and proportionally. Although how fast ... too fast for the escort service and not fast enough for those who need copies of the databases. And that's why.
For testing, it is necessary that the test stand is fully consistent with the current version of production (the same applies to the stand of production). The main backup of the system is copied for a full day, then deployed on the stand. During these lengthy operations, the stands are unavailable, so copying is transferred to weekends and holidays, when no one works with the stands. We get a delay of 1 to 5 days. To preliminary coordinate the copying process itself, it also takes time - after all, we have several test benches, usually from three to six. Add 2-3 days to coordinate stand idle time. Before you get to the approval of the administrator, the application is still in the queue. In sum, we get a very large delay.
What helped Delphix
What can speed up the process and save space? Database Virtualization. We looked at various options: Oracle SnapClone, NetApp + Oracle Cloud, just snapshots on arrays. Everything requires complex setup. Oracle solutions, moreover, work only with Oracle databases.
Then looked narrowly at Delphix. It is easy to implement it, it supports different databases - Oracle, SQL Server, DB2, Sybase ASE. An interface is provided for all operations. Through it, developers and testers can independently manage their copies - update, save / restore state, stop / start, etc. There is also an API and CLI for integration with CI / CD systems or their processes.
The deployment of Delphix takes not a lot of time - a few hours. The source can be connected much longer, here time is proportional to size. In our case, the source was the selling copy of the database, and its connection took almost a day. Preparing the source and servers for database clones does not require much effort. On the target server, you need to install the appropriate ORACLE_HOME, and also create a special user for the connection. For virtual test copies, we use the same servers that previously had physical copies.
Delphix allows you to create test benches almost in real time, without any coordination, because the stands are completely isolated from each other. Some time is spent only on updating the database to the current state - from 20 minutes to several hours, there is no question about any days.
We try to conduct load testing in conditions as close as possible to the product. If it is sold on physical disks, then the load stand, too. In this case, the presence of the V2P button in Delphix, which allows you to make an "honest" base of the virtual one, helps.
As for saving disk space, the readings of our Delphix dashboard, of course, are coined - reducing the volume 73 times is too fabulous. In our processing, a copy of a sale with daily snapshots and archived transaction logs for 2 weeks (200 GB per day) takes up 4.5 TB of disk space. Another 1.5 TB - nine clones sizes from 50 to 500 GB, each of which also stores the history of daily snepshots. In total it turns out 6 TB.
We add another 15% free space (900 GB) so that Delphix can work normally. Thus, spending only about 7 TB, we can get a test copy with data that is relevant at any point in time during the last two weeks.
Previously, in order to store nine physical copies of the database in 6 TB, we needed 54 TB (or ~ 20 TB, taking into account the compression 2-3 times on the storage system). And, unlike the new system, here the developers could not manage these copies and restore the previous states - when data was destroyed, it was only possible to perezalit from the copy of the sale.
Also Delphix allows you to quickly make for different projects different branches of the same container - and all this in the shortest possible time. Developers are not afraid to damage the data, they can roll back and restore the previous state. This gives a performance boost.
But there are nuances ...
Impressions of Delphix are mostly positive, although not everything is perfect. The biggest problem is the use of disks. Each data block is stored only once for all virtual copies, and until all virtual copies stop using the block, it cannot be deleted. Organizational problems may arise here - not all users are ready to support the short life cycle of their stands. If the test stand lives on an old copy of the sale, then the space occupied increases. We solve this issue extensively, by expanding the disks, which can be done without interrupting the service. We make sure that 15% of free space is always saved. If it is less, the system simply stops performing any operations with virtual copies. Although the bases themselves will remain available.
For systems with intensive disk I / O, network bandwidth is likely to be the limiting factor. Depending on the specific load profile, the system may start working better with virtualization. Depending on the load, the average db file sequential read time is 5-10 ms, which is pretty good even for industrial systems.
To Delphix, “classic” drives are connected in any way that supports ESX, and the vendor has a list of recommendations on how to do this with maximum performance. We use SAN. The system itself presents the disks on which the virtual database files are located, only using the NFS protocol. For this reason, you need to be attentive to the bandwidth of the channel and its workload. In our case, it makes sense to place only data files for Delphix on disk arrays, so that no activity in the bank would affect the I / O speed for virtual databases.
Now we are working on Delphix 5.1.9, but we are looking at version 5.2 - in it the user interface is gone from flash and, according to the vendor, it has become much more convenient. Delphix made an impression on our colleagues, and after processing we are thinking of transferring profile, CRM, Internet bank to this system.