Morning admin: add space on dozens of servers for coffee
Every day I have to add space on one, two, three, five, and sometimes - and ten database servers. Why? Because they are characterized by natural growth bases. There are hundreds of servers, they are all virtuals with disks on thin provisioning. If they are given a lot of space in advance, then there will definitely be some kind of “runaway”, such as an upgrade with overflow of tables, which will devour the whole place, and if it does not, it will nestle. As you know, thin provisioning is a one-way path if the place is gobbled up, but you cannot return it back.
As a result, most servers dangle somewhere near the border of 90% of the space used - precisely because an alert is triggered on the border of 90%. As soon as I give a little, just a little space - the server goes to the 80% -85% used area, and in a month another place needs to be added again. And, nevertheless, I will not give much at once - too many precedents with runaways.
I so often did the mechanical work of expanding disk space that I was tired of and I decided to automate it using Jenkins:
I apologize for the somewhat boring and detailed description below, but if it can be useful to someone, then just small details are important - it is on them that you always spend the most time. If the details are not important, immediately scroll to the last picture.
Of course, before adding a place, I look at the growth history of the space used% automatically attached to the alert by my secret monitoring system . Most often, natural growth:
And here it is better to first understand:
Let's go back to Jenkins, which is just an interface to the Powershell script. We have a lot of VMware Vcenters, so a special SQL procedure by server name determines where this server is located and connects to it:
Now we read the disk size, add a delta, and set a new value.
True, I blatantly use the fact that all servers are cast in this way, for example, D: always Hard disk 2. If you have it wrong, you will have to suffer.
Now the disk is expanded from the point of view of VMware, but not from the point of view of guest (Windows). We must use the allocated space. To do this, you must perform inside the guest command DISKPART.
We brazenly stuff into the root of D: the file BAT and IN, and with the help of WMIC force the machine to execute these commands. The file doresizeX.bat (X is the name of the drive) contains only
And doresizeX.in contains:
Now everything is ready, you just have to wait a bit (the command is asynchronous!) And read the result, filtering out the unnecessary:
And waiting for a letter from Jenkins:
As a result, most servers dangle somewhere near the border of 90% of the space used - precisely because an alert is triggered on the border of 90%. As soon as I give a little, just a little space - the server goes to the 80% -85% used area, and in a month another place needs to be added again. And, nevertheless, I will not give much at once - too many precedents with runaways.
I so often did the mechanical work of expanding disk space that I was tired of and I decided to automate it using Jenkins:
I apologize for the somewhat boring and detailed description below, but if it can be useful to someone, then just small details are important - it is on them that you always spend the most time. If the details are not important, immediately scroll to the last picture.
Of course, before adding a place, I look at the growth history of the space used% automatically attached to the alert by my secret monitoring system . Most often, natural growth:
And here it is better to first understand:
Let's go back to Jenkins, which is just an interface to the Powershell script. We have a lot of VMware Vcenters, so a special SQL procedure by server name determines where this server is located and connects to it:
Now we read the disk size, add a delta, and set a new value.
True, I blatantly use the fact that all servers are cast in this way, for example, D: always Hard disk 2. If you have it wrong, you will have to suffer.
Now the disk is expanded from the point of view of VMware, but not from the point of view of guest (Windows). We must use the allocated space. To do this, you must perform inside the guest command DISKPART.
We brazenly stuff into the root of D: the file BAT and IN, and with the help of WMIC force the machine to execute these commands. The file doresizeX.bat (X is the name of the drive) contains only
diskpart <d: \ doresizeX.in> d: \ doresize.out
And doresizeX.in contains:
rescan
select volume X
extend
exit
Now everything is ready, you just have to wait a bit (the command is asynchronous!) And read the result, filtering out the unnecessary:
And waiting for a letter from Jenkins: