Automatic adding of a place on the virtual server
Hello!
In this article we will talk about how we automated the task of expanding disk space on one of our servers. And what is so difficult in such a simple task that you had to automate it - you ask? Nothing if you are not using cascade-merged mount. I feel there are more questions !? Well then let's go under the cat.
First, I’ll tell you why we use cascade-combined mount.
We have one system that needs storage for small files (scans of documents, etc.). The average file size is from 200kb to 1 megabyte, the data is static, does not change. There are billions of files in it and the number is growing every day. Once, when the volume was already over 6TB, we realized that problems would soon begin, one of which was the time of backup and recovery. Then we decided to split the data on the disks, and UnionFS was called to help us with this.
The algorithm defined the following: data is written to disk no more than 2TB, when it ends we add a new disk to the virtual machine, mark it, add it to UnionFS, translate the old one into ReadOnly, remove a copy from it, write to tape, remove from the operational backup.
As you already understood, this algorithm is quite demanding of the attention of the administrator - any awkward movement and storage is not available. Therefore, we decided to eliminate the human factor completely and remembered that we have ZABBIX, which can easily cope with this itself if we add a little magic PowerShell and Bash to the algorithm.
Now how it is made.
In Zabbix, a trigger is configured for free space and a button is made for manual mode:
When a trigger is triggered, a task is formed in the sheduller of the robot server on which all our automation scripts are located:
Powershell.exe "Enable-ScheduledTask \PROD_TASKS\Add_HDD_OS0226”At the appointed time, a script is run on the server that:
Adds the disk of the necessary VM:
(at the same time it selects the most free volume)
$vm = Get-VM -Name $vmName
New-HardDisk -VM $vm -CapacityGB $newHDDCapacity -Datastore $datastoreName –ThinProvisioned
Looking for server access details:
Offtoptop
У нас используется кастомизированное хранилище реквизитов доступа на базе TeamPass, поэтому скрип находит нужный сервер в системе и получает его реквизиты автоматически. Так сделано потому что каждый месяц у нас происходит автоматическая смена всех паролей, но это тема отдельной статьи
#Generate TeamPass API request string
$vmTPReq = "Строка запроса к TeamPass"
#Send request to TeamPass
$vmCreds = Invoke-WebRequest($vmTPReq) -UseBasicParsing | ConvertFrom-Json
#Convert credentials
$credential = New-Object System.Management.Automation.PSCredential($vmCreds.login,(ConvertTo-SecureString $vmCreds.pw -asPlainText -Force))
Comes on SSH:
#Create partition & FS, mount disk to directory, edit fstab...etc.
New-SSHSession -ComputerName $vmCreds.url -Credential $credential -Verbose -AcceptKey:$true
$results = Invoke-SSHCommand -SessionId 0 -Command "/mnt/autodoit.sh"
Remove-SSHSession -Index 0 -Verbose
Lays it down and adds to UnionFS mount:
(autodoit.sh)
#!/bin/bash
fstab="/etc/fstab"
newdisk=$((
(
parted -lm >&1
) 1>/tmp/gethddlog
) 2>&1)
newdisk=$(echo$newdisk | cut -d ':' -f 2)
if [[ $newdisk == "" ]] ;
thenprintf"New disk not found! Exit\n".
exitfiprintf"New disk found: $newdisk\n"echo#Create new partitionecho Create new partition ...
parted $newdisk mklabel gpt unit TB mkpart primary 0.00TB 2.00TB print
sleep 10
#Create filesystemecho Create filesystem xfs ...
newdisk="$newdisk$((1))"
mkfs.xfs $newdisk#Create new DATA directory
lastdata=$(ls /mnt/ | grep 'data[0-9]\+$' | cut -c5- | sort -n | tail -n1)
lastdatamount="/mnt/data$((lastdata))"
newdata="/mnt/data$((lastdata+1))"printf"Create new direcory: $newdata\n"
mkdir $newdata
chown -R nobody:nobody $newdata
chmod -R 755 $newdata#Mount new partition to new directoryprintf"Mount new partition to $newdata...\n"
mount -t xfs ${newdisk}${newdata}#Get UUID of new partition
uuid=$(blkid $newdisk -o value -s UUID)
printf"New disk UUID: $uuid\n"#Add mountpoint for new partitionprintf"Add mountpoint for new disk to fstab...\n"
lastdatamount=$(cat $fstab | grep "$lastdatamount\s")
newdatamount="UUID=$uuid$newdata xfs defaults,nofail 0 0"
ldm=$(echo$lastdatamount | sed -r 's/[\/]/\\\//g')
ndm=$(echo$newdatamount | sed -r 's/[\/]/\\\//g')
sed -i "/$ldm/a $ndm"$fstab#Change UnionFS mountpoint stringprintf"Modify mountpoint for UnionFS in fstab...\n"
prevunion=$(cat $fstab | grep fuse.unionfs)
newunion=$(echo$prevunion | sed -e "s/=rw/=ro/")
newunion=$(echo$newdata=rw:$newunion)
sed -i "s|$prevunion|$newunion|"$fstab#Remount UnionFSprintf"Remount UnionFS...\n"
service smb stop
sleep 0.6
umount /mnt/unionfs
mount /mnt/unionfs
service smb start
printf"Done!\n\n"
rm /tmp/gethddlog
Unfortunately, at the time of writing this article, we have not solved several issues related to the automatic creation of tasks in VEEAM for archiving an old disk and recording it on tape, so for now this is done manually. But we will update the script as soon as we solve a couple of problems.
The author is Vitaly Rozman ( PBCVIT ).
A piece of code for gluing arrays was honestly borrowed, the references in the code to the author are preserved.
Complete script
#Set VM name$vmName = "OS0226"#Set TeamPass ID of linux server$vmTPId = "1161"#Set capacity of new HDD in GB$newHDDCapacity = 2048
#Set Log file$logFile = "C:\SCRIPTS\Log\NewHardDisk-OS0226.log"#Import module for SSH connections
Import-Module Posh-SSH
#Add VEEAM Snap-In
Add-PSSnapin VeeamPSSnapin
#Initialize VMWare PowerCLI
& 'C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1'#Add function for array join
Function Join-Object { # https://powersnippets.com/join-object/
[CmdletBinding()]Param ( # Version 02.02.00, by iRon
[Object[]]$RightTable, [Alias("Using")]$On, $Merge = @{}, [Parameter(ValueFromPipeLine = $True)][Object[]]$LeftTable, [String]$Equals
)
$Type = ($MyInvocation.InvocationName -Split "-")[0]
$PipeLine = $Input | ForEach {$_}; If ($PipeLine) {$LeftTable = $PipeLine}
If ($LeftTable -eq $Null) {If ($RightTable[0] -is [Array]) {$LeftTable = $RightTable[0]; $RightTable = $RightTable[-1]} Else {$LeftTable = $RightTable}}
$DefaultMerge = If ($Merge -is [ScriptBlock]) {$Merge; $Merge = @{}} ElseIf ($Merge."") {$Merge.""} Else {{$Left.$_, $Right.$_}}
If ($Equals) {$Merge.$Equals = {If ($Left.$Equals -ne $Null) {$Left.$Equals} Else {$Right.$Equals}}}
ElseIf ($On -is [String] -or $On -is [Array]) {@($On) | ForEach {If (!$Merge.$_) {$Merge.$_ = {$Left.$_}}}}
$LeftKeys = @($LeftTable[0].PSObject.Properties | ForEach {$_.Name})
$RightKeys = @($RightTable[0].PSObject.Properties | ForEach {$_.Name})
$Keys = $LeftKeys + $RightKeys | Select -Unique
$Keys | Where {!$Merge.$_} | ForEach {$Merge.$_ = $DefaultMerge}
$Properties = @{}; $LeftOut = @($True) * @($LeftTable).Length; $RightOut = @($True) * @($RightTable).Length
For ($LeftIndex = 0; $LeftIndex -lt $LeftOut.Length; $LeftIndex++) {$Left = $LeftTable[$LeftIndex]
For ($RightIndex = 0; $RightIndex -lt $RightOut.Length; $RightIndex++) {$Right = $RightTable[$RightIndex]
$Select = If ($On -is [String]) {If ($Equals) {$Left.$On -eq $Right.$Equals} Else {$Left.$On -eq $Right.$On}}
ElseIf ($On -is [Array]) {($On | Where {!($Left.$_ -eq $Right.$_)}) -eq $Null} ElseIf ($On -is [ScriptBlock]) {&$On} Else {$True}
If ($Select) {$Keys | ForEach {$Properties.$_ =
If ($LeftKeys -NotContains $_) {$Right.$_} ElseIf ($RightKeys -NotContains $_) {$Left.$_} Else {&$Merge.$_}
}; New-Object PSObject -Property $Properties; $LeftOut[$LeftIndex], $RightOut[$RightIndex] = $Null
} } }
If ("LeftJoin", "FullJoin" -Contains $Type) {
For ($LeftIndex = 0; $LeftIndex -lt $LeftOut.Length; $LeftIndex++) {
If ($LeftOut[$LeftIndex]) {$Keys | ForEach {$Properties.$_ = $LeftTable[$LeftIndex].$_}; New-Object PSObject -Property $Properties}
} }
If ("RightJoin", "FullJoin" -Contains $Type) {
For ($RightIndex = 0; $RightIndex -lt $RightOut.Length; $RightIndex++) {
If ($RightOut[$RightIndex]) {$Keys | ForEach {$Properties.$_ = $RightTable[$RightIndex].$_}; New-Object PSObject -Property $Properties}
} }
};
Set-Alias Join Join-Object
Set-Alias InnerJoin Join-Object; Set-Alias InnerJoin-Object Join-Object -Description "Returns records that have matching values in both tables"
Set-Alias LeftJoin Join-Object; Set-Alias LeftJoin-Object Join-Object -Description "Returns all records from the left table and the matched records from the right table"
Set-Alias RightJoin Join-Object; Set-Alias RightJoin-Object Join-Object -Description "Returns all records from the right table and the matched records from the left table"
Set-Alias FullJoin Join-Object; Set-Alias FullJoin-Object Join-Object -Description "Returns all records when there is a match in either left or right table"#Connect to vCenter
Connect-VIServer vcenter.mmc.local
#Get datastore$datastores = get-datastore | where-object Name -like "*TIERED_VM_PROD*"if ($datastores.Count -gt 0) {
if (($datastores | Sort -Descending {$_.FreeSpaceGB})[0].FreeSpaceGB -gt 2048) {
$datastoreName = ($datastores | Sort -Descending {$_.FreeSpaceGB})[0].Name
} else {
Write-Host("ERROR: No enought space on datastore for new HDD!")
break
}
} else {
Write-Host("ERROR: No Datastores found!")
break
}
#Generate TeamPass API request string$vmTPReq = "строка запроса к TeamPass"#Send request to TeamPass$vmCreds = Invoke-WebRequest($vmTPReq) -UseBasicParsing | ConvertFrom-Json
#Convert credentials$credential = New-Object System.Management.Automation.PSCredential($vmCreds.login,(ConvertTo-SecureString $vmCreds.pw -asPlainText -Force))
if ((Test-Connection $vmCreds.url -Count 1 -Quiet) -eq $false) { $err = $error[0].FullyQualifiedErrorId }
try
{
# Get disks information from Linux
New-SSHSession -ComputerName $vmCreds.url -Credential $credential -Verbose -AcceptKey:$true$linuxCommand1 = 'ls -dl /sys/block/sd*/device/scsi_device/*'$linuxDisksData1 = Invoke-SSHCommand -SessionId 0 -Command $linuxCommand1$linuxCommand2 = 'lsblk -l | grep /mnt'$linuxDisksData2 = Invoke-SSHCommand -SessionId 0 -Command $linuxCommand2
Remove-SSHSession -Index 0 -Verbose
$linuxMounts = $linuxDisksData2.Output -replace '\s+', ' ' |
Select @{N='Partition';E={($_.split(" ")[0])}},
@{N='linuxMount';E={($_.split(" ")[6])}}
$linuxDisks = $linuxDisksData1.Output -replace '\s+', ' ' |
Select @{N='Partition';E={($_.split(" ")[8]).split('/')[3]+'1'}},
@{N='SCSIAddr';E={(($_.split(" ")[8]).split('/')[6]).split(':')[1]+':'+(($_.split(" ")[8]).split('/')[6]).split(':')[2]}}
$linuxDisks = $linuxDisks | sort SCSIAddr
}
catch
{
$err = $error[0].FullyQualifiedErrorId
}
#Get disks information from vmware$vmDisks = Get-VM -Name $vmName | Get-HardDisk |
Select @{N='vmwareHardDisk';E={$_.Name}},
@{N='vSCSI';E={$_.uid.split("/")[3].split("=")[1]}},
@{N='SCSIAddr';E={[string]([math]::truncate((($_.uid.split("/")[3].split("=")[1])-2000)/16))+':'+[string]((($_.uid.split("/")[3].split("=")[1])-2000)%16)}}
$vmDisks = $vmDisks | sort SCSIAddr
#Get total information about VM Disks$OLAYtotalEffects = $vmDisks | InnerJoin $linuxDisks SCSIAddr -eq SCSIAddr | InnerJoin $linuxMounts Partition -eq Partition| sort vmwareHardDisk
#Display total information about VM Disks$OLAYtotalEffects | ft
$OLAYtotalEffects | ft 2>$logFile#Get latest mount$linuxLatestDiskMount = [string](($OLAYtotalEffects | select linuxMount | where linuxMount -like "/mnt/data*" | % {[int](($_.linuxMount.Split("/")[2]).Replace("data",""))} | Measure -Maximum).Maximum)
#Get latest vSCSI number$latestDiskvSCSI = ($OLAYtotalEffects | where {$_.linuxMount -eq "/mnt/data$linuxLatestDiskMount"}).vSCSI
#Add HDD to VM$vm = Get-VM -Name $vmName
New-HardDisk -VM $vm -CapacityGB $newHDDCapacity -Datastore $datastoreName -ThinProvisioned
#Let the disk takes root
Write-Host("Let the disk takes root...")
sleep 10
if ((Test-Connection $vmCreds.url -Count 1 -Quiet) -eq $false) { $err = $error[0].FullyQualifiedErrorId }
try
{
#Create partition & FS, mount disk to directory, edit fstab...etc.
New-SSHSession -ComputerName $vmCreds.url -Credential $credential -Verbose -AcceptKey:$true$results = Invoke-SSHCommand -SessionId 0 -Command "/mnt/autodoit.sh"
Remove-SSHSession -Index 0 -Verbose
$results.Output
}
catch
{
$err = $error[0].FullyQualifiedErrorId
}
#After adding a new disk, some checks are just performedif ((Test-Connection $vmCreds.url -Count 1 -Quiet) -eq $false) { $err = $error[0].FullyQualifiedErrorId }
try
{
# Get disks information from Linux
New-SSHSession -ComputerName $vmCreds.url -Credential $credential -Verbose -AcceptKey:$true$linuxCommand1 = 'ls -dl /sys/block/sd*/device/scsi_device/*'$linuxDisksData1 = Invoke-SSHCommand -SessionId 0 -Command $linuxCommand1$linuxCommand2 = 'lsblk -l | grep /mnt'$linuxDisksData2 = Invoke-SSHCommand -SessionId 0 -Command $linuxCommand2
Remove-SSHSession -Index 0 -Verbose
$linuxMounts = $linuxDisksData2.Output -replace '\s+', ' ' |
Select @{N='Partition';E={($_.split(" ")[0])}},
@{N='linuxMount';E={($_.split(" ")[6])}}
$linuxDisks = $linuxDisksData1.Output -replace '\s+', ' ' |
Select @{N='Partition';E={($_.split(" ")[8]).split('/')[3]+'1'}},
@{N='SCSIAddr';E={(($_.split(" ")[8]).split('/')[6]).split(':')[1]+':'+(($_.split(" ")[8]).split('/')[6]).split(':')[2]}}
$linuxDisks = $linuxDisks | sort SCSIAddr
}
catch
{
$err = $error[0].FullyQualifiedErrorId
}
#Get disks information from vmware$vmDisks = Get-VM -Name $vmName | Get-HardDisk |
Select @{N='vmwareHardDisk';E={$_.Name}},
@{N='vSCSI';E={$_.uid.split("/")[3].split("=")[1]}},
@{N='SCSIAddr';E={[string]([math]::truncate((($_.uid.split("/")[3].split("=")[1])-2000)/16))+':'+[string]((($_.uid.split("/")[3].split("=")[1])-2000)%16)}}
$vmDisks = $vmDisks | sort SCSIAddr
#Get total information about VM Disks$OLAYtotalEffects = $vmDisks | InnerJoin $linuxDisks SCSIAddr -eq SCSIAddr | InnerJoin $linuxMounts Partition -eq Partition| sort vmwareHardDisk
#Display total information about VM Disks$OLAYtotalEffects | ft
$OLAYtotalEffects | ft 2>$logFile
Disconnect-VIServer -Confirm:$false
Disable-ScheduledTask \PROD_TASKS\Add_HDD_OS0226There are no claims to UnionFS, it has been working steadily for more than two years.
The question of why storage is so organized in general, let’s leave rhetorical, just accept it as it is.
I draw your attention to the fact that different types of disk mapping should be used for different systems. so be careful and power will come with you.