September 30, 2017 at 23:04

Is DevOps coming to our home? Home Minecraft server in Azure using modern DevOps practices

  • Tutorial
This article is NOT about minecraft. This article is about Azure and about modern approaches to software distribution.

If you just want to deploy your minecraft server - ask google "minecraft server hosting" - it will be both easier and cheaper.

But if you want to look at the real case of using the approaches of Infrastructure as Code , Configuration as Code as applied to non-adapted to Azure software using the example of minecraft, then welcome

Roadmap


Azure Stack is huge and I want to poke in different parts of it, so this article will be not one at all.

The outline of the articles so far seems like this:

  • Automated deployment and server configuration using Azure CLI 2 and Powershell DSC
  • Maintain a consistent server configuration using Azure Automation
  • Automated server upgrade to the new version
  • Obtaining minecraft logs, analysis and display using Azure Stream Analytics and Power BI
  • Organizing an active / passive cluster with management through Azure Automation
  • ...

But of course, the process of the appearance of new articles can be stopped by the lack of your interest)

Disclaimer


  • I'm not an it engineer, not a devops engineer, not a minecraft player. Therefore, if you rummage and it seems to you that I am not doing everything right - then most likely the way it is - write in comments as it should.
  • All examples to the article are 100% working, but delivered “as is”. If something doesn’t work for you, fix it yourself or hammer it.

So.

Automated deployment and server configuration using Azure CLI 2 and Powershell DSC


About the task


The task is not synthetic. The server was asked to deploy the children and it will actually be used. The last thing I want is to constantly repair and lift it. Even less I want to lose data during updates. Therefore, the requirements for it are quite prod-like)

General approach


  • 100% automation.
  • We treat scripts as code
  • We comply with the basic requirements of configuration management for the process distribution to prod environments

I will explain all the points along the way.

About software


As the server, I took the official server minecraft.net/en-us/download/server . A cursory study revealed that:

  • The server is a jar file and requires java to run
  • To start, he needs an eula.txt file with certain contents next to him
  • The server is configured using the server.properties file, which must also lie nearby
  • Server listens on port 25565 over TCP
  • The server stores the data (map) in the World folder next to it (and this will be a bit of a problem)
  • The server can upgrade data if it was created by a previous version (and this will greatly simplify life when upgrading)
  • Logs are stored in the Logs folder next to

Infrastructure


We apply the most basic scheme:

one Resource Group c:

  • Network Security Group (ports 25565 (minecraft) and 3389 (rdp) will be open
  • VNET c 1 subnet
  • NIC and static public IP with DNS name
  • 1 VM under Windows Server 2016 core + 2 Managed Disk (OS + Data)
  • Storage account and 1 container for Blob storage

image

We will store the data separately from the virtual machine on a separate managed disk and link it to the World folder next to the jar using Symlink (I wrote above that the location of the world folder is not configurable and should always be next to the jar).

This will make it possible to start the server with or without a linked disk (for tests) - in this case, the world folder will be created with an empty map.

We will configure the server based on:

  • Distribution (jar with server + jre + initial data), which will be preloaded into Blob storage
  • Configurations (Powershell DSC), which will also be preloaded into Blob storage

I promised above to follow some requirements of configuration management - here is the first:

All artifacts that are required to deploy the application must be published in a controlled availability with high availability.

That is why we will pre-assemble the distribution and configuration from artifacts on the Internet and republish it to Blob Storage.

Thus, the deployment procedure algorithm is as follows:

  • Create a new Resource Group, Storage Account and Storage Container
  • Form a distribution kit (jar with server + jre + initial data) and upload it to Blob Storage
  • Create the rest of the infrastructure in the Resource Group
  • Create a DSC configuration and upload it to Blob Storage
  • Configure the server based on configuration and distribution

Source code


  • Store on github
  • We adhere to some intelligible strategy of brunching and release cycle (well, for example, gitflow and release each article :))
  • We use azure cli 2 and bash wherever possible (by the way, in this article we managed to completely avoid the use of powershell api. But in the next it will fail)

Implementation of the steps of the deployment procedure


1. Create a new Resource Group, Storage Account and Storage Container


the implementation is trivial, just cli calls

initial_preparation.sh
az configure --defaults location=$LOCATION group=$GROUP
echo "create new group"
az group create -n $GROUP
echo "create storage account"
az storage account create -n $STORAGE_ACCOUNT --sku Standard_LRS
STORAGE_CS=$(az storage account show-connection-string -n $STORAGE_ACCOUNT)
export AZURE_STORAGE_CONNECTION_STRING="$STORAGE_CS"
echo "create storage container"
az storage container create -n $STORAGE_CONTAINER --public-access blob


2. Form a distribution kit (jar with server + jre + initial data) and upload it to Blob Storage


Download the server and map, take jre from the current computer and pack everything into
the distribution structure zip :

  • .jar
  • jre
  • initial_world - folder with the initial map

prepare_distr.sh
mkdir $DISTR_DIR
cd $DISTR_DIR
echo "download minecraft server from official site"
curl -Os https://s3.amazonaws.com/Minecraft.Download/versions/1.12.2/minecraft_server.$MVERSION.jar 
echo "copy jre from this machine"
cp -r "$JRE" ./jre
echo "create ititial world folder"
mkdir initial_world
cd initial_world
echo "download initial map"
curl -Os https://dl01.mcworldmap.com/user/1821/world2.zip 
unzip -q world2.zip
cp -r StarWars/* .
rm -r -f StarWars
rm world2.zip
cd ../
echo "create archive (zip utility -> https://ranxing.wordpress.com/2016/12/13/add-zip-into-git-bash-on-windows/)"
cd ../
zip -r -q $DISTR_ZIP $DISTR_DIR 
rm -r -f $DISTR_DIR


3. Create a DSC configuration and upload it to Blob Storage


DSC configuration is one ps1 file (about the structure of it - later), which will run on the server and configure it.

But the scripts in the ps1 file have depends on external modules that will not be installed on the server automatically.

Therefore, the configuration must be transferred to the server not just as a ps1 file, but as an archive with:

  • ps1 file
  • All dependent modules, folded in folders of the same name next to the ps1 file

Dependent modules can simply be downloaded by curl as a nuget package from the powershell gallery repository and unzipped to the desired location.

You can also use the powershell Save-Module command - which includes these two steps.
Please note that the versions of the dependent modules in the ps1 file and in the scripts for creating the archive with the configuration are indicated and match.

And here we again follow the requirements of the next configuration management rule:

All dependencies must be precisely defined and always uniquely resolved

prepare_config.sh
echo "prepare server configuration"
curl -s -L -o configuration/xPSDesiredStateConfiguration.zip "https://www.powershellgallery.com/api/v2/package/xPSDesiredStateConfiguration/7.0.0.0"
curl -s -L -o configuration/xNetworking.zip "https://www.powershellgallery.com/api/v2/package/xNetworking/5.1.0.0"
curl -s -L -o configuration/xStorage.zip "https://www.powershellgallery.com/api/v2/package/xStorage/3.2.0.0"
cd configuration
unzip -q xPSDesiredStateConfiguration.zip -d xPSDesiredStateConfiguration
rm -r xPSDesiredStateConfiguration.zip
unzip -q xNetworking.zip -d xNetworking
rm -r xNetworking.zip
unzip -q xStorage.zip -d xStorage
rm -r xStorage.zip
zip -r -q ../$CONFIG_ZIP . *
rm -r -f xPSDesiredStateConfiguration
rm -r -f xNetworking
rm -r -f xStorage
cd ../


4. Create the rest of the infrastructure in the Resource Group


the implementation is trivial, just calls cli:

iaas_preparation.sh

echo "create network security group and rules"
az network nsg create -n $NSG
az network nsg rule create --nsg-name $NSG -n AllowMinecraft --destination-port-ranges 25556 --protocol Tcp --priority 100
az network nsg rule create --nsg-name $NSG -n AllowRDP --destination-port-ranges 3389 --protocol Tcp --priority 110
echo "create vnet, nic and pip"
NIC_NAME=minesrvnic
PIP_NAME=minepip
SUBNET_NAME=servers
az network vnet create -n $VNET --subnet-name $SUBNET_NAME
az network public-ip create -n $PIP_NAME --dns-name $DNS --allocation-method Static
az network nic create --vnet-name $VNET --subnet $SUBNET_NAME --public-ip-address $PIP_NAME -n $NIC_NAME 
echo "create data disk"
DISK_NAME=minedata
az disk create -n $DISK_NAME --size-gb 10 --sku Standard_LRS
echo "create server vm"
az vm create -n $VM_NAME --size $VM_SIZE --image $VM_IMAGE \
                --nics $NIC_NAME \
                --admin-username $VM_ADMIN_LOGIN --admin-password $VM_ADMIN_PASSWORD \
                --os-disk-name ${VM_NAME}disk --attach-data-disk $DISK_NAME


5. Configure the server based on configuration and distribution


Here, probably the most interesting place. Server configuration is based on the DSC configuration previously downloaded using the DSC extension for vm.

DSC extension is fed with a config that shows the URL to the archive with the configuration and parameter values

server_configuration.sh
echo "prepare dsc extension settings"
cat MinecraftServerDSCSettings.json | envsubst > ThisMinecraftServerDSCSettings.json
echo "configure vm"
az vm extension set \
   --name DSC \
   --publisher Microsoft.Powershell \
   --version 2.7 \
   --vm-name $VM_NAME \
   --resource-group $GROUP \
   --settings ThisMinecraftServerDSCSettings.json
rm -f ThisMinecraftServerDSCSettings.json


About Powershell DSC


Well, this is some kind of add-on over powershell, which allows you to declaratively describe the required state of the computer.

The DSC configuration is analyzed on the target machine, a differential is constructed relative to the current configuration of this machine, and then the configuration is brought to the desired one. The closest equivalent to powershell DSC in the "opensource" world is Ansible .

MinecraftServer DSC Implementation


Structurally, the configuration consists of a set of steps that are performed in a specific order (taking into account the dependencies). Let's go over all:

xRemoteFile DistrCopy
{
   Uri = "https://$accountName.blob.core.windows.net/$containerName/mineserver.$minecraftVersion.zip"
   DestinationPath = "$mineHome.zip"
   MatchSource = $true
}

He declares that the zip with the server distribution should be on the computer along the DestinationPath path, if it does not lie there, it will be downloaded at the Uri address:

Archive UnzipServer 
{
   Ensure = "Present"
   Path = "$mineHome.zip"
   Destination = $mineHomeRoot
   DependsOn = "[xRemoteFile]DistrCopy"
   Validate = $true
   Force = $true
}

He declares that on the computer in the Destination folder there must be an unzipped archive with the server distribution if it does not lie there (or if the composition of the files differs from that in the archive) - they will be added:

File CheckProperties
{
   DestinationPath = "$mineHome\server.properties"
   Type = "File"
   Ensure = "Present"
   Force = $true
   Contents = "....."
}
File CheckEULA
{
   DestinationPath = "$mineHome\eula.txt"
   Type = "File"
   Ensure = "Present"
   Force = $true
   Contents = "..."				
   DependsOn = "[File]CheckProperties"
}

Declares that the DestinationPath path must contain files with a server config and with a license. If they are not, they are created with the Contents content:

xWaitForDisk WaitWorldDisk
{
   DiskIdType = "Number"
   DiskId = "2"   
   RetryIntervalSec = 60	
   RetryCount = 5
   DependsOn = "[File]CheckEULA"
}
xDisk PrepareWorldDisk
{
   DependsOn = "[xWaitForDisk]WaitWorldDisk"
   DiskIdType = "Number"
   DiskId = "2"
   DriveLetter = "F"
   AllowDestructive = $false
}
xWaitForVolume WaitForF
{
   DriveLetter      = 'F'
   RetryIntervalSec = 5
   RetryCount       = 10
   DependsOn = "[xDisk]PrepareWorldDisk"
}

Here we declare that the managed disk with the data must be initialized in the OS and the letter F must be assigned to it:

File WorldDirectoryExists
{
   Ensure = "Present" 
   Type = "Directory"
   Recurse = $true
   DestinationPath = "F:\world"
   SourcePath = "$mineHome\initial_world"
   MatchSource = $false
   DependsOn = "[xWaitForVolume]WaitForF"    
}

We declare that the world directory (with the world) must be present on the F drive. If it is not there, we believe that the deployment is occurring for the first time and we need to create this folder using the map from the initial_world distribution as a basis:

Script LinkWorldDirectory
{
   DependsOn="[File]WorldDirectoryExists"
   GetScript=
   {
      @{ Result =  (Test-Path "$using:mineHome\World") }
   }
   SetScript=
   {
      New-Item -ItemType SymbolicLink -Path "$using:mineHome\World" -Confirm -Force -Value "F:\world"                
   }
   TestScript=
  {
      return (Test-Path "$using:mineHome\World")
   }
}

This is a custom step - we check that we have the World folder in the server folder. If it is not there - link it from the F. drive. Script works as follows - if TestScript returns false - SetScript is called. Otherwise, it is not called:

Script EnsureServerStart
{
   DependsOn="[Script]LinkWorldDirectory"
   GetScript=
   {
      @{ Result = (Get-Process -Name java -ErrorAction SilentlyContinue) } 
   }
   SetScript=
   {
      Start-Process -FilePath "$using:mineHome\jre\bin\java" -WorkingDirectory "$using:mineHome" -ArgumentList "-Xms512M -Xmx512M  -jar `"$using:mineHome\minecraft_server.$using:minecraftVersion.jar`" nogui"                
   }
   TestScript=
   {
      return (Get-Process -Name java -ErrorAction SilentlyContinue) -ne $null
   }
}

Another custom step. Check that the java process with the server is running). If it is not running, run:

xFirewall FirewallIn 
{ 
   Name = "Minecraft-in"             
   Action = "Allow" 
   LocalPort = ('25565')
   Protocol = 'TCP'
   Direction = 'Inbound'
}
xFirewall FirewallOut 
{ 
   Name = "Minecraft-out"             
   Action = "Allow" 
   LocalPort = ('25565')
   Protocol = 'TCP'
   Direction = 'Outbound'
}

And the last steps - open port 25565.

Epilogue


The article turned out and so long, it is necessary to round off. Remained overboard debug issues of this DSC (very interesting in themselves) - I will cover them in the next.

Thank you all for your attention.

How to start it all from Windows
We will need


Launch Rolout Procedure

git clone https://github.com/AndreyPoturaev/minecraft-in-azure
cd minecraft-in-azure
git checkout v1.0.0
export MINESERVER_PASSWORD=
export MINESERVER_DNS=
export MINESERVER_STORAGE_ACCOUNT=
az login
. rollout.sh

Result:

image
Tags:

Also popular now: