FirstDEDIC Server Diagnostics / FirstVDS / FirstDEDIC Blog

Automatic diagnostics is the first thing we do before selling a Dedic .

If this is a new server, we check the correct operation and enter the configuration information into the database.

If the server was already in operation, we check the health of the components and update the data in the database. The information on the site should correspond to what we sell. It happens that disks were replaced to a previous client, and nowhere is this indicated, the tariff has not been changed. Then the next client risks getting 240 GB SSD instead of the claimed 4000 GB HDD.

We take these risks into account. If we do not update the information manually, the system does this automatically for each new or vacant server. Boots over the network with the Linux kernel and launches a diagnostic program that:

• collects data on new disks and puts it in the database, from there they are uploaded to the site;
• identifies server failures.

What are we checking

CPU

• CPU temperature
• correct processor operation.

For a CPU stress test, mprime-bin starts for 30 minutes.

/usr/bin/timeout 30m /opt/mprime -t
/bin/grep -i error /root/result.txt

Every minute ipmi sensors check the temperature of the processor, the permissible temperature is less than 60C. The program detects CPU architecture errors in the / proc / kmsg and mprime results.txt files.

RAM

Some memory cells may be damaged - you need to check each. Classic Memtest + will not work. In the free version, it does not save the results, only displays on the screen. Therefore we use memtester. We start it from under the OS, while cells that are not occupied by the OS are checked.

memtester `cat /proc/meminfo |grep MemFree | awk '{print $2-1024}'`k 5

We look at the result of the query: if the memory is working properly, the program returns 0.

Repository Does the

program find all the devices in / dev / sd? and / dev / cciss / c0d? and checks each element - whether it is a disk or not.

hdlist() {
 HDLIST=$(ls /dev/sd?)
 HDLIST="${HDLIST} $(ls /dev/cciss/c0d? 2>/dev/null)"
 REAL_HDLIST=""
 for disk in ${HDLIST}; do
        	if head -c0 ${disk} 2>/dev/null; then
     REAL_HDLIST="${REAL_HDLIST} ${disk}"
        	fi
 done
 echo "${REAL_HDLIST}"
}

Now you need to check all the drives.

HDD

- completely clear the hard drive from the data of the previous user:

for DISK in $(hdlist)
  do
    echo "Clearing ${DISK}"
    parted -s ${DISK} mklabel gpt
    dd if=/dev/zero of=${DISK} bs=512 count=1
  done
  if [ "($FULL_HDD_CLEAR)" = "YES" ]; then
  echo "Clearing disks full (very slow)"
  wget -O /dev/null -q --no-check-certificate "${STATEURL}&info=slowhddclear"
  for DISK in $(hdlist)
  do
    echo "Clearing ${DISK}"
    dd if=/dev/zero of=${DISK} bs=1M
  done
  fi

• check the value of the smart attribute Reallocated Sectors Count - must be at least 100,
• check the speed of the disk.

The program estimates the speed at three disk offsets: at the beginning, middle and end - each offset is 4 GB. This is enough to make a general conclusion. For each offset, we use this function:

sysctl -w vm.drop_caches=3 > /dev/null
        	zcav -c 1 -s ${SKIP_COUNT} -r ${OFFSET} -l /tmp/zcav1.log -f ${DISK}
        	if [ $? -ne 0 ]; then
                    	echo err
                    	exit
        	fi
        	SPEED=$(cat /tmp/zcav1.log | awk '! /^#/ {speed+=$2; count+=1}END{print int(speed/count)}')

SSD-drive

Check the value of smart attributes:

Media_Wearout_Indicator - this is the lifetime or wear of the disk: for the new - 100, the minimum allowable - 10.

Reallocated_Sector_Count - the number of reassigned sectors - must be less than 100.

RAID Status We

identify the drive using the RAID model and check the array status. If it is in working mode, it will be “optimal”.

detect_raid_type() {
  RAIDSTR=$(lspci | grep -i raid)
  if echo ${RAIDSTR} | grep -iq adaptec; then
    # THis is adaptec
    echo "adaptec"
  elif echo ${RAIDSTR} | grep -iqE 'lsi|megaraid'; then
    # THis is LSI
    echo "lsi"
  elif echo ${RAIDSTR} | grep -iq '3ware'; then
    # THis is 3ware
    echo "3ware"
  elif echo ${RAIDSTR} | grep -iqE 'Hewlett-Packard.*Smart'; then
    # THis is HP Smart Array
    echo "HP-SmartArray"
  elif dmesg | grep -q cciss/ ; then
    echo cciss
  else
    echo "unknown"
  fi
}
raid_status_adaptec() {
  RSTATUS=$(arcconf getconfig 1 ld | awk -F: '/Status of logical device/ {print $2}')
  if ! echo "${RSTATUS}" | grep -q 'Optimal' ;then
    echo "${RSTATUS}"
    return 1
  fi
}
raid_status_3ware() {
  echo "We have not support 3ware yet"
  return 0
}
raid_status_lsi() {
  RSTATUS=$(megacli -LDInfo -Lall -aALL |awk -F: '$1 ~ /State/ {print $2}')
  if ! echo "${RSTATUS}" | grep -q 'Optimal' ;then
    echo "${RSTATUS}"
    return 1
  fi
}
raid_status_unknown() {
  echo "Unknown RAID"
  return 0
}
raid_status_cciss() {
  RSTATUS=$(cciss_vol_status /dev/cciss/c*d0)
  if ! echo ${RSTATUS} | grep -q "OK" ; then
    echo "${RSTATUS}"
    return 1
  fi
}

Network We

check the download speed over the network - should be more than 300 Mbit.

curl -k --progress-bar -w "%{speed_download}" -o /dev/null "($CGI_MGR_URLv4)/speedtest_cgi?id=($AUTH_ID)&func=server.speedtest"

Statistics

The diagnostic program checks an average of 323 servers per month, 124 of them fail the test - we do not sell these servers. First, data center engineers change drives, repair coolers. CPU and RAM we usually change under warranty.

Let's see the statistics on working HDDs. For analysis, we took 1800 reports for different disks - a total of 103 models.

Attribute name	min	Expected Value	max	Standard deviation	Description
Temperature_Celsius	14	25.81	40	4.09	25C - excellent temperature for the disc
Power_on_hours	407	24033	59363	12910	It's funny Some discs worked for 6 years
Reallocated_Sector_Ct	0	92.3496	10728	496	100 is a good threshold
Raw_Read_Error_Rate	0	32416965	4294967295	126899820.1	All values ​​are great. With the smallest problem, a lot of errors on the trigger
SSD Power_On_Hours	10	23159	918502	134915	More than two years - not bad

Great numbers, now let's check how much the average HDD works. To do this, we compiled statistics on broken disks, focused on the Raw Read Error Rate.

Attribute name	min	Expected Value	max	Standard deviation	Description
Power_on_hours	0	25040	57178	12030	HDD works 33 ± 16 months. Large scatter - hard to draw conclusions

Statistics is an interesting thing, but not the main one. We do not do diagnostics for the sake of numbers, but for clients: so that there are working servers in the data center, and updated information on the site. Then each client receives:

• server of the required capacity - payment according to the tariff;
• reliable equipment - no interruptions in the work of projects.