Remote reinstallation of Linux via ssh without access to the console

I needed to reinstall the server, which seemed to be hosted by friends of my acquaintances. There was a very outdated Debian, and, most importantly, the system was on regular partitions without lvm and the space was allocated very poorly. It was almost impossible to get physical access to it; it was possible to ask a local administrator to do something, but it could take a week. The server had a virtual KVM, but it was impossible to access it from the outside; as if the hoster had no extra IP addresses, and it was impossible to get into its network. It was necessary to reinstall the server from under a running system via ssh. Yeah, let's change the rotor at the turbine without turning it off, then restart it and it will work with the new rotor!

The first idea was to create a chroot environment on a ram-disk and create lvm from it and fill the system. But there it was, the system does not allow changing the partition table.

The second idea was to take the sources of the Debian distribution, flush the server IP address into them, rebuild the initrd with the Debian installer, ssh server and my IP, substitute this initrd into the grub config with the default block and reboot. After that, I had to get the ssh console with the network installer. At the stand I did it! But in battle, everything ended in failure, the server did not rise. The hosts didn’t really need the server, and the business died down, but I still have a feeling of an unsolved problem.

Somehow, colleagues discussed all sorts of destructive actions with the system (such as rm -rf /) and one of the colleagues said that you can disconnect the scsi disk on which the root partition is located and the system will not pick up. This gave me the idea of ​​number three, take the idea of ​​one, tear off the disk, return the disk and the returned disk will be different, not the one that the system did not give. That is exactly what happened. And now, point by point, how can I reinstall the system without access to a physical console?

Warning! You need to understand that all we will do is the one-way road, if we make a mistake we lose access to the system! It is possible that you will have to travel 1,500 kilometers and climb into the mine in order to reanimate the server.

We assume that the IP of our system is 192.168.56.102. That was exactly what was on my stand. Plus access to the Internet through a proxy:

http://proxy:8080

Getting started on the source system.

# System # 0

We go by ssh to the server:

ssh 192.168.56.102

We create a directory and a file system for the “Killer System”, mount it:

mkdir /target
mount none -t tmpfs -o size=1G /target/

We install the excellent debootstrap utility, which deploys the minimal installation of Debian, using it we will create a chroot environment:

export http_proxy='http://proxy:8080'
apt-get -y install debootstrap

There are similar utilities for Fedora and Centos, respectively febootstrap and yumbootstrap, but I did not work with them.

Expand chroot:

debootstrap jessie /target/ http://mirror.mephi.ru/debian/

The first argument is the version, the second is the installation directory, the third is the repository.

We backup the most necessary:

mkdir /target/backup
cp /etc/network/interfaces /target/backup

The most important thing is the settings of network interfaces, without them you will not be able to get into the reinstalled system.

Name the chroot environment:

echo "Killer_system" > /target/etc/debian_chroot

The word "Killer_system" will appear at the bash prompt. This is an important thing, without it it will not be clear where we are at the moment.

We move into a new environment.

# System # 1

chroot /target

Mount useful fs:

mount none -t proc /proc/
mount none -t sysfs /sys/
mount none -t devtmpfs /dev/
mount none -t devpts /dev/pts/

Once again, put debootstrap:

apt-get -y install lvm2 debootstrap

Further my troubles: the Debian openssh-server package in the recommended packages has the xauth package, and it has all sorts of x libraries in its dependencies. I, as a supporter of minimalism, do not want x-bits to be placed on a server where there were no graphics. Therefore, we put with the key --no-install-recommends:

apt-get -y install openssh-server openssh-client openssh-blacklist openssh-blacklist-extra --no-install-recommends

We correct configs. We set an alternative port for the ssh daemon so that we can log in to the chroot system via ssh:

sed -i 's/^Port .*$/Port 11122/' /etc/ssh/sshd_config

And allow root access:

sed -i 's/^PermitRootLogin .*$/PermitRootLogin yes/' /etc/ssh/sshd_config
/etc/init.d/ssh restart

You can not give root access, but create a user and give him sudo rights, but here I deliberately simplify.

Next, you need to set the root password, since debootstrap does not set any passwords by default:

passwd root

We go into the chroot environment by ssh:

ssh 192.168.56.102 -l root -p 11122

We are doing this in order to completely get rid of the old system, from which we will tear off the disks. And so we will have a fully autonomous system in RAM, not connected with the old one.

Such a trick is very well suited if we leave the hoster, but we really do not want to leave our files to it (I know, paranoia). At this stage, we simply drive the disks with zeros if we want to quickly:

dd if=/dev/zero of=/dev/sda bs=1M

Or random data in several passes, if we want well. The advantage of the method is that we can wait until dd is finished and, if necessary, repeat it. If you overwrite the disks directly from the combat system, then we will not be able to look at the results of dd.

Let's try a simple way, delete volumes and partitions:

# lvremove /dev/mapper/vg_old-root
  Logical volume vg_root/lv_root contains a filesystem in use.

# fdisk /dev/sda
Command (m for help): d
Selected partition 1
Partition 1 has been deleted.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

But failure. In this case, the partition will be deleted and the system will crash, but it will not work in a simple way without a reboot. And there will be nowhere to be overloaded.

We will go the other way. We check where we have what is located:

pvs
lsblk

We assume that the root partition is on the sda ​​drive.

We overwrite the disk so that in no case lvm picks it up.

Warning! After this moment, there is no return, even the next step is not so malicious. Let's think for a minute, check the console, at which we sit and justify the name of our chroot:

dd if=/dev/zero of=/dev/sda bs=1M count=100

We tear off disks:

echo 1 > /sys/block/sda/device/delete

We check that the disk has come off:

lsblk

We connect the disk back:

for i in /sys/class/scsi_host/host?/scan ; do echo "- - -" > $i ; done

Check that returned:

lsblk

Was sda, became sdb, great.

An important point: on the boot disk, you must create one primary partition the size of the entire disk and give this partition to lvm so that grub can stand on it. All other disks can be given to lvm entirely without creating a partition system (pvcreate / dev / sdc). Create a partition table and one primary partition of type 8e, Linux LVM:

fdisk /dev/sdb
n
t 8e
w
# create new primary partition from start to end; 8e type

The original version of the script was creating one logical volume for the whole system, but when my colleague reinstalled Linux using this script, it turned out that creating several partitions was a bit difficult, especially a separate section for logs. Attention should be paid to the procedure for creating mount points and actually mounting partitions.

pvcreate /dev/sdb1
vgcreate vg_root /dev/sdb1
lvcreate -Zn -L500M -n lv_swap0 vg_root
lvcreate -Zn -L1G -n lv_root vg_root
lvcreate -Zn -L2G -n lv_usr vg_root
lvcreate -Zn -L2G -n lv_var vg_root
lvcreate -Zn -L1G -n lv_var_log vg_root
lvcreate -Zn -L1G -n lv_home vg_root
mkswap /dev/vg_root/lv_swap0
mkfs.ext4 /dev/mapper/vg_root-lv_root
mkfs.ext4 /dev/mapper/vg_root-lv_usr
mkfs.ext4 /dev/mapper/vg_root-lv_var
mkfs.ext4 /dev/mapper/vg_root-lv_var_log
mkfs.ext4 /dev/mapper/vg_root-lv_home
mkdir /target
mount /dev/mapper/vg_root-lv_root /target/
mkdir /target/usr /target/var /target/home
mount /dev/mapper/vg_root-lv_usr /target/usr
mount /dev/mapper/vg_root-lv_var /target/var
mkdir /target/var/log
mount /dev/mapper/vg_root-lv_var_log /target/var/log
mount /dev/mapper/vg_root-lv_home /target/home

We are already deploying the combat system to a new place on the hard drive:

export http_proxy='http://proxy:8080'
debootstrap jessie /target/ http://mirror.mephi.ru/debian/
echo "NEW_system" > /target/etc/debian_chroot

We return to place backup copies of configs:

cp /backup/interfaces /target/etc/network

Now a new system awaits us:

# System # 2

chroot /target

Notice in the command prompt now the name of the new chroot environment.

Mount file systems:

mount none -t proc /proc/
mount none -t sysfs /sys/
mount none -t devtmpfs /dev/
mount none -t devpts /dev/pts/

You can also mount these file systems from the parent chroot:

mount -o bind /proc/ /target/proc
mount -o bind /sys/ /target/sys
mount -o bind /dev/ /target/dev
mount -o bind /dev/pts /target/dev/pts

Install and configure openssh:

apt-get -y install openssh-server openssh-client openssh-blacklist openssh-blacklist-extra --no-install-recommends

sed -i 's/^PermitRootLogin .*$/PermitRootLogin yes/' /etc/ssh/sshd_config
passwd root

Install packages that are indispensable:

apt-get -y install vim sudo linux-image-3.16.0-4-amd64 grub2 lvm2 psmisc vlan 

Yes, I can't live without vim and hate nano:

update-alternatives --set editor /usr/bin/vim.basic

In principle, grub is written where it is necessary even during installation, but, nevertheless, to support pants and morale, we repeat:

update-grub
grub-install /dev/sdb

Now edit the configs, at first the most important, without which the system will not rise:

cat > /etc/fstab <
/dev/mapper/vg_root-lv_root      /               ext4    errors=remount-ro 0       1
/dev/mapper/vg_root-lv_usr       /usr            ext4    defaults          0       2
/dev/mapper/vg_root-lv_var       /var            ext4    defaults          0       2
/dev/mapper/vg_root-lv_var_log   /var/log        ext4    defaults          0       2
/dev/mapper/vg_root-lv_home      /home           ext4    defaults          0       2
EOF

Everything should be in order in the interfaces file, because somehow the network worked for us?

vim /etc/network/interfaces

In the apt's config, add the proxy information:

echo 'Acquire::http::Proxy "http://proxy:8080";' > /etc/apt/apt.conf

Change hostname:

echo new-system > /etc/hostname

Add a line to / etc / hosts:

echo "192.168.56.102 new-system.corp new-system" >> /etc/hosts

Add admin:

adduser admin
usermod -a -G sudo admin
visudo

Unmount file systems:

umount /dev/pts
umount /dev/
umount /proc/
umount /sys/

And exit the chroot:

exit

Unmount file systems:

umount /target/usr/ /target/var/log/ /target/var/ /target/home/

If / dev was not able to be unmounted, then / target cannot be unmounted, but this is not scary.

If successful, then do this:

umount /target/

If not, then like this:

sync ; sync ; sync ; mount -o remount,ro /target/

These commands will flush disk caches and remount the root file system in read only. After that, you can overload.

Here we are waiting for a surprise from everyone beloved systemd! He knows that we are in a chroot and does not allow to reboot! Google gives advice to exit chroot, but we have nowhere to go. But Magic SysRq comes to the rescue!

We activate SysRq (it is most likely activated, but do we need to make sure?).

echo 1 > /proc/sys/kernel/sysrq

And overloaded:

echo b > /proc/sysrq-trigger

Drum roll, alarming expectation, have we really forgotten something, and the server has not risen?

ssh 192.168.56.102

Hurrah! We are in the new system!

Recreating initrd. This is not necessary, but in the future it will eliminate some errors during reboot:

update-initramfs -u

Delete the file with the name chroot environment:

rm /etc/debian_chroot

That's all.

## The end

References:

An interesting article on server overload methods