HipChat Server data migration from self-hosted VM to Amazon EC2

The company I work for now has been using HipChat Server for internal communications in the development team and admins for about a year now. Moreover, this is a self-hosted virtual machine on KVM with qcow2 disk image. And now, recently there was a need to transport internal services to AWS, I began to look for ways. In fact, there were few options:
- Converting qcow2> raw> ami.
- The official approach is the export of chats, users, rooms and files (without passwords, integrations and api keys).
- The third option, which I will describe below.

I decided to postpone the first two options for later. I did not want to consider the official method, because I really did not want to have problems with recovering passwords and keys, although this would not have taken much time in my case. And the option to convert to an AMI image is simply less interesting.

Initial data - HipChat 1.4.1 server at the source and at the receiver. The source is the kvm / qcow2 virtual machine. The receiver is a clean (without users, not counting me) HipChat Server EC2 instance but with the initial setup and trial license. The configuration is the same, the disk size is the same (1 CPU, 2Gb RAM, 50Gb HDD).

# VARS
SOURCE=hipchat.example.com
DEST=new_hipchat.example.com

So, I decided to dig a bit into the insides of the server and figure out which files and settings of which services you need to transfer. A quick review revealed that the following technology stack is used:
- nginx,
- php5-fpm,
- mysql,
- elasticsearch,
- redis,
- gearman-job-server,
- memcached,
- and a few other related services.
Plus a number of scripts in python, ruby, chef-solo.
Is that familiar?
I did not dig far, I just needed to find out what needs to be transferred and how.

It was possible to dump the database (mysql, elasticsearch, redis), but I decided that in any case I would completely stop all services at the source and at the receiver, so I would just copy files from the source to the receiver.
Before you start, you need to (yes, it would be necessary) to warn the team that n-hours will be unavailable (it is more logical to do this after hours), then stop all services (both at the source and at the receiver) from the list below:

# ON $DEST AND $SOURCE
/etc/init.d/cron stop
/etc/init.d/monit stop
/etc/init.d/nginx stop
/etc/init.d/php5-fpm stop
/etc/init.d/hipchat stop
/etc/init.d/integrations-0 stop
/etc/init.d/tetra-proxy stop
/etc/init.d/tetra-proxy-0 stop
/etc/init.d/tetra-app-0 stop
/etc/init.d/barb-0 stop
/etc/init.d/coral-0 stop
/etc/init.d/crowd stop
/etc/init.d/cumulus stop
/etc/init.d/curler stop
/etc/init.d/elasticsearch stop
/etc/init.d/gearman-job-server stop
/etc/init.d/memcached stop
/etc/init.d/redisserver stop
/etc/init.d/mysql stop

I did not immediately guess about crowns, it turned out there is a task that monit periodically checks, and that in turn checks the other services (in general, everything is logical, well done). In fact, not all services from this list work - for example, I did not remember tetra-proxy, for example, some other one.

Scrolling through the contents of the directories, I put together a list of the necessary (not the first time, something empirically):

# Copy all files from $SOURCE to $DEST
rsync -avz /etc/nginx/ $DEST:/etc/nginx/
rsync -avz /etc/mariadb_grants $DEST:/etc/mariadb_grants
rsync -avz /etc/chef/ $DEST:/etc/chef/
rsync -avz /etc/crowd/ $DEST:/etc/crowd/
rsync -avz /etc/mysql/debian.cnf $DEST:/etc/mysql/debian.cnf
rsync -avz --del /chat_history/ $DEST:/chat_history/
rsync -avz --del /data_bags/ $DEST:/data_bags/
rsync -avz --exclude='file_store/archive/pool' /file_store/ $DEST:/file_store/
rsync -avz --del /hipchat/ $DEST:/hipchat/
rsync -avz --del /hipchat-scm/ $DEST:/hipchat-scm/
rsync -avz --del --exclude='home/admin/.ssh' /home/ $DEST:/home/
rsync -avz --del /ops/ $DEST:/ops/
rsync -avz --del /opt/ $DEST:/opt/
rsync -avz --del /var/lib/mysql/ $DEST:/var/lib/mysql/
rsync -avz --del /var/lib/redis/ $DEST:/var/lib/redis/
rsync -avz --del /var/lib/cloud/ $DEST:/var/lib/cloud/

It is important not to overwrite .ssh / authorized_keys so that you do not have to worry about regaining access to the instance!

To start copying files from source to receiver, you first need to configure key access and an ssh server. The config has a whitelist by default and prohibits root user authorization, so I had to make the following changes temporarily:

editor /etc/ssh/sshd_config
...
PermitRootLogin without-password
...
# Whitelist to HipChat admin
DenyUsers ubuntu hipchat
AllowUsers root admin nessus

and restart ssh:

/etc/init.d/ssh restart

After the copying is completed and hipkip is started, these changes will be automatically rolled back.

When everything is successfully copied, you can try to start all the services back on the receiver (it’s better not to touch the source until you make sure that everything is fine), but it’s better to reboot, or rather I had to do it to make it work.

# ON DEST
/etc/init.d/memcached start
/etc/init.d/redisserver start
/etc/init.d/mysql start
/etc/init.d/cron start
/etc/init.d/monit start
/etc/init.d/nginx start
/etc/init.d/php5-fpm start
/etc/init.d/hipchat start
/etc/init.d/integrations-0 start
/etc/init.d/tetra-proxy start
/etc/init.d/tetra-proxy-0 start
/etc/init.d/tetra-app-0 start
/etc/init.d/barb-0 start
/etc/init.d/coral-0 start
/etc/init.d/crowd start
/etc/init.d/cumulus start
/etc/init.d/curler start
/etc/init.d/elasticsearch start
/etc/init.d/gearman-job-server start

Another important note is that hipchats are tied to dns records and entries in the hosts file, and the latter cannot be corrected (it is overwritten at startup) so that the server “takes off” it must resolve itself using the hostname that is written in dns and which you specified at the primary customization (like hipchat.example.com).

cat /etc/hosts
127.0.0.1       localhost localhost.localdom
# Network nodes
192.168.0.10   hipchat.example.com # Можно было бы указать так, но не выйдет chef все перезапишет при запуске сервера
# Services
192.168.0.10   graphite.hipchat.com  
192.168.0.10   mysql.hipchat.com             
192.168.0.10   redis-master.hipchat.com       
192.168.0.10   redis-slave.hipchat.com       
# IPv6
::1             ip6-localhost ip6-loopback
fe00::0         ip6-localnet
ff00::0         ip6-mcastprefix
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
ff02::3         ip6-allhosts

To check the receiver's operation, you will have to change the dns record for your hip server. Or in the case of AWS, you can use the internal dns server by placing a private zone with your domain in Route 53 and specifying a private-ip instance in it.
And only after that make service hipchat restart (or better reboot, to be sure that when you restart the system everything will be ok), and everything will work.

I intentionally tried not to interfere with the logic that exists in the server, because all my changes would disappear during the next update and could possibly lead to a loss of serviceability of the service.

As a result, my experiment was a success, this morning the guys did not even notice what happened, everyone continued to use team chat without problems.

I do not recommend using this article as a guide to action if you have started such a transfer. There may be errors and inaccuracies in the article. Therefore, do everything at your own peril and risk. Everything needs to be done carefully and clearly understand what you are doing and what it can turn into.

Make backups and check them.

All a long uptime and good mood.