Transfer content distribution to BitTorrent

Under the cat, an example of transferring file servers to BitTorrent is described.

My friend has a small local area network, there are 200-300 subscribers, in fact, a regular local area network. Several game servers, several file servers and a garland of unmanaged switches. Typical Pioneer. Net beginning-mid-zero. Everything works, everyone is happy. But with the increase in the number of subscribers, file servers experienced problems with the delivery of content. After much deliberation and other experiments, it was decided to switch to bittorrent. And now for almost a year and a half there have been no problems with the servers, and my friend does not cease to rejoice at this.
What kind of server should be, which system to choose, how many disks to install, whether to use raid, lvm, which file system to prefer - each administrator has his own achievements. Personally, I prefer raid5 from 5-6 hard drives and xfs, the performance obtained from such a bundle completely suits me.
Wikipedia quick reference: BitTórrent is a peer-to-peer (P2P) network protocol for cooperative file sharing over the Internet. Files are transferred in parts, each torrent client, receiving (downloading) these parts, at the same time gives (uploads) them to other clients, which reduces the load and dependence on each client source and ensures data redundancy. Before starting the download, the client connects to the tracker at the address indicated in the torrent file, tells it its address and the hash amount of the torrent file, to which the client receives the addresses of other clients downloading or distributing the same file.
Let's start with the tracker, I chose opentracker. Easy to install and configure, undemanding to resources. In the settings of the tracker, I allowed the registration of any torrent files, so I wanted it, and even easier.

fs:~# apt-get install gcc make cvs
fs:~# cvs -d :pserver:cvs@cvs.fefe.de:/cvs -z9 co libowfat
fs:~# cd libowfat/
fs:~/libowfat# make
fs:~/libowfat# cd ..
fs:~# cvs -d:pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co opentracker
fs:~# cd opentracker/
fs:~/opentracker# make
fs:~/opentracker# cp opentracker /usr/local/bin/
fs:~# cat /etc/rc.local |grep opentracker
start-stop-daemon --start --quiet -m -b --pidfile /var/run/opentracker.pid --exec /usr/local/bin/opentracker


I chose btpb as the client. Again, easy to install and configure, very undemanding to resources and without any patches it works fine with a large number of files. We will generate torrent files using ctorrent. Actually now we have a torrent tracker, a torrent client and a bunch of files that need to be distributed to users. Suppose, in addition to a bunch of files, there is also a directory with a beautiful interface and a database where all the information we need is stored. In my case, there was a files plate in which the file path and the unique file id were stored. To control the generation of torrent files, I created an additional torrents table

fs:~# wget www.murmeldjur.se/btpd/btpd-0.15.tar.gz
fs:~# tar -xf btpd-0.15.tar.gz
fs:~# cd btpd-0.15
fs:~/btpd-0.15# chmod +x configure
fs:~/btpd-0.15# ./configure
fs:~/btpd-0.15# make
fs:~/btpd-0.15# make install
fs:~# cat /etc/rc.local |grep btpd
/usr/local/bin/btpd -d /root/.btpd


fs:~# apt-get install ctorrent






mysql> desc torrents;

+--------------+---------------+------+-----+---------+----------------+
| Field     | Type     | Null | Key | Default | Extra     |
+--------------+---------------+------+-----+---------+----------------+
| id        | int(16)    | NO   | PRI | NULL  | auto_increment |
| file_id    | int(16)    | YES   |    | NULL  |        |
| is_torrent | enum('1','0') | YES |    | 0     |        |
+------------+---------------+------+-----+---------+----------------+
3 rows in set (0.01 sec


and 3 triggers

mysql> show triggers;

+-----------+--------+-------+--------
| Trigger  | Event | Table | Statement | Timing |
+-----------+--------+-------+--------
| on_insert|INSERT|files| insert into torrents (file_id) values(LAST_INSERT_ID())| AFTER
| on_update|UPDATE|files| update torrents set is_torrent='0' where file_id=NEW.id | AFTER
| on_delete|DELETE|files| delete torrents.* from torrents where torrents.file_id=old.id | AFTER
+-----------+--------+-------+--------


The table stores data from torrent files, and thanks to triggers the creation, regeneration, or removal of the torrent file occurs.

Now the script itself controls the state of torrent files.
fs:~# cat add_torrent.sh


#!/bin/bash
#определяем адрес трекера
tracker="http://tracker_domain:6969/announce"
#корневая папка где храниться контент
video_home="/home/video";
#папка для хранения торрент файлов
torrent_dir=" /var/www/torrents";

ctorrent="/usr/bin/ctorrent";
btcli="/usr/local/bin/btcli";
mysql="/usr/bin/mysql -pdpass -u duser -D video";

#получаем id файлов для которых не сгенерирован торрент
for i in `echo "select files.id from files,torrents where torrents.file_id=files.id and torrents.is_torrent='0' limit 1;"|${mysql}|sed 1d`
do
id=${i};
#получаем полный путь к файлу
path=`echo "select files.path from files where files.id="${id}" limit 1;"|${mysql}|sed 1d`
#формируем название торрента
torrent_file=${torrent_dir}/${id}.torrent
#если торрент уже был сгенерирован удаляем его из торрент клиента
if [ -e ${torrent_file} ]
then
[ `${btcli} list|awk '{print $1}'|grep -w ${id}|wc -l` -ne 0 ] && ${btcli} del ${torrent_file}
rm ${torrent_file}
fi
cd ${video_home}
path1=`dirname "${path}"`
#генерируем торрент и если все прошло успешно обновляем базу данных
${ctorrent} -t -u ${tracker} -s ${torrent_file} "${path}" && echo "update torrents set is_torrent='1' where file_id="${id}|${mysql}
#если торрент сгенерился добавляем в торрент клиент
[ -e ${torrent_file} ] && ${btcli} add -d "${path1}" -n ${id} --topdir ${torrent_file}
done

#проверяем все ли сгенерированные торрент файлы загружены
ls ${torrent_dir}|sed 's/.torrent//'|sort -n > /tmp/tor_list.files
${btcli} list|awk '{print $1}'|sed 1d|sort -n > /tmp/tor_list.load

diff_list=(`diff /tmp/tor_list.files /tmp/tor_list.load|grep "<"|awk '{print $2}'`)
diff_num=$((${#diff_list[*]}-1))
for i in `seq 0 $diff_num`
do
id=${diff_list[${i}]};
path=`echo "select files.Path from files where files.id="${id}";"|${mysql}|sed 1d`
torrent_file=${torrent_dir}/${id}.torrent
${btcli} del ${torrent_file}
if [ -z ${path[0]} ]
then
rm ${torrent_file};
else
path=`dirname "$path"`
${btcli} add -d ${path} -n ${id} --topdir ${torrent_file}
fi
done


As you can see from the listing, the script generates torrent files, if the torrent already existed, regenerates if necessary, it also makes sure that all generated torrents are added to the client.
For normal operation of the system, it is enough to run this script once 2-3 minutes.
The only thing that is not rationalized is the removal of torrent files and content when deleting records about it from the database.
I’m not afraid to repeat this system has been working for a year and a half and there are no problems, the only thing I recommend is not to use the Cyrillic in the names of files and folders, anything can happen.