Transfer content distribution to BitTorrent

    Under the cat, an example of transferring file servers to BitTorrent is described.

    My friend has a small local area network, there are 200-300 subscribers, in fact, a regular local area network. Several game servers, several file servers and a garland of unmanaged switches. Typical Pioneer. Net beginning-mid-zero. Everything works, everyone is happy. But with the increase in the number of subscribers, file servers experienced problems with the delivery of content. After much deliberation and other experiments, it was decided to switch to bittorrent. And now for almost a year and a half there have been no problems with the servers, and my friend does not cease to rejoice at this.
    What kind of server should be, which system to choose, how many disks to install, whether to use raid, lvm, which file system to prefer - each administrator has his own achievements. Personally, I prefer raid5 from 5-6 hard drives and xfs, the performance obtained from such a bundle completely suits me.
    Wikipedia quick reference: BitTórrent is a peer-to-peer (P2P) network protocol for cooperative file sharing over the Internet. Files are transferred in parts, each torrent client, receiving (downloading) these parts, at the same time gives (uploads) them to other clients, which reduces the load and dependence on each client source and ensures data redundancy. Before starting the download, the client connects to the tracker at the address indicated in the torrent file, tells it its address and the hash amount of the torrent file, to which the client receives the addresses of other clients downloading or distributing the same file.
    Let's start with the tracker, I chose opentracker. Easy to install and configure, undemanding to resources. In the settings of the tracker, I allowed the registration of any torrent files, so I wanted it, and even easier.

    fs:~# apt-get install gcc make cvs
    fs:~# cvs -d -z9 co libowfat
    fs:~# cd libowfat/
    fs:~/libowfat# make
    fs:~/libowfat# cd ..
    fs:~# cvs co opentracker
    fs:~# cd opentracker/
    fs:~/opentracker# make
    fs:~/opentracker# cp opentracker /usr/local/bin/
    fs:~# cat /etc/rc.local |grep opentracker
    start-stop-daemon --start --quiet -m -b --pidfile /var/run/ --exec /usr/local/bin/opentracker

    I chose btpb as the client. Again, easy to install and configure, very undemanding to resources and without any patches it works fine with a large number of files. We will generate torrent files using ctorrent. Actually now we have a torrent tracker, a torrent client and a bunch of files that need to be distributed to users. Suppose, in addition to a bunch of files, there is also a directory with a beautiful interface and a database where all the information we need is stored. In my case, there was a files plate in which the file path and the unique file id were stored. To control the generation of torrent files, I created an additional torrents table

    fs:~# wget
    fs:~# tar -xf btpd-0.15.tar.gz
    fs:~# cd btpd-0.15
    fs:~/btpd-0.15# chmod +x configure
    fs:~/btpd-0.15# ./configure
    fs:~/btpd-0.15# make
    fs:~/btpd-0.15# make install
    fs:~# cat /etc/rc.local |grep btpd
    /usr/local/bin/btpd -d /root/.btpd

    fs:~# apt-get install ctorrent

    mysql> desc torrents;
    | Field     | Type     | Null | Key | Default | Extra     |
    | id        | int(16)    | NO   | PRI | NULL  | auto_increment |
    | file_id    | int(16)    | YES   |    | NULL  |        |
    | is_torrent | enum('1','0') | YES |    | 0     |        |
    3 rows in set (0.01 sec

    and 3 triggers

    mysql> show triggers;
    | Trigger  | Event | Table | Statement | Timing |
    | on_insert|INSERT|files| insert into torrents (file_id) values(LAST_INSERT_ID())| AFTER
    | on_update|UPDATE|files| update torrents set is_torrent='0' where | AFTER
    | on_delete|DELETE|files| delete torrents.* from torrents where | AFTER

    The table stores data from torrent files, and thanks to triggers the creation, regeneration, or removal of the torrent file occurs.

    Now the script itself controls the state of torrent files.
    fs:~# cat

    #определяем адрес трекера
    #корневая папка где храниться контент
    #папка для хранения торрент файлов
    torrent_dir=" /var/www/torrents";

    mysql="/usr/bin/mysql -pdpass -u duser -D video";

    #получаем id файлов для которых не сгенерирован торрент
    for i in `echo "select from files,torrents where and torrents.is_torrent='0' limit 1;"|${mysql}|sed 1d`
    #получаем полный путь к файлу
    path=`echo "select files.path from files where"${id}" limit 1;"|${mysql}|sed 1d`
    #формируем название торрента
    #если торрент уже был сгенерирован удаляем его из торрент клиента
    if [ -e ${torrent_file} ]
    [ `${btcli} list|awk '{print $1}'|grep -w ${id}|wc -l` -ne 0 ] && ${btcli} del ${torrent_file}
    rm ${torrent_file}
    cd ${video_home}
    path1=`dirname "${path}"`
    #генерируем торрент и если все прошло успешно обновляем базу данных
    ${ctorrent} -t -u ${tracker} -s ${torrent_file} "${path}" && echo "update torrents set is_torrent='1' where file_id="${id}|${mysql}
    #если торрент сгенерился добавляем в торрент клиент
    [ -e ${torrent_file} ] && ${btcli} add -d "${path1}" -n ${id} --topdir ${torrent_file}

    #проверяем все ли сгенерированные торрент файлы загружены
    ls ${torrent_dir}|sed 's/.torrent//'|sort -n > /tmp/tor_list.files
    ${btcli} list|awk '{print $1}'|sed 1d|sort -n > /tmp/tor_list.load

    diff_list=(`diff /tmp/tor_list.files /tmp/tor_list.load|grep "<"|awk '{print $2}'`)
    for i in `seq 0 $diff_num`
    path=`echo "select files.Path from files where"${id}";"|${mysql}|sed 1d`
    ${btcli} del ${torrent_file}
    if [ -z ${path[0]} ]
    rm ${torrent_file};
    path=`dirname "$path"`
    ${btcli} add -d ${path} -n ${id} --topdir ${torrent_file}

    As you can see from the listing, the script generates torrent files, if the torrent already existed, regenerates if necessary, it also makes sure that all generated torrents are added to the client.
    For normal operation of the system, it is enough to run this script once 2-3 minutes.
    The only thing that is not rationalized is the removal of torrent files and content when deleting records about it from the database.
    I’m not afraid to repeat this system has been working for a year and a half and there are no problems, the only thing I recommend is not to use the Cyrillic in the names of files and folders, anything can happen.

    Also popular now: