October 27, 2012 at 23:11Mass recording from cameras in elections
Attention. Habr - not for politics. Please refrain from discussing it in the comments.
In anticipation of the first elections in Russia, during which all polling stations were equipped with webcams, many expressed a desire to record video from the camera for themselves. For this, a variety of solutions were offered, from recording by FRAPS, to using ffmpeg and so on. The most successful, in my opinion, was the Qwertovsky utility , laid out here.
In this small topic, I would like to offer my solution and briefly recall how the whole system works in general, since tomorrow’s parliamentary elections will take place in fraternal Ukraine, during which anyone can watch on the site vybory2012.gov.ua .
So what you should know:
For each camera, a fresh playlist is generated every 15 seconds, which contains direct links to the last 4 pieces of video, the duration of each piece is 15 seconds. Accordingly, once a minute the playlist is completely updated, and chunks continue to be available for some time.
The playlist is duplicated on several servers, available via a link of the form http: // server /variant.m3u8?cid= uid & var = orig and it looks something like this:
A link of the form /segment.ts?cid= f0ffd596-aaa6-4601-9432-70d717dd666a & var = orig & ts = 1351335728.24-1351335741.20 in this case denotes a piece of video from the camera f0ffd596-aaa6-4601-9432-70d717dd666a and an interval of 135.13351,335 in 728. complicated Unix time format.
Periodically, once a minute, parsing the playlist and downloading all available pieces will give us as a result maximum information from the camera, except for those moments when Internet access will disappear on the side of the camera. For example, something like this:
We run this script, indicating to it the folder for storing chunks and, optionally, the address of the syslog server where all current information will be poured.
However, we ignored one point - where to get the source data for playlists, namely the server and camera id? It is worth making a small digression. This system was successfully used in elections in Russia, in particular, with the help of the aforementioned python script, I was able to record all the cameras in my city. However, since then there have been some changes in the engine. Previously, all the necessary data could be obtained without even logging in. Now, you first need to add the camera to your favorites, then at / account / channels? Station_id = cid(where cid is the camera id) a file with camera hashes and server IP addresses will become available. Somewhere around two o’clock in the afternoon, I decided to collect a complete hash database and put it in public, but the servers now, even before the main load, periodically throw 502 errors, which complicates the process =) Of 32,183 sections, hashes are available from about 5,000, and little by little this figure is increasing. Current data
In principle, current data can be obtained by opening the desired camera and running CorePlayer.instances.core_player_1.source.origin in Firebug or its equivalents, but it will only get the current server (and it will fall off, judging by the experience of the Russian elections).
At the moment, plugs are shown on the cameras, apparently Ukrainian colleagues took into account the Russian experience, and decided not to show the everyday life of schools, libraries and hostels.
UPD 7:50 MSK: Data collected on 99% of the cameras. The picture from the cameras has gone, the script successfully completes.
In anticipation of the first elections in Russia, during which all polling stations were equipped with webcams, many expressed a desire to record video from the camera for themselves. For this, a variety of solutions were offered, from recording by FRAPS, to using ffmpeg and so on. The most successful, in my opinion, was the Qwertovsky utility , laid out here.
In this small topic, I would like to offer my solution and briefly recall how the whole system works in general, since tomorrow’s parliamentary elections will take place in fraternal Ukraine, during which anyone can watch on the site vybory2012.gov.ua .
So what you should know:
For each camera, a fresh playlist is generated every 15 seconds, which contains direct links to the last 4 pieces of video, the duration of each piece is 15 seconds. Accordingly, once a minute the playlist is completely updated, and chunks continue to be available for some time.
The playlist is duplicated on several servers, available via a link of the form http: // server /variant.m3u8?cid= uid & var = orig and it looks something like this:
#EXTM3U
#EXT-X-MEDIA-SEQUENCE:6885
#EXT-X-TARGETDURATION:15
#EXT-X-ALLOW-CACHE:NO
#EXT-X-PROGRAM-DATE-TIME:2012-10-27T11:02:08Z
#EXTINF:13,
/segment.ts?cid=f0ffd596-aaa6-4601-9432-70d717dd666a&var=orig&ts=1351335728.24-1351335741.20
#EXTINF:11,
/segment.ts?cid=f0ffd596-aaa6-4601-9432-70d717dd666a&var=orig&ts=1351335741.20-1351335752.28
#EXTINF:11,
/segment.ts?cid=f0ffd596-aaa6-4601-9432-70d717dd666a&var=orig&ts=1351335752.28-1351335763.40
#EXTINF:14,
/segment.ts?cid=f0ffd596-aaa6-4601-9432-70d717dd666a&var=orig&ts=1351335763.40-1351335776.92
#EXTINF:11,
/segment.ts?cid=f0ffd596-aaa6-4601-9432-70d717dd666a&var=orig&ts=1351335776.92-1351335788.36
A link of the form /segment.ts?cid= f0ffd596-aaa6-4601-9432-70d717dd666a & var = orig & ts = 1351335728.24-1351335741.20 in this case denotes a piece of video from the camera f0ffd596-aaa6-4601-9432-70d717dd666a and an interval of 135.13351,335 in 728. complicated Unix time format.
Periodically, once a minute, parsing the playlist and downloading all available pieces will give us as a result maximum information from the camera, except for those moments when Internet access will disappear on the side of the camera. For example, something like this:
# -*- coding: utf-8 -*-
# vybory2012 (Proof of concept), yegorov-p
import urllib
import os
from time import strftime, localtime, sleep
import socket
import threading
#Адрес сервера syslog, куда сыпалась вся статистика
syslog_server='127.0.0.1'
syslog_port=514
#Корневая папка, куда будем складировать чанки
directory='dumps'
#Массив с id камер, их хешами и списком серверов, откуда можно тянуть чанки
cams=[
['563-1', "f0ffd596-aaa6-4601-9432-70d717dd666a",["82.207.0.3","82.207.0.3","82.207.0.3"]]
]
#обозначения уровней для сислога
LEVEL = {
'emerg': 0, 'alert':1, 'crit': 2, 'err': 3,
'warning': 4, 'notice': 5, 'info': 6, 'debug': 7
}
#процедурка, пишущая в сислог. Копирайты потеряны, сорри =(
def syslog(message, level=LEVEL['notice'], host=syslog_server, port=syslog_port):
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
data = '<%d>%s' % (level + 24, message)
sock.sendto(data, (host, port))
sock.close()
#Основная процедура
def cam_rip(num,hash,servers):
syslog('Recording cam %s at %s'%(hash, num), level=LEVEL['info'])
try:
os.mkdir('./%s/%s'%(directory,num))
except:
pass
#Запускаем бесконечный цикл
while 1:
try:
server=servers[0]
#Скачиваем текущий плейлист
page = urllib.urlopen("http://%s/variant.m3u8?cid=%s&var=orig"%(server,hash)).read()
#Если плейлист кошерен, выдираем ссылки на отдельные чанки и скачиваем их
if '/segment' in page:
for i in page.split('\n'):
if '/segment' in i:
filename=strftime("%d-%b-%H-%M-%S", localtime(int(i[-13:-3])))
f=open('./%s/%s/%s.ts'%(directory,num,filename),'wb')
#syslog('Chunk %s saved'%(filename), level=LEVEL['notice'])
f.write(urllib.urlopen("http://%s%s"%(server,i)).read())
f.close()
else:
#В противном случае ругаемся в сислог и делаем ротацию сервера
syslog('No signal!Rotating server on cam %s at %s'%(hash,num), level=LEVEL['err'])
servers.append(servers[0])
del servers[0]
sleep(60)
except Exception,e:
syslog('Error on cam %s: %s'%(hash,e), level=LEVEL['err'])
servers.append(servers[0])
del servers[0]
try:
os.mkdir(directory)
except:
pass
#Запускаем кучу потоков для каждой из камер
for i in cams:
threading.Thread(target=cam_rip, kwargs={"num": i[0],"hash": i[1],"servers": i[2]}).start()
sleep(1)
syslog('System started.', level=LEVEL['notice'])
We run this script, indicating to it the folder for storing chunks and, optionally, the address of the syslog server where all current information will be poured.
However, we ignored one point - where to get the source data for playlists, namely the server and camera id? It is worth making a small digression. This system was successfully used in elections in Russia, in particular, with the help of the aforementioned python script, I was able to record all the cameras in my city. However, since then there have been some changes in the engine. Previously, all the necessary data could be obtained without even logging in. Now, you first need to add the camera to your favorites, then at / account / channels? Station_id = cid(where cid is the camera id) a file with camera hashes and server IP addresses will become available. Somewhere around two o’clock in the afternoon, I decided to collect a complete hash database and put it in public, but the servers now, even before the main load, periodically throw 502 errors, which complicates the process =) Of 32,183 sections, hashes are available from about 5,000, and little by little this figure is increasing. Current data
In principle, current data can be obtained by opening the desired camera and running CorePlayer.instances.core_player_1.source.origin in Firebug or its equivalents, but it will only get the current server (and it will fall off, judging by the experience of the Russian elections).
At the moment, plugs are shown on the cameras, apparently Ukrainian colleagues took into account the Russian experience, and decided not to show the everyday life of schools, libraries and hostels.
UPD 7:50 MSK: Data collected on 99% of the cameras. The picture from the cameras has gone, the script successfully completes.