Video surveillance from idea to ... idea

    I have a main job, there are friends who always want to “start a new business”. They have their own, but always want to expand. And every time they want it, they come to me for advice. The idea lies on the surface. "We do the Internet, but let's do something else on our topic." So we began to engage in telephony, selling network equipment, hosting, colocation. The statement of the problem has always pleased me. “Let's do the telephony”, “Let's do the hosting.” Well, the last thing was “Let's do video surveillance.”
    Here “Let's engage in video surveillance" it was the treasured TK.
    What came of this and experience will describe in this article.

    So it has historically, that I work in organizations where I am surrounded by pi salespeople, and IDartanyan is a technical specialist, as it turned out (over time) a wide profile.
    During my professional career, I have built more than one service-technical department, but people did not reach the level of educated (programmers). That's why I am always in sight, all questions on IT are for me. I have been consulting them (friends) for many years in the field of IT, being a system and business analyst, free :). Now to the topic.

    Once people came to me and said "we want video surveillance." I was busy and there was no desire to develop in this direction, then even without this I had enough worries. They found another "specialist", practiced for 6 months, it ended with nothing. The man turned out to be “beautiful”, he sang about the power of zoneminder and motion, that everything will work cloudy. For 6 months, not a single ready-made solution, not a single client.
    They are tired, come to me. Since I’ve been working with them for 10 years now, why not take part in an adventure again. The offer, as always, impressed with its originality and novelty:
    - We want to do cloud video surveillance. Investments 0, we will buy equipment, for development 0, we divide money 50/50.
    By the way, it always has been, and always money was 50/50, or 30/30/30, depending on the number of persons involved.
    Why not, I thought. Experience will not be superfluous. (although it’s worthwhile from this experience if you are a specialist in a wide profile and change this profile once a year :)

    The idea is voiced: "I want cloud video surveillance."

    People who want to sell something, they will always sell something.
    It remains to come up with this "Something" and arrange.

    Cloud. Money 0, the load is big. So you need to come up with a prototype that will solve these problems.
    At first glance fell on Openstack. It is possible to expand computing power on the fly, but several machines are required to run. Having played with different versions of Openstack, I had to look for a more suitable solution. My eyes fell on Proxmox.
    A wonderful solution that suited our needs at the start. And always those VMs can be ported to hardware or another cloud.
    I did not manage to take part in the purchase of iron, before my appearance a workhorse was purchased: Intel Core i5-3570K @ 3.40GHz
    And on this iron it was necessary to do cloud surveillance. An absurd situation :) But we do not give up.
    It comes up with some kind of sales system. Video surveillance in conjunction with security systems and all kinds of alerts with ties to private security companies (there is a way out at the state level).
    This was stuck in the future
    1) continuous video recording
    2) motion recording
    3) timelapse recording
    4) SMS notification
    5) a call from the security center when motion is detected. The operator checks the record, calls the chop, notifies the client.
    The private security company in Moscow will expand from year to year. Mi Police every year reduces the volume of security business. So all this gradually creeps on private traders.

    The general system looked like this:


    Cameras come to any piece of hardware, it can be tied to security. This piece of hardware can be either a software client or a hardware solution on OpenWrt. RPC is drawn for a reason. The idea was to create 2 modules. The first is completely console. This would allow it to be implanted in iron, the second graphic module that controls the console. This would allow you to control the piece of iron from anywhere. From a user's computer, from a support computer, etc. Also, the first module is needed to standardize streams and still frames from cameras. Such glands we have already run in. They were suitable for the prototype, experience was. No one forbade the logic of this piece of iron to be transferred to our server.

    Now there are dances with a tambourine on how to drive the h264 zoo to disk, and to attach detectors and a similar crap in web +, and all this fits on one i5.
    ZoneMinder may be a good system, but after adding 4 cameras, it loaded all 4 system cores by more than 70%. They immediately abandoned him.

    For the prototype took VLC. It has been developed for a long time, it was run in by many, we have used it many times for some projects.
    There are at least 2 tasks that need to be solved
    1) send to the network
    2) record the stream The

    zoo of cameras consists of rtsp and http streams. VLC can do everything. We decided to use VLM. Has control over tlenet and http.
    The link is fed to input, to output
    We get an HTTP stream from any incoming url.
    HTTP is convenient for us than? You can drive in hginx, distribute the load, encrypt. send to https.
    1 client - 1 vlc process.
    They decided to script the prototype in PHP, since there is only one developer, time is
    running out , funding is 0. The launch team. (I will give the finished code, it’s not difficult to get the command out of it, if necessary)
    $vlc_ifs = "-I http --http-host= --http-port {$this->getHttpPort()} -I telnet --telnet-port {$this->getTelnetPort()}  --telnet-password ".TLPWD;
                $vlc_logs = "--extraintf=http:logger --file-logging --log-verbose 0 --logfile {$this->getLogFile()}";
                $vlc_logs = '--extraintf=http';
            //$vlc_shell = VLCBIN." --rtsp-tcp --ffmpeg-hw --http-reconnect --http-continuous --sout-keep ".VLCD." $vlc_ifs  --repeat --loop --network-caching ".VLCNETCACHE." --sout-mux-caching ".VLCSOUTCACHE." $vlc_vlm --pidfile {$this->getPidFile()} $vlc_logs \n";
            $vlc_shell = VLCBIN." --rtsp-tcp ".VLCD." $vlc_ifs --repeat --loop --live-caching ".VLC_LIVE_CACHE." --network-caching ".VLCNETCACHE." --sout-mux-caching ".VLCSOUTCACHE."  --sout-ts-dts-delay 400 $vlc_vlm --pidfile {$this->getPidFile()} $vlc_logs ";
                $vlc_shell = "valgrind -v --trace-children=yes --log-file={$this->getValgrindFile()} --error-limit=no --leak-check=full $vlc_shell";

    Since we already have a live stream, we can write from it.
    return "#std{access=file{append},mux=ts{use-key-frames},dst=$filePath.avi}";

    VLM is created simply
    on the camera, you can create 3 streams
    1) live stream
    2) rec stream
    3) motion stream
    $this->execute("new {$this->cam} broadcast enabled loop");
    after you can play, stop do
    $command = "control {$this->cam} $command";
    enough for rec, for example, do stop every 10 seconds, change $ filePath, do play
    We get a set of files with notes. For motion play and stop.
    VLC can also be launched both with us and with them.
    A matter of movement.
    There is a motion tool. He calculates the movement by Jpeg. Cameras give a still image in Jpeg. We take a freeze frame, feed in motion, at the output we get what we need.

    We have a zoo, we need to make a system out of it.
    So we do. We call everything System and try to make a loosely coupled system so that in the future it is possible to replace system elements without loss.
    We have cameras, Vlc, Motion, recording to disk, online broadcasting.

    Vlc and motion are unix processes. Call them demons. (abstract class Daemon)
    It turns out that we need 2 streams from the camera. jpeg and h264.
    In the system we create the entities
    user - the user
    dvr system that is responsible for the daemons and cameras associated with the daemons
    cam - camera entities, stores the settings of the physical camera and virtual streams
    stream - stream, any abstract stream.

    Straem can be started, stopped, unstable, “pulled” at the time of “updating” the system; on these events, he himself decides what to do.

    We get the System-common entity, which consists only of abstract declarations.

    We get the general factory for building the system.
    abstract class AbstractFactory {
        private static $instance = null;
         * @var array
        private $commands;
         * @return AbstractFactory
        public static function getInstance(){
            if(self::$instance == null) self::$instance = new static;
            return self::$instance;
         * permanentCommand
         * @param ICommand $command
        protected function addPermanentCommand(ICommand $command){
            $this->commands[] = $command;
         * @param ISystem $system
        protected function addCommands(ISystem $system){
            foreach($this->commands as $command)
        //todo buildSystem method
         * @return ISystem
        public function createSystem(){
            return System::getInstance();
         * @return array of Users
        public function createUsers(){
            return array(AbstractFactory::getInstance()->createUser(1));
         * @param int $id
         * @return IUser
        public function createUser($id)
            return new User($id);
         * @param IUser $user
         * @return IDVR
        public function createDvr(IUser $user)
            $dvr = new Dvr($user);
            $cams = $this->createCams($dvr);    //new+add
            foreach($cams as $cam){
                /** @var $cam Cam */
            $daemons = $this->createDaemons($dvr);
            foreach($daemons as $daemon){
                /** @var $daemon Daemon */
            return $dvr;
         * @param DVR $dvr
         * @return array of Cams
        abstract protected function createCams(DVR $dvr);
         * @param DVR $dvr
         * @return array of Daemons
        abstract protected function createDaemons(DVR $dvr);
         * @param IDVR $dvr
         * @param ICamSettings $cs
         * @return ICam
        public function createCam(IDVR $dvr, ICamSettings $cs)
            return new Cam($dvr, $cs);
         * @param $from
         * @param $to
         * @return MoveToNfsCommand
        public function createMoveToNfsCommand($from, $to){
            return new MoveToNfsCommand($from, $to);
         * @param ICam $cam
         * @return ICamStream
        abstract public function createStream(ICam $cam);

    This allows us to hang absolutely any streams on the camera and do with them what comes to our mind. And it allows you to at least somehow control our zoo. (a prototype he is)

    Since we have a workhorse this is VLC, it starts to customize VLC for our system.
    Create an abstract class VlcStream extends Stream, class LiveVlcStream extends VlcStream.
    VlcStream is “friends” with Vlm ($ this-> vlm = new HttpVlm ($ this-> getVlcName (), 'localhost', HTSTART + $ this-> cam-> getDVR () -> getID ());)
    And it allows you to process start, stop
    abstract class VlcReStream extends VlcStream
    VlcReStream - takes LiveVlcStream as an input
    class RecVlcStream extends VlcReStream - records, “moves” video from Ram to Nfs

    And the Vlc extends Daemon class is created, it will be contained in the entity DVR.

    DVR handles start and stop
    public function start()

    starting the camera starts all the streams associated with the camera.
    Those. We run unix processes Vlc, motion, do VLM through HTTP.
    The stop occurs in the reverse order.
    Instead of VLC, you can use any thing that you like.

    But we have video surveillance in the cloud. So you need Sql and similar storage systems.
    The working system is as follows:

            $e = new BBLogMotionEvent(Motion::EVENT_MOTION_STOP);
            $e = new BBLogMotionEvent(Motion::EVENT_MOTION_DETECTED);
            $e = new BBLogMotionEvent(Motion::EVENT_CAMERA_LOSS);
            $recMotionEvent = new BBRecMotionEvent(Motion::EVENT_MOTION_START);
            $recMotionEvent = new BBRecMotionEvent(Motion::EVENT_MOTION_STOP);
            //удалить записи старше 30 дней при каждом update
            //$system->addPermanentCommand(new RotateRecCommand());
            $this->addPermanentCommand(new RotateRecCommand());
            return $system;
         * @param DVR $dvr
         * @return array of Daemons
        protected function createDaemons(DVR $dvr)
            $vlc = new Vlc($dvr);
            $this->addPermanentCommand(new BBDaemonWatchdog($vlc));
            // нам необходимы только те камеры, в которых включен mtn
            $cams = $dvr->getCamIDs();
            $ids = array();
            foreach($cams as $id){
                $cam = $dvr->getCam($id);
                $cs = $cam->getSettings();
                /** @var $cs BBCamSettings  */
                if($cs->mtn) $ids[] = $id;
            $motion = new Motion($dvr, $ids);
                $this->addPermanentCommand(new BBDaemonWatchdog($motion));
            return array($vlc, $motion);
         * @return array
        public function createUsers()
            $users = array();
            $db = \Database::getInstance();
            $q = "select id from users where banned=0";
            $r = $db->query($q);
            while(($row = $r->fetch_row())){
                    $users[] = AbstractFactory::getInstance()->createUser($row[0]);
                catch(\Exception $e){
                    Log::getInstance()->put($e->getCode().' '.$e->getMessage()."\n".$e->getTraceAsString()."\n", __CLASS__, Log::ERROR);
            return $users;
         * @param DVR $dvr
         * @return array
        protected function createCams(DVR $dvr){
            $db = \Database::getInstance();
            $q = mysql::getQuery(mysql::CAM_SETTINGS, array('{dvr_id}' => $dvr->getID()));
            $r = $db->query($q);
            $cams = array();
            while(($row = $r->fetch_object('system2\BBCamSettings')) != null){
                /** @var BBCamSettings $row */
                //$dvr->addCam(new BBCam($this, $row));
                $cam = $this->createCam($dvr, $row);
                $cams[] = $cam;
                //Так как у нас Motion создает раз в минуту картинку, то создаем таймлапсы
                $timelapse = new BBArchiveTimelapseCommand($cam);
            return $cams;
         * @param ICam $cam
         * @return ICamStream
        public function createStream(ICam $cam)
            $stream = new Streams($cam);
            $cs = $cam->getSettings();
            /** @var $cs BBCamSettings */
            $motion = new MotionStream($cam, $cs);
            $motion->setEnabled($cs->live && $cs->mtn);
            $live = new BBLiveStream($cam);
            $live->setTestInputCommand(new BBTestInputFailSaveCommand($cam, $live));
            $hls = new HLSVlcStream($cam, $live);
            //$this->streams[] = new FlvVlcReStream($this, $live);
            //nginx rtmp stream
            //$this->streams[] = new RtmpVlcReStream($this, $live);
            $rec = new BBRecStream($cam, $live);
            $rec->setEnabled($cs->live && $cs->rec);
            $rec->setTestInputCommand(new BBTestInputFailSaveCommand($cam, $rec));
            $mtn = new BBRecStream($cam, $live, TIME_LOCK_RECORD, Path::MOTION);
            $mtn->setEnabled($cs->live && $cs->mtn && BBRecMotionEvent::isMotion($cam));
            $mtn->setTestInputCommand(new BBTestInputFailSaveCommand($cam, $mtn));
            $stream->addStream($mtn, Path::MOTION);
            //motion flv stream
            $flv = new UrlFlvVlcStream($cam, "http://localhost:".(MOTION_STREAM_PORT + $cam->getID()));
            return $stream;

    createSystem, add the events we are interested in. The System class is responsible for their processing.
    are just what is called from the moton config
    # Command to be executed when an event starts. (default: none)
    # An event starts at first motion detected after a period of no motion defined by event_gap
    on_event_start php ~/dvr/bin/system2/main.php event {user_id} {cam_id} motion_start 0 "%Y-%m-%d %T"
    # Command to be executed when an event ends after a period of no motion
    # (default: none). The period of no motion is defined by option event_gap.
    on_event_end php ~/dvr/bin/system2/main.php event {user_id} {cam_id} motion_stop 0 "%Y-%m-%d %T"
    # Command to be executed when a picture (.ppm|.jpg) is saved (default: none)
    # To give the filename as an argument to a command append it with %f
    ; on_picture_save value
    # Command to be executed when a motion frame is detected (default: none)
    ; on_motion_detected php ~/dvr/bin/system2/main.php event {user_id} {cam_id} motion_detected 0 "%Y-%m-%d %T"
    # Command to be executed when motion in a predefined area is detected
    # Check option 'area_detect'.   (default: none)
    ; on_area_detected value
    # Command to be executed when a movie file (.mpg|.avi) is created. (default: none)
    # To give the filename as an argument to a command append it with %f
    ; on_movie_start value
    # Command to be executed when a movie file (.mpg|.avi) is closed. (default: none)
    # To give the filename as an argument to a command append it with %f
    ; on_movie_end value
    # Command to be executed when a camera can't be opened or if it is lost
    # NOTE: There is situations when motion don't detect a lost camera!
    # It depends on the driver, some drivers dosn't detect a lost camera at all
    # Some hangs the motion thread. Some even hangs the PC! (default: none)
    on_camera_lost php ~/dvr/bin/system2/main.php event {user_id} {cam_id} motion_camera_lost 0 "%Y-%m-%d %T"

    there is a call to main.php event uid cid event_name parameters
    System class it processes.
    By cron once every 10 minutes (once in any number of mines), System-update is called.
    According to update, update of all subsystems and

    Commands are what subsystems thrown over, and persistent commands, such as RotateRecCommand, are responsible for deleting old records.

    When creating daemons, there is an addition, for example, BBDaemonWatchdog, it monitors if the daemon has fallen.

    Users are already created from the database.
    Then the cameras from the database. A permanent time-lapse command is added. Which, in turn, works only once in TIME_LOCK_TIMELAPSE (8 hours) and pops information in Mysql.

    The last - the most interesting thing is that streams
    create a stream of motion, live, hls (HTTP Live Stream), rec, mtn, flv.
    live - normal http
    rec, mtn - streams for writing
    hls and flv for output to the browser.


    There is logic. Need "physics."
    There are many cameras, many threads, one processor.
    All recording is done in ramdisk, hls in ramdisk, vlc configuration is done so that there is no transcoding. A clean stream was taken away, a clean stream was kept. Moving from the ram disk to the storage (in the prototype was NFS) went through ffmpeg codec copy.
    This allowed us to display entries in the browser via html5.
    3 types of records. Full recording with slicing every 10 minutes, motion recording and timelaps recording in 8 hours where 3 minutes are displayed per second.

    The construction of the system was planned independent placement of DVR (recording machines), even geographically with minimal code changes.

    What was implemented
    1) all the records we want
    2) output to web, hls, flash, records in html5, because of the zoo codecs h264 live in h264 until I went to the web, output to mjpeg (allows you to watch online video on old nokia)
    3) web interface of this terrible thing with https , nginx security url and the like.
    We stopped on adding cameras via Onvif.

    Why the project stalled.
    1) poor sales (lack of a salesperson and a sales system)
    2) lack of financing (at least an admin needs such a system)
    3) lack of free time
    4) completely free development
    5) not a desire to get cameras for a test
    6) not a desire to put free cameras on test, although with the district council was an agreement.
    7) I change my permanent job. There will be no opportunity to do this.
    8) Purchase of design and purchase of slices, but in the future they chopped on their own.
    9) The absence of a person who would constantly monitor the system (found bugs both in motion and in vlc, but they are solved by the administration of the system)
    10) "Sales" for the year did not come up with any sales and tariff systems.
    11) I did a lot more than parallel to the main work, the creation of video surveillance and personal life.
    12) Everyone screwed a bolt on their negotiated responsibilities :)

    But we had 3 clients, they liked it. They didn’t take money for the system, they took money only for installing cameras.

    What it looked like :)

    If the topic goes well, then I will describe in more detail about VLC and pitfalls. And about the "mythical" piece of iron and experiments with it, too.

    Also popular now: