We design storage systems for video surveillance



    In the previous article, we talked about the features of organizing the IT infrastructure and the data storage subsystem for video surveillance, and came to the conclusion that it is advisable to use storage systems (data storage systems) for organizing video archive storage.

    In this publication, we will discuss the problem of choosing storage for video surveillance, the advantages of RAIDIX-based storage and examples of their real implementations.

    The choice of storage for video surveillance


    Varieties of storage


    Storage systems can be used for a wide range of data storage tasks: file storages, video surveillance, business applications, databases, virtualization, archives and backups, etc.

    For slow tasks with large volumes, usually choose less productive solutions with large arrays on the HDD (file and backup storage). For virtualization and general-purpose tasks, hybrid solutions (flash + HDD) are used to meet the demand for both speed and volume. For delay-sensitive applications and databases, very fast and expensive storage systems are chosen - as a rule, today it is all-flash, where speed comes to the fore, the volumes there are generally small.

    For video surveillance tasks, of these three categories, they mainly choose the first volume-oriented one to get a low price per terabyte. They can also choose more expensive hybrid solutions, but there will be little sense from them, the flash-cache (or dash) will clog from streaming (sequential) storage traffic and everything will run into slow HDDs. Fast random-access all-flash storage in sequential operations, of course, will also be very good, but for video surveillance that requires large volumes, this is an unreasonably expensive and inefficient solution.

    Features I / O for video surveillance


    In the context of data storage, video surveillance is characterized by multithreaded sequential I / O operations:

    • Each IP camera creates its own serial data stream for recording in the video archive storage. As a result, a serious multi-threaded recording load is formed.
    • When viewing video from the archive, sequential read operations are performed, consisting of one or more streams.

    From the point of view of the load on the storage, it is optimal when the project does not involve the constant viewing of video from the archive by many operators from several workstations. Real-time monitoring does not require reading data, and viewing from the archive takes place on one video recording - occasionally or even constantly. In this case, the main load - 90% or more - will be recorded, the storage system will not be subjected to a noticeable load on reading.

    Otherwise, when constant parallel viewing of multiple videos from different cameras from the archive (for example, a casino, stadium, shopping center) is required, we will receive many competitive reading streams and the load on the storage system will increase dramatically. Therefore, the same number of cameras will have to put a steeper storage, more productive at least twice, or reduce the number of cameras per array (controller).

    Problem of choice


    Today on the market there are many models of storage systems from various manufacturers:

    • There are vendors who are engaged only in data storage, and storage systems are their main or only products.
    • There are giants of the IT industry who make all kinds of hardware and software, while supporting and developing their own lines of storage and storage solutions.
    • Many large manufacturers of software and hardware for video surveillance offer their own storage.

    Conventionally, all storage models can be divided into two classes:

    • Brand Store. In other words, storage from Class A manufacturers or the premium segment. They are cool, reliable, they have good service. Their price tag is quite expensive, even on an entry-level model, and serious productive models are like an airplane. The most annoying thing is that it’s often impossible to install arbitrary screws (HDD) in them, they’ll be plugged into them normally, they will work and will not be deprived of the hardware guarantee only the “native” drives of the storage vendor. In fact, these will be the same server screws from manufacturers that produce hard drives, but with branded sleds and a branded sticker from the storage manufacturer, but they will cost several times more. Actually, with the screws for branded servers, the situation is the same. For video surveillance projects, where people are used to counting money and do not want to overpay at times for the brand, these solutions are not suitable. Of course,

    • Budget decisions . This is a ready-made storage from "popular" storage vendors or prefabricated solutions on inexpensive standard (commodity) server hardware and software storage systems (SDS), paid or free. This includes products from video surveillance vendors. The prices are pleasant, you can set any screws, but there may be problems with reliability, performance and service. This solution is suitable for small and non-critical projects in a limited budget. For serious systems where guaranteed reliability, performance, fault tolerance and service are needed, such solutions are unacceptable, this must be understood.

    Ideally, for a serious video surveillance project, I would like to get a storage solution that would combine only the strengths of both categories of storage systems on the market - proper quality for a reasonable price. In this segment, such a solution are products based on RAIDIX. Indiscreet, but perhaps this is a fact. Below we reveal the truth of this statement.

    RAIDIX Storage Features


    Architecture


    RAIDIX - software storage or SDS (Software Defined Storage), which allows you to build reliable, efficient and fault-tolerant data storages based on standard server hardware. Basically, any x64 servers are suitable for this, including:

    • 1-2 Intel Xeon processors of the appropriate model and the required amount of RAM;
    • one or more SAS HBA adapters for connecting internal and / or external disk baskets; hardware RAID controllers are not needed and are even contraindicated;
    • one or more interfaces for synchronizing the cache in a dual-controller configuration; There are several options: SAS, InfiniBand, Fiber Channel (FC), Ethernet; duplication of interfaces is possible; in a single-controller version they are not needed;
    • Interfaces for connecting to a SAN and / or NAS: Ethernet, InfiniBand, FC; direct connection to hosts (clients) via SAS is possible;
    • SAS / SATA disks (HDD) large (3.5 ") or small (2.5") form factor; any suitable model from any manufacturer, without restrictions; SSDs are also supported, but in the context of video surveillance they are not interesting;
    • A server platform suitable for installing the above equipment.

    To connect a large number of disks, it is supposed to use external disk shelves (baskets) connected via SAS. It is recommended that you use internal and external drive bins that support hot-swappable drives.

    Of course, there is a compatibility sheet, a list of recommended and tested equipment.

    Deployment options


    RAIDIX offers two deployment options: single and dual controller. In the first version, RAIDIX software is installed on one physical server acting as a storage controller. Disks are combined into a fault-tolerant RAID array, but the server itself and some of its components form single points of failure. This may be acceptable for non-critical tasks.

    The dual-controller configuration involves installing RAIDIX software on two identical physical servers, each of which becomes a storage controller. It can be separate server platforms, or a single platform with two server nodes (nodes, blades). Both controllers are physically connected to a single disk pool located on internal and external disk baskets. RAIDIX combines the two servers into a fault-tolerant active-active cluster, the controller cache is synchronized via dedicated interfaces.

    In normal mode, the load is evenly distributed between two controllers - half of the volumes created on the disk array are served by one controller, the other half - to the second controllers. If for some reason one of the controller nodes fails, the entire load will automatically switch to the "survivor" controller without interruption or data loss. This solution eliminates the presence of single points of failure and is suitable for critical projects that are sensitive to downtime. AIC HA401-LB2

    can be a good example of a server platform for dual-controller RAIDIX storage configuration. This is a 4U platform for highly available storage servers (cluster-in-a-box) with two identical server nodes, redundant power supplies and an internal disk basket on a 24 HDD 3.5 ”hot-swappable. Each server node supports two Xeon processors, up to 2TB of RAM and up to 6 PCIe expansion slots. This is enough to deploy a very productive and capacious storage system to several hundred disks. This platform can be called one of the recommended, it is successfully used in many projects based on RAIDIX.

    This approach makes it possible to create the optimal storage for any project, with the right volumes and performance. Volumes can vary from 12 (or less) to hundreds of disks per storage. At the same time, the required performance is determined by the selected Intel Xeon processors, the amount of RAM, the throughput of interfaces and adapters - you can lay them with a margin for the possibility of increasing disk capacity. We choose what we need, and nothing more - the maximum flexibility and no overpayments.

    Key features


    Today RAIDIX is a universal solution, suitable for file storages, and for virtualization, and for other tasks, it copes with everything quite successfully. However, the product was originally developed for media storage - video post-production, in particular in the film industry.

    For these guys, the decisive factors were:

    • Fast sequential (streaming) operations for working with heavy uncompressed video. Streams are large, they must be managed to record without loss and read without brakes.
    • Large usable capacity. The volumes are large, standard solutions on RAID-10 are too expensive, you need solutions with checksums a la RAID-5/6.
    • No performance drawdown during disk failure when the array is in a degraded state. Standard RAID arrays in the normal state, when all disks are alive, can give the desired performance, but when it degrades, it drops dramatically, this can lead to frame loss, terrible brakes and inability to work.
    • High fault tolerance (availability). The absence of single points of failure, the guarantee of data safety with the simultaneous failure of at least two disks in the array. The price of data loss, which was taken and mounted for a very long time and expensive, is too high; losing this data is unacceptable.

    It was possible to satisfy these requirements only with the help of branded Hi-End storage systems that cost “blue” money. Therefore, I had to look for an alternative, and a product was developed that later grew into a standalone solution, became RAIDIX, and now a lot of films are mounted on it at studios around the world. General storage tasks on RAIDIX are also successfully resolved. At the same time, his cost is quite acceptable, one might say, popular.

    RAIDIX for video surveillance


    Main advantages


    It is easy to draw parallels between media tasks and video surveillance and come to the conclusion that they have the same storage infrastructure requirements: everything that is relevant for media storage is also important for a video archive. Therefore, RAIDIX-based storage systems are also optimal for video surveillance projects.

    Processing input / output to the disk subsystem and calculating checksums to ensure redundancy in RAIDIX is carried out using the computing resources of Intel Xeon processors, SAS HBA adapters without RAID functions are used to connect disks. Using proprietary RAIDIX algorithms through the use of internal CPU instructions, they perform these tasks much faster and more efficiently than hardware RAID controllers and ASIC devices. As a result, we get the capabilities necessary for a large video surveillance infrastructure:

    • Fast sequential (streaming) operations for processing data from a large number of IP cameras. For example, hundreds and thousands of cameras with FullHD resolution and higher, 25 fps, medium or high complexity of the scene.
    • Large useful capacity due to the ability to work effectively with large RAID groups of 12-24 disks, providing simultaneous failure processing for up to two or more disks. In this case, disks of large volume - 6-10TB can be successfully used.
    • The lack of performance drawdowns during crashes due to the power of Xeon processors and an efficient approach to using their instructions. Data from failed screws is very quickly and invisibly for end-users of storage resources calculated from checksums.
    • Support for a dual-controller storage configuration, eliminating the presence of single points of failure.

    Additional features


    In addition, in the arsenal of RAIDIX, there are the following “buns” unique and useful for the field of video surveillance:

    • In addition to standard RAID levels 10, 5, 6, RAID 7.3 and N + M are supported, which provide failover of up to three (RAID-7.3) and more (RAID-N + M, N disks - usable volume, M disks - can fly out) disks at once.
    • The ability to recover hidden data corruption on disks - protection against silent data corruption.
    • Proactive Reconstruction. SHD monitors the performance of all drives. In case of malfunctions, individual disks may continue to work, but so slow that recovery of their data from checksums is faster than reading from these disks. In this case, the record will remain unchanged, but instead of reading from problem disks, the data will be computed from the checksums of the array for optimal performance.
    • Partial reconstruction. When the drive is temporarily disconnected after it is reconnected, only the missing part of the data will be restored. This can be useful when disconnecting disks by mistake, losing a disk connection, moving disks between shelves, disconnecting entire shelves for the duration of the service (with the corresponding number of shelves and distributing RAID groups among them).

    You can verify the truth of the capabilities stated above by deploying RAIDIX on a test bench. To do this, you need to request a demo from us and conduct testing. Of course, this assumes that the hardware necessary for installing RAIDIX software is available.

    Existing RAIDIX Projects


    For greater persuasiveness, I propose moving from theory to practice. In addition to many installations in the media sector and solving general-purpose storage tasks, storage infrastructures have been successfully built on the basis of RAIDIX in several large video surveillance projects.

    At the moment, at the implementation stage, there is a large-scale project in the territory of the Russian Federation on video surveillance on a city-wide scale, in which 4 dual-controller storage systems RAIDIX, for 200 HDDs of 6 TB each, were used to organize the storage. Another serious video surveillance project based on RAIDIX is being developed for a European casino: a dual-controller storage system with 300 HDDs of 10TB each. We can talk about them in subsequent publications, after the introduction of systems into commercial operation.

    Today, we have three major successful CCTV projects in South Korea. All three projects relate to road safety and fixing violations, carried out within the whole city. The largest of them is presented in the next section.

    2000 IP Camera Project in Korea


    Initial data


    Yongin is one of the fastest growing cities in South Korea, equal in area to Seoul, the population of the city is about a million people. To fix traffic violations and ensure security, the authorities installed 2,000 surveillance cameras in the city. The equipment for recording and storing video was located in 5 server racks and occupied an entire room. At the same time, cameras shot low-resolution video, which significantly complicated the work of law enforcement agencies, since it was not possible to distinguish a car number or a person in an image of such quality. The videos were kept for less than a month, and the police were limited in resources during the investigation of offenses - after this period, cases were closed, and fines for violations were not received in the budget.



    To solve this problem, the city administration planned to record and store video in HD / FullHD resolution. Camera recording mode - 24/7 (around the clock) with a frequency of 16-25 frames per second. The archive storage period is not less than a month. This approach significantly increases the speed of the video stream and the amount of data, therefore, it requires a significant increase in performance (throughput) and the capacity of the storage system storing the video archive.

    TK requirements for storage


    The initially installed storage equipment clearly could not cope with the task, so there was a need for its replacement or modernization. As a result, the following requirements were imposed on the new storage infrastructure:

    • fast sequential recording and reading, eliminating delays and frame loss while recording 2000 streams of HD / FullHD;
    • the absence of a drawdown in the performance of the disk array in case of failure of the number of disks allowed by the configuration (array degradation);
    • high reliability and fault tolerance, automatic failover of any element of the system;
    • optimal price / quality ratio of the solution;
    • simplicity and convenience of implementation and maintenance;
    • high-quality technical support;
    • high density equipment;
    • high usable storage capacity sufficient to store the video archive from all cameras for at least one month in depth;
    • the ability to scale.

    Solution structure


    As potential solutions, various options for brand and budget repositories were considered. The former turned out to be too expensive, the latter did not meet the stated requirements. The optimal solution among all the options considered, satisfying all the requirements of the project, was RAIDIX-based storage. The composition and structure of the implemented repository and the project as a whole are presented below.

    In addition to storage systems, the IT infrastructure of the project included:

    • 15 physical rack servers based on Intel platform running Windows Server for deploying VMS-servers (video surveillance software);
    • 2 InfiniBand (IB) Mellanox switches with 40 Gbps port bandwidth for connecting VMS servers to storage.

    To deploy storage, a 4U rack-mount AIC HA401-CP2 server platform was used with two server nodes and an internal basket for 24 3.5-inch disks - an analogue of the newer AIC HA401-LB2 platform. This platform is ideal for deploying a dual-controller RAIDIX fault-tolerant configuration, since each computing (server) node is a separate server with its motherboard, processors, RAM, network interfaces, and HBA adapters (for connecting disk baskets). Power supplies are duplicated and are common to all nodes. Thus, due to the duplication of all necessary components, single points of failure of the platform as a whole are eliminated.

    Below is a block diagram of the solution.



    Accessories


    The hardware configuration of server nodes - storage controllers is identical, each controller includes:

    • One Intel Xeon processor E5-2609 (4 cores at 2.4 GHz). Not the freshest and most powerful “percent”, but it was enough.
    • 128GB RAM - many disks and RAID groups require a large amount of RAM.
    • 2 dual-port Mellanox ConnectX-3 InfiniBand adapters, 56Gb / s transfer rate per port. 2 ports for connecting to IB switches, 2 ports for synchronizing the controller cache - 2 crossover connections between server nodes.
    • One Broadcom 9207-4i4e SAS HBA adapter for connecting to 1 external (60-drive JBOD) and 1 internal drive cage (24 drives). 4 external (4e) and 4 internal SAS lines of 6 Gb / s, 1 external Mini-SAS port.
    • One Broadcom 9207-8e HBA adapter for connecting three external disk shelves (3 JBODs for 60 disks, 2 cascaded shelves). 8 external (8e) SAS lines of 6 Gb / s, 2 external Mini-SAS ports.

    In the project, 208 enterprise-class SAS HDD 3.5 '' 4TB 7200 rpm hard drives were used to achieve the capacity needed to store the video archive. In addition to the internal disk basket of the storage platform (24 HDDs), 4 external disk shelves (JBOD) AIC XJ3000-4603S were used to accommodate such a number of disks:

    • 4U form factor;
    • capacity - 60 HDD 3,5 `` with vertical installation and hot swapping;
    • 2 SAS expanders 6Gb / s, 4 Mini-SAS ports for expander;
    • duplication of power supplies.

    All disk baskets of the solution (internal and external) supported hot swapping of disks. The internal and two external baskets were full of disks (24 + 2x60 = 144HDD), according to the structural diagram, each of them is independently connected to one port of the HBA adapter on each controller. The remaining 64 disks were equally divided between the remaining two external shelves (32 disks per shelf), as shown in the diagram, these shelves are connected in cascade (sequentially) - the first shelf to the HBA adapters of the controllers, the second - in two ways to the first.

    Expandability


    The solution supports further vertical scaling. Since the two disk baskets are not full, it is possible to install up to 56 additional disks in them. In this case, it is better to put on the controllers with an additional HBA adapter (Broadcom 9207-8e) and connect the cascaded disk shelf to them directly. On the remaining free ports of the new adapters (while occupying one of the two Mini-SAS ports), you can hang another shelf on 60 disks.

    If there is a lack of computing power, you should add one processor per controller and increase the amount of RAM. Thus, it is possible to increase storage capacity and performance by 1.5 times with minimal investment - the existing platform is expanding vertically without the need to buy new servers.

    Array configuration


    When configuring the disk array, a simple and logical approach was chosen. Each of the 15 VMS servers was assigned one volume (LUN), each of which was located on a separate RAID group of 12-15 disks. To create each disk group, the level of RAID 7.3 was selected, which ensures data integrity while simultaneously failing up to three of any disks.

    As a result, the total useful storage capacity was about 600 TB.

    Placement density


    All equipment necessary for organizing storage was placed in one rack, as shown in the figure below.



    It should be noted that at the same time, a significant part of the rack space remained empty, the usable space occupied by the storage hardware is 20U (5 nodes each 4U), and the density is quite high.

    Conclusion


    This project was implemented in 2014, at that time the solution was quite modern and technological. The system has been successfully operating for the fourth year and is coping with its tasks; no modernization or expansion has been carried out and is not yet required.

    Today, the equipment used in this project is outdated: new generations and models of processors, HBA adapters, disk volumes have grown to 8-10TB. Therefore, the capacity of the project can be achieved and exceeded on a new hardware with a placement density of 8U instead of 20U.

    Thus, on modern hardware and software RAIDIX storage systems, it is possible to successfully build efficient data warehouses for large video surveillance systems that can provide high density and volumes of up to several petabytes per storage system, while ensuring high performance and fault tolerance for a reasonable price.

    Also popular now: