Archives and the spiral of history
The rapid growth in the amount of information generated around the world is spoken on every corner. Usually this is remembered when it comes to network infrastructure, client content, search technologies and many other things. The same situation is observed in the corporate segment: in most organizations, the volume of stored information is growing many times. According to a Forrester Research report, about 85% of the data in corporate systems is static content that will never change. Various requirements of the authorities and regulators oblige organizations to store various information for several years (for example, information on all customers and completed transactions, etc.). As a result, the business has to spend significant funds on ensuring the storage of this information, investing in a server fleet, storage, software acquisition, etc.Another consequence of the growth in the volume of stored data was the desire of many companies to be able to analyze and search the entire available volume of accumulated information. From a certain moment, such a task turns into a Big Data processing task. As a result, there is a need to find solutions that are more suitable for storing and working with such arrays of information. Therefore, many are looking for more profitable solutions for storing such arrays of information and working with them.
An interesting example is Nokia, which recently sold its mobile division to Microsoft. According to the terms of the contract, the Finns had to transfer the entire information archive of the unit to the new owner. Given the large amount of data, Nokia approached this creatively: a compact archive storage system was acquired, into the database of which all the necessary information was uploaded, and then the whole system was simply sent to Microsoft.
Speaking about the growth in the volume of stored information, it is necessary to mention the accumulation of data from outdated applications. With the modernization of information systems, working environments change, new software packages are introduced, and the database structure is being rebuilt. As a result, a large array of information is stored, stored in an already unused form. But, since it is often required to ensure the availability of this data, additional funds are spent over the years to maintain the already outdated hardware and software.
About myths
Today, many tend to view archive systems as obsolete or irrelevant technology. For example, it is believed that backups successfully replace archiving. In fact, these are not at all interchangeable concepts. Unlike backup, the archive is designed to save information without its excessive duplication, allows you to structure and index data, provides access to it with the ability to search with optional encryption and application of different policies. In addition, the transfer of static data to the archive allows you to reduce the load on applications and cost less server clusters and storage.
It is also widely believed that the archive is a kind of messy pile of information that reflects the history of the company and is unnecessary for solving current and future business problems. However, above we have already mentioned such a trend as the analysis of an array of data on the company’s work accumulated during its existence. According to Gartner's forecast , by 2017, about 75% of organizations will use their own archive as the initial source of information. Today, such organizations are about 10%.
The following prejudice regarding archiving is related to the desire to hide some uncomfortable information from the regulator. Indeed, in the archive it is much easier to find something. However, this situation has a downside: the amount of fines for failure to provide the information requested by the regulator may amount tomillions of dollars . And this is many times more expenses for creating an archive.
Speaking of expenses. There is an opinion that archiving is an expensive pleasure. However, in fact, archival systems provide significant savings. This is due to the use of cheaper media, with a decrease in the cost of support and an increase in the productivity of the main working systems. It is also necessary to recall that in 2014 it became a record in the number of information leaks, and the volume of stolen data increased by 78% compared to 2013 . Reputation and legal costs can also be much more expensive than using an archive system with data encryption.
Finally, another argument against the creation of archives is the opinion that the ECM platforms provide the same functionality. But there are a number of differences. Firstly, the archive is designed to work simultaneously with structured and unstructured data. It is optimized for storing billions of records and documents. Secondly, as noted above, storing data in an archive is cheaper due to the transfer to cheaper media, as well as reducing the size of the backup and freeing up the resources of the working system.
Archive of a modern sample
The modern archival system allows you to solve five main problems:
- Saving data for future use.
- Ensuring continuous user access to stored data.
- Ensuring confidentiality of access.
- Reducing the load on working systems by transferring static data to the archive.
- Use of data storage policies.
Another important property of the archive is the storage of structured and unstructured information in a single database. Naturally, the database should be deployed on a separate horizontally scalable storage system so that the archive can be painlessly expanded as data volumes increase.
As a similar solution, EMC InfoArchive can be used . This is a comprehensive product, which is a combination of "SHD + software platform for archiving and encryption." InfoArchive will also be useful when you need to store inherited data from heterogeneous systems and in different formats, as well as for tasks of analyzing data lakes. By “data lake” is meant a repository with a very large amount of raw data in the original formats, without any hierarchical structure.
Depending on specific conditions (the amount of structured and unstructured data; the presence and composition of legacy systems supported by the company; the need to use analytical tools; creating cloud services, etc.) InfoArchive can be built on the basis of EMC Isilon , DataDomain , Atmos or Centera . On the selected storage system, the EMC Documentum Dynamic Delivery Services (DDS) database is deployed , based on xDB and using a number of international standards, including the open XML standards and OAIS (Open Archival Information System).
A feature of InfoArchive is that all data must be transferred to the system either in the form of SIP information packets, according to the OAIS standard, or in the form of simple XML structures, if the customer does not require compliance with the OAIS standard. Also, all information stored in InfoArchive can be accessed through JDBC for later use / recovery in the original application.
Data in relational databases is presented in the form of linked tables. When the user requests some information, the application sends out queries according to the tables, aggregates the received answers and provides it to the user.
XML files are used to store, organize and transfer structured information and meta-data of unstructured information. This allows you to create an archive that combines data from disparate applications. InfoArchive implements the ability to search all stored data and use storage policies, provide on-the-fly encryption and manage access to certain data and their sets. Regardless of the size of the archive, only one DBMS is used.
System performance depends on the number and configuration of storage, as well as on the configuration of the platform itself. For example, for some clients, InfoArchive's performance when receiving structured data reaches 2 million records / hour (up to 60 Gb / hour). The system is capable of processing up to 15,000 search queries per hour, the search for one document on average takes 0.5 seconds., Records - 2.5 seconds.
To protect data, access policies and encryption using EMC RSA KeyManager are used. InfoArchive can also be integrated with other encryption systems.
Conclusion
Today, archival systems are primarily implemented in those companies where the problems of increasing data volumes are most acute, which must be stored and made available at the request of regulators. First of all, this is the financial sector, the telecommunications industry, utilities and the public sector. And as the practice of our company shows, medium-sized companies are actively showing interest in archives, actively trying to strengthen their market positions. Visual evidence of the advent of the information era.
