Intel Distribution for Apache Hadoop: Make Big Data Comfort
There is probably no reason for hawkers interested in "big data" to talk about the Apache Hadoop project. Recently, this framework has become especially popular and often mentioned on Habré. So, just recently, Mail.ru experts talked about the migration of their search engine to Hadoop.
And how to make a good software system even better? For example, add hardware components to it. This is exactly the way Intel went with releasing Intel Distribution for Apache Hadoop - “localizing” Hadoop under the Intel Xeon platform, equipped with many useful features to facilitate its management and operation.
In addition to the main Apache Hadoop applications, as shown in the figure, the Intel package includes an additional Intel Manager for Hadoop Software component that facilitates the deployment, configuration, and monitoring of new applications. In addition, the Intel Active Tuner for Apache Hadoop module automatically configures the system performance metrics. As an additional component, the Intel Graph Builder for Apache Hadoop library for graphical reports is available.
Optimization of the framework for the Intel Xeon platform has achieved two important results:
- Implementation of data encryption without slowing down disk operations in the HDFS file system due to hardware implementation of the AES encryption algorithm in Intel Xeon processors;
- Significant increase in overall system performance due to optimal interaction between Hadoop software and Xeon I / O subsystems. According to the results of internal testing by Intel, the developers of the company managed to achieve fivefold overclocking of the system compared to the non-optimized version.
Intel is developing several projects related to cloud technologies and storage networks. So, relatively recently, the Intel Intelligent Systems Framework was released - a framework of systems for collecting, analyzing and transmitting information from end devices to data centers. But another project is only at the very beginning of the journey. At the end of February, Intel and the Russian company Mirantis, a vendor of the OpenStack platform, signed a cooperation agreement in the field of software development. This means that in the future we will also see optimization of OpenStack for Intel processors, which, as in the case of Hadoop, will certainly provide a significant gain.