March 6, 2018 at 11:02Overview of the InfiniBox F2230
Recently the company "ONLANTA" purchased for your enterprise cloud Oncloud.ru InfiniBox F2230 companies Infinidat system. There are practically no reviews of this system on the Internet, so we decided to eliminate the gap. Today we will try to tell you as much as possible about this storage system.
Infinidat is a young Israeli company. It was founded in 2011 by Moshe Yanay, who at one time led the development of EMC Symmetrix, and later founded the company XIV and was the chief architect of this system. By the way, XIV in 2008 was acquired by IBM. So, Infinidat’s experience in the production of Enterprise-level storage systems should not be taken. One of the key factors of Infinidat systems is their high reliability (99.99999%). Infinidat is a leader among disk array visionaries, according to Gartner analysts.
The F2230 series system takes up only 18U. At the same time, the peak power consumption is only 3.5 kW, which, with a useful capacity of 250 TB, is quite economical. This system fully supports compression, which can increase the effective volume up to 1PB.
The main development of the company is an operating system that allows you to manage equipment in a quality manner. In essence, this is Software Defined Storage (SDS). These systems, which are now actively fighting for the market with their "classic" competitors. Infinidat is delivered as a complete system with all hardware components and its own rack.
Unload and unpack the
heart of the system - three servers operating in Active - Active - Active mode.
The use of three controllers allows you to reduce performance losses in the event of failure of one of the controllers. In the classical two-single-core configuration, in the event of a failure of one controller, the system is not able to ensure the safety of the write cache. This leads to its forced reset, which in turn leads to an increase in delays. In the case of a three-wire configuration, when one of the controllers fails, the cache continues to synchronize between the remaining two nodes.
Controllers and BBU
The total RAM of all controllers is 384GB. Our system also has 24TB flash cache and two disk shelves with 60 NL-SAS disks.
We install disks in shelves
Disk shelves front view
Disk shelves, rear view
Each of the nodes in the system is equipped with several SSD drives, which act as a global cache for all nodes (the data block is not cached twice). The address space of the volume beats into pieces, the pieces are distributed among the controllers, each controller maintains its pieces and caches them. When a request comes to the controller not for its piece, it takes it via InfiniBand from another controller.
The connection between nodes is carried out using InfiniBand. Disk shelves are connected to the controllers via SAS. Access to data can be arranged both via FC and via Ethernet.
InfiniRAID - The logical architecture of Infinidat uses small blocks (64k) of disks as RAID elements (instead of the entire disk). This minimizes disk rivalry and allows each node in the system to constantly use all the disks for maximum performance.
As the data arrives at the system, they are placed in RAM (for quick access), together with the DIF (Data Integrity Field) field, which remains with the data at all levels of storage and processing, to prevent logical errors and lost records. The DIF is read along with the data during each read operation and checked to detect all non-detectable errors (Silent-Data Corruption), which are then corrected using RAID.
When the data distribution algorithm decides to write data from memory to permanent media, it selects the blocks for writing and optimizes them before that. This includes organizing blocks into groups of 14 sections (together with DIF) + 2 parity blocks.
The algorithm distributes these groups (stripes) across all drives in the InfiniBox, while also checking that:
Such a distribution of data, where each virtual RAID group includes partitions from various disk groups, allows you to use all system disks at once for data recovery. Therefore, InfiniBox can recover large disks in minutes instead of days. Thanks to the sufficient space allocated for protection against disk failure (12 disks in the older model), and using the ability to take unallocated space for protection, the system can withstand 12 broken disks or more.
Most read operations come from the cache located on SSDs. This allows you to minimize the load on NL-SAS disks by accessing them for only a small amount of data that did not get into the cache.
Write operations fall into the DRAM-protected cache, which contains all changes to the blocks before they “cool down”, after which they are already sent to the disks. When data blocks are sent to disks, they are written to all disks simultaneously, which gives the system the ability to write them at high speed. Blocks of data that the system considers most likely to be read later are written not only to NL-SAS disks, but also to the SSD cache for faster reading.
This unique data layout allows InfiniBox to avoid using SAS disks and instead use a combination of SSD and NL-SAS disks to achieve high performance and low IO latencies.
The installed, connected and configured system is ready for operation.
It is worth noting that this is a system that does a large number of tasks for users: it does not give access to disks - RAIDs are already done in advance, load balancing across disks and between controllers is done automatically, the controller firmware is updated with support etc.
This icon in the upper right corner indicates that you have established a connection with the Infinidat server and support will be able to access the system for some kind of diagnostics, etc. When a support employee connects to your array, the entire icon will turn green. Access for support can be disabled, and enabled only when necessary.
I would like to start with the fact that these systems do not support deduplication. At all. Only compression, about which I will tell a little more.
To prevent data compression from affecting system performance, Infinibox compresses it when data is transferred from the cache to disks. Currently, the system uses the LZ4 compression algorithm, but the storage system architecture is built in such a way that in the future it will be possible to replace it with some newer and more efficient one. Do not forget that data compression is a processor-dependent operation and even when using modern processors it can affect the overall system performance. Compression is done in 64KB blocks, which allows to achieve a higher compression ratio.
Compression works with both thick and thin volumes, but its work in these cases is different.
So, how InfiniBox saves disk space without affecting performance: the data is in the RAM cache in an uncompressed form. Compression occurs when data is written to disk. But it is the record on the HDD, because what gets to the SSD cache is also there in uncompressed form. Infinidat assumes that most of the data you are working with is cached, so compression will not increase response time. In fact, we understand that this is far from always the case, but if you look at the general architecture, which involves storing data on a large number of disks in small blocks, this avoids a strong increase in response.
If you are interested in learning more about the operation of this system, I suggest referring to the article “ Storage Architecture for a Digital Enterprise””Vasily Kostryukov, Technical Director, Infinidat Russia.
InfiniBox supports both synchronous and asynchronous replication. Asynchronous replication is based on snapshots. Synchronous replication sends I / O to both systems (primary and backup) and this increases the response time, because the host receives confirmation of data recording only after recording to both systems. Reading occurs only from the local system.
Asynchronous replication
If, according to the system, the quality of communication between the systems will not satisfy the requirements of synchronous replication, which is already configured, it will automatically transfer it to asynchronous mode and return it to synchronous mode when restoring the proper connection quality.
Asynchronous replication is supported on:
Synchronous replication to:
Operations that are not supported on replicated entities:
In synchronous replication mode, changing the direction of replication is supported.
To create a link for asynchronous replication, you will need four ports on each system (three data ports and one management port), while synchronous replication will require seven ports (six data ports and one management port).
Infinidat Host PowerTools for VMware enables virtual environment administrators to independently manage disk allocation. This facilitates and saves the time spent on basic operations. Host PowerTools also supports VASA (vStorage APIs for Storage Awareness).
Literally on March 1, a plugin for integrating this backup system with Infinibox arrays appeared on the Veeam website , which was developed by Infinidat in collaboration with Veeam, which allows us to hope for the high quality of this product. This plugin allows you to use consistent storage snapshots to increase backup speed and reduce the impact of backups on a productive environment.
I will depart from the tradition of taking many screenshots of the interface and this time I made a video that was accompanied by the necessary comments.
InfiniBox systems perfectly integrate into any cloud infrastructure management system thanks to its API, which allows you to quickly and conveniently work with the system. It does not matter whether you provide services and the allocation of resources is done using the control panel or the system only serves your own infrastructure and you need to quickly and conveniently add a large number of elements. All this is perfectly implemented using the API. There is also an InfiniSDK for Python .
Add 1000 hosts? Not a problem!
The other day, the portal code.infinidat.com even appeared , on which you can find various scripts for integration with third-party systems (Openstack and Saltstack), modules for Ansible, and much more. So if you are faced with the task of integrating this array not just as part of the physical infrastructure, but convenient management, billing, etc., you will find many useful things on this portal.
Given that cloud Oncloud.ru company "ONLANTA" provides power for corporate customers under heavy loads (such as ERP, CRM, database, etc.), it has developed the architecture and develop their own capacity management tools, InfiniBox F2230 system perfectly Suitable for our infrastructure.
Infinidat is a young Israeli company. It was founded in 2011 by Moshe Yanay, who at one time led the development of EMC Symmetrix, and later founded the company XIV and was the chief architect of this system. By the way, XIV in 2008 was acquired by IBM. So, Infinidat’s experience in the production of Enterprise-level storage systems should not be taken. One of the key factors of Infinidat systems is their high reliability (99.99999%). Infinidat is a leader among disk array visionaries, according to Gartner analysts.
The F2230 series system takes up only 18U. At the same time, the peak power consumption is only 3.5 kW, which, with a useful capacity of 250 TB, is quite economical. This system fully supports compression, which can increase the effective volume up to 1PB.
The main development of the company is an operating system that allows you to manage equipment in a quality manner. In essence, this is Software Defined Storage (SDS). These systems, which are now actively fighting for the market with their "classic" competitors. Infinidat is delivered as a complete system with all hardware components and its own rack.
heart of the system - three servers operating in Active - Active - Active mode.
The use of three controllers allows you to reduce performance losses in the event of failure of one of the controllers. In the classical two-single-core configuration, in the event of a failure of one controller, the system is not able to ensure the safety of the write cache. This leads to its forced reset, which in turn leads to an increase in delays. In the case of a three-wire configuration, when one of the controllers fails, the cache continues to synchronize between the remaining two nodes.
The total RAM of all controllers is 384GB. Our system also has 24TB flash cache and two disk shelves with 60 NL-SAS disks.
Each of the nodes in the system is equipped with several SSD drives, which act as a global cache for all nodes (the data block is not cached twice). The address space of the volume beats into pieces, the pieces are distributed among the controllers, each controller maintains its pieces and caches them. When a request comes to the controller not for its piece, it takes it via InfiniBand from another controller.
The connection between nodes is carried out using InfiniBand. Disk shelves are connected to the controllers via SAS. Access to data can be arranged both via FC and via Ethernet.
InfiniRAID - The logical architecture of Infinidat uses small blocks (64k) of disks as RAID elements (instead of the entire disk). This minimizes disk rivalry and allows each node in the system to constantly use all the disks for maximum performance.
As the data arrives at the system, they are placed in RAM (for quick access), together with the DIF (Data Integrity Field) field, which remains with the data at all levels of storage and processing, to prevent logical errors and lost records. The DIF is read along with the data during each read operation and checked to detect all non-detectable errors (Silent-Data Corruption), which are then corrected using RAID.
When the data distribution algorithm decides to write data from memory to permanent media, it selects the blocks for writing and optimizes them before that. This includes organizing blocks into groups of 14 sections (together with DIF) + 2 parity blocks.
The algorithm distributes these groups (stripes) across all drives in the InfiniBox, while also checking that:
- each section on a different disk,
- sections are evenly distributed between the shelves (to protect against shelf problems),
- sections are evenly distributed between all disks (for optimal distribution of capacity and performance).
Such a distribution of data, where each virtual RAID group includes partitions from various disk groups, allows you to use all system disks at once for data recovery. Therefore, InfiniBox can recover large disks in minutes instead of days. Thanks to the sufficient space allocated for protection against disk failure (12 disks in the older model), and using the ability to take unallocated space for protection, the system can withstand 12 broken disks or more.
Most read operations come from the cache located on SSDs. This allows you to minimize the load on NL-SAS disks by accessing them for only a small amount of data that did not get into the cache.
Write operations fall into the DRAM-protected cache, which contains all changes to the blocks before they “cool down”, after which they are already sent to the disks. When data blocks are sent to disks, they are written to all disks simultaneously, which gives the system the ability to write them at high speed. Blocks of data that the system considers most likely to be read later are written not only to NL-SAS disks, but also to the SSD cache for faster reading.
This unique data layout allows InfiniBox to avoid using SAS disks and instead use a combination of SSD and NL-SAS disks to achieve high performance and low IO latencies.
It is worth noting that this is a system that does a large number of tasks for users: it does not give access to disks - RAIDs are already done in advance, load balancing across disks and between controllers is done automatically, the controller firmware is updated with support etc.
This icon in the upper right corner indicates that you have established a connection with the Infinidat server and support will be able to access the system for some kind of diagnostics, etc. When a support employee connects to your array, the entire icon will turn green. Access for support can be disabled, and enabled only when necessary.
Storage efficiency
I would like to start with the fact that these systems do not support deduplication. At all. Only compression, about which I will tell a little more.
To prevent data compression from affecting system performance, Infinibox compresses it when data is transferred from the cache to disks. Currently, the system uses the LZ4 compression algorithm, but the storage system architecture is built in such a way that in the future it will be possible to replace it with some newer and more efficient one. Do not forget that data compression is a processor-dependent operation and even when using modern processors it can affect the overall system performance. Compression is done in 64KB blocks, which allows to achieve a higher compression ratio.
Compression works with both thick and thin volumes, but its work in these cases is different.
- Thin volumes - space-saving from data compression remains available in the physical capacity of the pool.
- Thick volumes - the space freed up in this way is not available for re-allocation.
So, how InfiniBox saves disk space without affecting performance: the data is in the RAM cache in an uncompressed form. Compression occurs when data is written to disk. But it is the record on the HDD, because what gets to the SSD cache is also there in uncompressed form. Infinidat assumes that most of the data you are working with is cached, so compression will not increase response time. In fact, we understand that this is far from always the case, but if you look at the general architecture, which involves storing data on a large number of disks in small blocks, this avoids a strong increase in response.
If you are interested in learning more about the operation of this system, I suggest referring to the article “ Storage Architecture for a Digital Enterprise””Vasily Kostryukov, Technical Director, Infinidat Russia.
Replication
InfiniBox supports both synchronous and asynchronous replication. Asynchronous replication is based on snapshots. Synchronous replication sends I / O to both systems (primary and backup) and this increases the response time, because the host receives confirmation of data recording only after recording to both systems. Reading occurs only from the local system.
- The host sends I / O to write to source InfiniBox
- Source InfiniBox confirms I / O record to host
- Source InfiniBox replicates data to target InfiniBox
- Target InfiniBox confirms replication to source InfiniBox
If, according to the system, the quality of communication between the systems will not satisfy the requirements of synchronous replication, which is already configured, it will automatically transfer it to asynchronous mode and return it to synchronous mode when restoring the proper connection quality.
Asynchronous replication is supported on:
- volumes
- file systems
- consistency groups.
Synchronous replication to:
- volumes.
Operations that are not supported on replicated entities:
- removal
- recovery
- change of size,
- disable write-protect,
- file system export.
In synchronous replication mode, changing the direction of replication is supported.
To create a link for asynchronous replication, you will need four ports on each system (three data ports and one management port), while synchronous replication will require seven ports (six data ports and one management port).
Application Integration
Infinidat Host PowerTools for VMware enables virtual environment administrators to independently manage disk allocation. This facilitates and saves the time spent on basic operations. Host PowerTools also supports VASA (vStorage APIs for Storage Awareness).
Literally on March 1, a plugin for integrating this backup system with Infinibox arrays appeared on the Veeam website , which was developed by Infinidat in collaboration with Veeam, which allows us to hope for the high quality of this product. This plugin allows you to use consistent storage snapshots to increase backup speed and reduce the impact of backups on a productive environment.
Interface, RESTful API and SDK
I will depart from the tradition of taking many screenshots of the interface and this time I made a video that was accompanied by the necessary comments.
InfiniBox systems perfectly integrate into any cloud infrastructure management system thanks to its API, which allows you to quickly and conveniently work with the system. It does not matter whether you provide services and the allocation of resources is done using the control panel or the system only serves your own infrastructure and you need to quickly and conveniently add a large number of elements. All this is perfectly implemented using the API. There is also an InfiniSDK for Python .
Add 1000 hosts? Not a problem!
#!/usr/bin/python
#! -*- coding: utf-8 -*-
from infinisdk import InfiniBox
from infi.dtypes.iqn import make_iscsi_name
datastring = '''
iqn.1990-06.vmware:host1
…
iqn.1991-06.vmware:host1000
'''
system = InfiniBox('infinidat_ip', auth=('infinidat_login', 'infinidat_password'))
system.login()
datastring = datastring.splitlines()
for i, iqn in enumerate(datastring):
host_name = 'host-' + str(i)
host = system.hosts.create(name=host_name)
address = make_iscsi_name(iqn)
host.add_port(address)
The other day, the portal code.infinidat.com even appeared , on which you can find various scripts for integration with third-party systems (Openstack and Saltstack), modules for Ansible, and much more. So if you are faced with the task of integrating this array not just as part of the physical infrastructure, but convenient management, billing, etc., you will find many useful things on this portal.
Given that cloud Oncloud.ru company "ONLANTA" provides power for corporate customers under heavy loads (such as ERP, CRM, database, etc.), it has developed the architecture and develop their own capacity management tools, InfiniBox F2230 system perfectly Suitable for our infrastructure.
By the way, a number of interesting vacancies are open in our company