Rebuilding Data Center Ecosystem

Recently, Rob Ober, a systems architect at LSI, gave a detailed interview to the Chinese magazine CEO & CIO, in which many interesting questions were raised. I would like to bring to your attention a translation of the key points of this interview, since Rob very clearly describes the future of the industry.

CEO & CIO: In recent years, Internet companies have built highly scalable data centers. Unlike ordinary companies, IT market players take on the role of leaders in the development of technologies for data centers. From an industry perspective, what three technologies are key to the data center market? Describe them.

Rob: The software industry, the hardware industry, and the purely engineering disciplines collide in super-large data centers. Therefore, the number of innovations is so great that choosing only three is very difficult. I would say that the most important are innovations in hardware and infrastructure, and if you need to choose three, I would point out the following points:

Autonomous behavior and management

A Microsoft architect once told me: “If we hired administrators for our data centers as regular companies do, we would have to hire all the administrators in the world.” Now, in Microsoft's data centers, there are about one million servers. Ultra-scalable data centers required for their expansion the development of automatic, self-managing and often even self-implementing infrastructure. Extra-large data centers are pioneers in this area, with the help of them, specialists learn from their own mistakes, develop practices that allow to obtain an improved work / dollar ratio. These are highly specialized practices, but more and more IT industry players are starting to adopt them. OpenStack is the best example of how specialized knowledge and skills are “packaged” and distributed widely throughout the industry.

High Availability at the Datacenter or Individual Machine Levels

As systems become larger, they have more components, more failure points, and this complicates and increases the cost of maintaining their availability. As storage volumes increase, drives begin to fail more often. They are simply more often used. And this is against the background of constant pressure designed to reduce cost and complexity. Over time, mega-DPCs have grown very much, sometimes in hundreds of thousands of servers, often distributed across different DPCs, which led to the need to create solutions to increase absolute reliability, while individual system components became cheaper, simpler and less reliable. All this allowed the use of low-cost components in the "clouds", turning them into a reliable resource.

These solutions also appeared very on time, as many organizations felt the need to keep their data absolutely accessible in different data centers.

The traditional approach, requiring server availability at 99.999%, is giving way to a pragmatic approach, which is to maintain high availability at the macro level - throughout the data center. This approach allows for the failure of individual systems and components until it threatens the entire data center. Of course, this approach has not yet been fully developed. LSI works with mega-DPCs and OEMs to get improved operational efficiency and fault tolerance techniques, which will minimize the harm from failure of individual components while maintaining a reliable high-availability layer for the entire data center.

Big data

This term is used too often. It is hard to believe that a few years ago it did not exist. Hadoop was a real gift for the industry - an open source attempt to copy Google MapReduce and the Google File System - actually changed our world incredibly fast. Today, Hadoop and other big data applications offer us search, analytics, advertising, scalable, reliable file systems, genetic research, and much more, even services like Apple Siri use Hadoop. Big data has changed the concept of analytics from statistical simplification to analysis of all data. And this has already provided many breakthroughs in research in which patterns and patterns are searched empirically, rather than theoretically.

In general, I think big data has become one of the most transformative technologies of this century. Big data has shifted the focus of data centers from storage computing. Our hard drive controllers, SAS (Serial Attached SCSI) adapters, and RAID controllers are at the center of this evolution. Its next step will be the widespread use of graph analytics, which will allow you to analyze the relationship between the data, and not just the data itself.

CEO & CIO: With the widespread adoption of cloud computing, mobile communications and big data, the traditional IT ecosystem in production is changing. What are the three main changes in the current interactions of LSI with the ecosystem? How do LSI see changes in different relationships in traditional ecosystems? What new connections are worth considering? Please provide examples.

Rob: Cloud computing and the availability of data from mobile devices have already changed significantly and will continue to change our industry and ecosystem. In fact, the corporate market (customers, OEMs, technologies, applications, and applications) remained fairly stable for 10 to 20 years, but as soon as cloud computing became a significant part of the server market, it immediately affected ecosystem participants such as LSI

Time: It’s not enough to follow Intel’s streamlined, like clockwork product portfolio. Previously, data center development cycles ranged from 3 to 5 years. But these cycles are getting shorter. Now the need for solutions is approaching 6 months, forcing hardware suppliers to work with such short development cycles.

Mega DPCs also need the ability to quickly build resources to meet customer needs. As a result, it is in the data centers that new architectures, solutions and specifications are introduced without the traditional binding to the Intel roadmap. It also disrupts the ecosystem.

End users:hyper-scalable data centers now play a significant client role in the ecosystem. Sometimes their single order can make up to 5% of the server market. Despite the fact that OEMs are still incredibly important, they no longer perform such large deployments and are developing not so fast. This leads to the fact that suppliers of individual components or subsystems often win financially if they are able to offer a unique (or at least effective) solution to a real problem. This leads to the fact that the main profit moves from large OEMs to strong, fast innovators. Potentially, this can lead to a decrease in profit margins for the entire ecosystem, which will threaten the growth rate of innovation and reinvestment.

New Players:traditionally, several OEMs and several software vendors almost exclusively owned the data center market. However, the supply chains of hyper-scalable cloud companies have changed this. The market leaders' data centers developed, specified, and even built (as in the case of Google) their own infrastructure, although some mega-data centers continue to rely on proven solutions from Dell and HP.

Increasingly, data centers are built to specifications from suppliers like Quanta. New network equipment vendors such as Arista are expanding their market share. Hyper-scalable solution providers such as Nebula are also growing.

The software has moved significantly towards open source with paid support - a model originally developed by RedHat, now adopted by Cloudera, Mirantis, United Stack and others.

Open initiatives:yes, we have already seen Hadoop and derivatives being introduced everywhere, even in traditional industries: oil and gas, pharmaceutical, genetic research, etc. And we watched open databases push traditional solutions (like Casandra). But now we are seeing new initiatives such as Open Compute and Open Stack. Of course, they are useful for hyper-scalable data centers, but they also help smaller companies and universities to deploy infrastructure that is similar to hyper-scalable, and get the same level of automated control, efficiency and costs as the “big players” (of course, they don’t use that level of capabilities, but they are very close to this). In the future, this trend will be able to severely damage the traditional business models of OEMs and software vendors and reshape the markets in favor of new players,

New architectures and algorithms: now there is a clear movement towards technologies based on a resource pool. The development of such solutions was made possible through a partnership between companies such as Intel and LSI and architects of the largest data centers. Traditionally, new approaches in architecture were dictated by OEM, but recently this is not so. We can see an increasing spread of solutions aimed at using scalable rack-architecture (RSA): silicon phototonics, storage pools, software-defined networks, and soon we will see RAM pools and new types of non-volatile RAM.

In addition, we can see how new processor architectures are gaining their place in the data center: ARM 64 for quiet and cold storage and OpenPower P8 for powerful computing, multi-threaded, multi-tasking processing monsters. All this is very interesting to watch. Interest in application acceleration is growing: general-purpose computing on video card processors, regular expression processors for real-time stream analysis, etc. Right before our eyes, the first generation of graph analysis tools is unfolding.

Innovation:the pace of innovation is growing, or I'm just getting older. But quick income is over. On the one hand, data centers need an exponential increase in computing power and storage, they need to work faster from 10 to 1000 times. On the other hand, memory, processor cores, disks, and flash drives are not growing so fast. The only way to fill this gap is through innovation. So it is not surprising that a lot of interesting things are happening now at OEMs, software vendors, chip makers and turnkey solutions, as well as in the open source community and startups. Here is what makes the present so interesting.

Consumption Shift:we see a decline in the supply of PCs and laptops, a decline that led to a decrease in demand for storage in this segment. Laptops are increasingly moving from HDD to SSD, which is not bad for LSI, since our contribution to mobile HDDs was small, while the company plays a large role in the SSD market. Smartphones and tablets have led to increased consumption of cloud content, traffic and dependence on cloud storage. We see a significant increase in demand for large HDDs for cloud solutions, this trend is picking up speed, and we think that the market for “cloud” HDDs will feel good, and we will see the emergence of new, cloud-optimized HDDs that are very different from existing and Designed for quiet storage with low heat.

Cloud storage is growing in demand for PCIe SSD cards, which are used for databases, caches, virtual machines and other applications requiring low latency. Much of what we take for granted would not be possible without these flash products with high capacity and low latency. Very few companies can offer viable flash storage at an affordable price. This paves the way for startups experimenting with various solutions.