Kubernetes success stories in production. Part 7: BlackRock / Flant Blog

shurup January 23, 2018 at 10:47

Kubernetes success stories in production. Part 7: BlackRock

BlackRock is one of the largest investment companies in the world and the largest in terms of assets under management (USD 5.7 trillion as of July'17) . It is also called the world's largest “ shadow bank ”. With a 30-year history and such impressive performance, the company does not lag behind current trends in IT infrastructure. Last Friday, the organization CNCF blog appeared information that BlackRock managed the 100-day roll out its production-environment on Kubernetes. How did they come to this?

Background

BlackRock's potential interest in Kubernetes became publicly known in 2015-2016, when the company began to host relevant thematic meetings. For example, in September 2015, representatives of Mesosphere and Calico spoke at an event organized in BlackRock, and in January 2016, the next CoreOS London Meetup was held in the office of the company .

Aladdin and Docker

However, interest in technology alone does not guarantee anything when it comes to practical application. And in May last year, a note appeared on the BlackRock technical blog about the transfer of the Aladdin software platform to cloud architecture.

Aladdin (stands for Asset Liability and Debt and Derivatives Investment Network) offers investment managers risk analytics, combined with portfolio management and trading tools.

Aladdin functionality (for more details, see the product website )

The core originally created for this solution — the Aladdin Core Platform — seemed to be “a single unified technology stack for all applications, which was also called the Aladdin Operation System.” This "operating system" was launched on each host where the platform applications were running. The chosen approach ensured the availability of all necessary (common for applications) capabilities in the host system and did not require a virtualization layer.

However, over time, serious problems began to appear, caused by:

the need to run third-party utilities for Big Data analysis on the same machines (Hadoop and Spark);
the need for simple scaling when bursting the number of requests;
the need for a simple deploy of isolated test environments for various development teams.

All this led the company's engineers to the decision to switch to a cloud architecture , which implies the possibility of allocating a common pool of resources for any workloads (both main applications and third-party analytical software) and running processes on any hosts (for horizontal scaling of applications).

Schematic comparison of application architecture according to the BlackRock version

Aladdin decided to implement the cloud architecture using Docker containers . Then there was a need for an orchestration system, the key requirements for which were identified as follows: dynamic planning, service discovery and isolation. However, details about its selection and use at that time were not reported.

And last December from BitnamiThere is information that BlackRock is already a user of Kubeless , a serverless framework for Kubernetes. And later that same month , the first public details about the use of K8s all in the same Aladdin platform are distributed through various media outlets . Finally, they were complemented by a success story on Kubernetes.

Aladdin and Kubernetes

The tasks of the new web application for Aladdin, the infrastructure of which was implemented using Kubernetes, included providing a convenient interface that offers investors access to the analytical data they need. This approach has replaced the complex client (i.e., installed on user desktops) installations with Python and R, interacting with the project servers where distributed computing was launched. To implement it, a team of 20 people from different BlackRock divisions was assembled (for the development, operation of infrastructure and products, project management), who managed to achieve a successful result in 100 days .

The initial plan for creating a cloud-based web application was proposed by the developers of the company and was to use Ansible to describe all the necessary infrastructure. However, the operation specialists pointed out the poor prospects for this approach (according to their version, it would lead to “the appearance of a completely different product” and “prove to be too expensive”) and began to search for another solution.

Already having experience with “other cloud environments” (which, unfortunately, is not specified), the combined team of engineers opted for Kubernetes. The two main reasons called open source and confidence in the long-term prospects of the project. As the vice president of product operations at BlackRock (Uri Morris) explained, companies usually choose technologies that will be relevant in one form or another after another 5-10 years, and the young Kubernetes already looked quite convincing. This was partly due to the fact that the number of non-Google people who make changes to the K8s code base exceeded the number of Google commiters. As the distribution of Kubernetes, the implementation was chosen from a commercial company - OpenShift from Red Hat.

Pull request to add BlackRock to the OpenShift user list

Features and implementation results

In the process of implementing an architecture based on Kubernetes, the Aladdin platform was not without challenges. The main ones were:

corporate firewalls that prevented the installation of various packages according to ready-made instructions (as a result, some improvements were sent to the Minikube project code base);
difficulties in discovering services due to the use of a proprietary message bus for the interaction of numerous Aladdin services, which provide APIs for quickly creating applications;
integration with existing tools - for example, with the same message bus, which was done using a separate gateway through which the Kubernetes cluster interacts and whose built-in mechanisms made it possible to control and regulate incoming requests.

Note : The BlackRock Messaging System (BMS) is based on a request-response paradigm. The server side - BmsServer - is a multi-threaded application in C ++, and client libraries are available in Java, C ++, Python, JavaScript, Perl, C # and Julia. Read more about this system in the company's blog .

Illustrations of the device and operation of the BlackRock Messaging System

On the whole, the new infrastructure for the application based on Kubernetes was supplemented by existing and familiar tools for the operating team, which saved on staff (minimized the need to hire new people to service the project).

Процесс выкатывания инфраструктуры стал поэтапным и начался с окружения для разработки, за которым последовали тестовое и два production-окружения. На протяжении всего проекта в команде проводились еженедельные 1-часовые встречи со всеми участниками (распределёнными по разным странам), а также дополнительные более короткие обсуждения, посвящённые конкретным техническим моментам. Важной особенностью этого внедрения в целом представители BlackRock называют инициативность со стороны инженеров, которым доверили делать то, что они сами считают правильным.

The result of the project was the launch of the platform for internal users 100 days after the start of implementation. The infrastructure was designed for 30 users who appeared in the first hours after launch, so services were scaled up quickly to serve 150 people. The success of the implemented infrastructure allowed BlackRock to talk about its intention to transfer to Kubernetes and other applications of the company, but before making a final (and large-scale) decision on this issue, they want to accumulate operational experience in production, which, according to their estimates, will take from six months to a year. In addition, in the already mentioned interview, information is confirmed that the company began working with the serverless framework for Kubernetes - Kubeless.

And here is the technological stack used in BlackRock, which is listed in the current vacancy of the company - Senior Big Data Engineer :

Java, Python, Greenplum (SQL, SP, Functions), Java, Cassandra, ESP (Complex Event Processing), Git, Maven, Linux;
new product development may include Docker, Kubernetes, Spark, Kafka and Kafka Streams (the same list is indicated in the requirements for the candidate).

Among the responsibilities is also an item on the “deployment, design and scaling of microservices”.

Kubernetes success stories in production. Part 7: BlackRock / Flant Blog