The birth of a virtual mobile operator: Tinkoff Bank project
Today, banks are increasingly “digitizing” customer services and communication channels with their customers: personalize service based on customer data, introduce remote self-service services, chat bots, virtual assistants, including systems with artificial intelligence (AI) and speech recognition technologies . In an effort to expand the range of client services, banks are developing new areas of activity. One example is the technology of mobile virtual network operators (MVNO). Now we, together with Tele2, are participating in the Tinkoff Bank project to create the Tinkoff Mobile virtual operator.
What is a “virtual mobile operator”?
There are already good articles on Habré explaining the essence of the “virtual mobile operator”. Here we just say that a virtual mobile operator is a virtual (software) network managed by one company, using a physical communication network owned by another company.
Tinkoff Mobile is built under the Full MVNO scheme. This means that when all the components are put into operation, from the point of view of creating customer services, the new virtual operator will not differ much from the “real” operator. Tinkoff Mobile will lease the resources of the base station network from the partner operator, which is Tele2, and MVNO will have its own network core infrastructure and business applications. All client traffic will go through a virtual operator, which is actually an Internet entry point for subscribers.
Virtual Operator Architecture
Our company was chosen to build the core elements of a mobile network. This was the first such project for us - today MVNO is few in the world, especially in Russia. From the start of the project to the first customer call, only 4 months have passed, which is very fast by the standards of the telecom industry. The system is designed so that you can upgrade it without interruption, just adding compute nodes.
What are the “cubes” of the virtual operator architecture? At the moment, it looks like this:
The infrastructure consists of two identical, completely autonomous sites with the function of a geo-reserve: if one site fails, subscribers will automatically continue to service the second. At the same time, normally both sites operate in an active mode, equally sharing traffic between subscribers. All systems have multiple redundancy, as is customary in the industry. In addition, high availability clusters are widely used to increase fault tolerance.
All components shown in the diagram are software executable on ordinary serial servers. Basically, these are solutions not of traditional large telecom vendors, but new products of companies that are actively entering the telecom solutions market and crowding out old players. In particular, Affirmed Networks PGW / GGSN nodes, Procera Networks PCEF (DPI) nodes, and Oracle PCRF nodes were used. Of course, traditional vendors also began to move towards virtualized software solutions, but so far they have not been able to completely "get rid" of their old technologies, while new companies do not have such a legacy, and their solutions can be run on any platform, on any servers of any manufacturers.
In the telecom industry, Network Functions Virtualization (NFV) is increasingly being used, virtualization technology for network elements of a telecommunication network, in fact, virtualization of components providing a service for subscribers of a telecom operator. Today, one of the main trends is the rejection of the specialized and very expensive equipment of traditional vendors in favor of serial servers (commercial off-the-shelf, COTS) of x86 architecture with specialized software installed on them. Moreover, it works under the control of a virtual infrastructure, that is, it runs inside virtual machines.
NFV uses a variety of virtualization technologies. In addition to basic hardware virtualization, which allows executing software modules in a virtual machine that emulates interaction with the real hardware of the server, it is possible to build a software-defined SDN network based on virtual elements of network functions.
At high network loads, it is very important to ensure reliable interaction with the network subsystem from software modules running inside the virtual machine. That is, the issue of speed with network interfaces is especially acute. One way is to use PCI Passthrough mode, in which the entire PCI device is transferred to the guest operating system. This allows her to work directly with the device without using the emulation layer on the side of the hypervisor. However, this is a resource-intensive method, it does not scale and binds the guest OS, and hence the network function, to a specific instance of a PCI device.
Another drawback of PCI Passthrough mode is its low density of resources due to the impossibility of sharing a single device between several guest OSs, because each guest OS in this mode uses the device exclusively. Therefore, we proposed an alternative approach - Single Root I / O Virtualization (SR-IOV) technology.
SR-IOV allows you to use the device directly, as in PCI Passthrough mode, bypassing the hypervisor. But at the same time, the device is available simultaneously for several virtual machines, independent processing of interrupts and DMA for each machine is performed using Virtualization Technology for directed I / O (VT-d).
When the SR-IOV mode is enabled, the network device is “split” into one physical function (Physical Function or PF) and several virtual functions (Virtual Function or VF). Physical function (PF) remains at the level of the hypervisor and under its control. Virtual functions (VFs) are transferred to guest OSs and become network interfaces of virtual network functions (NFVs) for interacting with the outside world. The issue of VF performance inside VNF is solved by using the Data Plane Development Kit (DPDK) framework. It was originally developed at Intel, and then transferred to the open community. The framework can significantly improve the performance of NFV when processing network traffic. Using a combination of DPDK and SR-IOV to virtualize network functions is a must when building a high-performance NFV solution,
Summary
As already mentioned, the project was implemented very quickly. The composition of the equipment needed to be determined at the very first stage of the project, so that by the time the work on the design of the system was completed, the equipment was already delivered to the sites and ready for installation. The task was also complicated by the use of solutions of different vendors, this required the coordinated work of all the teams working on the project.
After starting the system, the first call and the first access to the network on a smartphone with a SIM card of a new virtual mobile operator, business rules were set up for another two months, acceptance tests were carried out. As a result, the system entered commercial operation on December 14, 2017.
