Understand the Container Storage Interface (in Kubernetes and not only)
- Transfer
Note trans. : about the so-called “out-of-tree” Kubernetes (Out-of-Tree CSI Volume Plugins) storage plug-ins, we first talked about in our review of the K8s 1.9 release , where this feature appeared in the alpha version status. The author of the new material, Anoop Vijayan Maniankara (Lead DevOps Engineer of the Finnish company Tuxera), collected key information about ideas and the CSI device, which helps to quickly get acquainted with the new concept, which some of our employees claim will be “next big thing”. For a more detailed and technical study of CSI at the end of the article are useful links, among which I will highlight the presentation of one of the authors of this specification (Jie Yu). But it’s still worth starting with the “big picture” ...
Container Storage Interface (CSI) is an initiative designed to unify the storage interface, such as Ceph, Portworx, NetApp, etc., in container orchestration systems: Kubernetes, Mesos, Docker Swarm, Cloud Foundry, and others. The idea is that the implementation of one CSI by the storage manufacturer is guaranteed to work with all these systems.
Image source: CSI talk from Jie Yu to CloudNativeCon EU 2018
Note : this article will only cover dynamic provisioning. Pre-configured volumes and flex volumes go beyond its scope. If you want to better understand what will be discussed, you should first read the Kubernetes documentation . In addition, the article will not be deeply immersed in the details of the implementation of CSI. I will present a high-level overview of CSI and lay the foundation for creating a CSI volume. One last thing: for examples and references to details, the information for Kubernetes is used.
Before diving into the topic, it is also important to know what sidecar-containers are in Kubernetes. They extend the capabilities of the main container ( main), existing in the same file, sharing storage and network.
At the time of this writing (August 13, 2018), the CSI components had the following versions:
The first release of CSI - v0.1 - took place in December 2017. Of course, provisioning for external storage in orchestration systems was possible before it appeared. In the case of Kubernetes, volume plugins answered for storage requirements: Volume plugins:
As you can see from the picture above, such plugins are part of the core of the orchestration system. Because of this, the following issues mentioned in the CSI architecture document occurred :
Representing CSI, the Kubernetes team has released external components that are not part of the kernel and are designed to interact with other external components sold by manufacturers. Between themselves, they communicate through domain sockets (UNIX domain sockets - approx. Transl.) Using gRPC .
They are fully implemented and maintained by the Kubernetes team and extend the actions of Kubernetes outside Kubernetes. Manufacturers can not worry about the features of their implementation. Consist of three parts:
Vendor specific implementation. Each manufacturer implements the necessary APIs as part of the functions of the gRPC service. For example, the implementation of GCE PD or Ceph , etc. They also consist of three components:
The emergence of CSI brought an obvious plus to orchestration systems and repository manufacturers. In addition, well-defined interfaces help the simple implementation and testing of CSI to both developers and future orchestration systems. If you decide to start implementing your CSI after reading this material, a good starting point will be the article “ How to write a Container Storage Interface (CSI) plugin ” from Fatih Arslan.
Read also in our blog:
Container Storage Interface (CSI) is an initiative designed to unify the storage interface, such as Ceph, Portworx, NetApp, etc., in container orchestration systems: Kubernetes, Mesos, Docker Swarm, Cloud Foundry, and others. The idea is that the implementation of one CSI by the storage manufacturer is guaranteed to work with all these systems.
Image source: CSI talk from Jie Yu to CloudNativeCon EU 2018
Note : this article will only cover dynamic provisioning. Pre-configured volumes and flex volumes go beyond its scope. If you want to better understand what will be discussed, you should first read the Kubernetes documentation . In addition, the article will not be deeply immersed in the details of the implementation of CSI. I will present a high-level overview of CSI and lay the foundation for creating a CSI volume. One last thing: for examples and references to details, the information for Kubernetes is used.
Before diving into the topic, it is also important to know what sidecar-containers are in Kubernetes. They extend the capabilities of the main container ( main), existing in the same file, sharing storage and network.
At the time of this writing (August 13, 2018), the CSI components had the following versions:
Before the advent of CSI
The first release of CSI - v0.1 - took place in December 2017. Of course, provisioning for external storage in orchestration systems was possible before it appeared. In the case of Kubernetes, volume plugins answered for storage requirements: Volume plugins:
As you can see from the picture above, such plugins are part of the core of the orchestration system. Because of this, the following issues mentioned in the CSI architecture document occurred :
- plugin volume development is tightly linked to and dependent on Kubernetes releases;
- the Kubernetes developers / community are responsible for testing and supporting all plug-ins, instead of just testing and maintaining a stable plug-in API;
- Bugs in volume plugins can drop not only the plugin itself, but also the critical components of Kubernetes;
- plugins receive full privileges of the components Kubernetes (kubelet and kube-controller-manager);
- plugin developers are forced to publish the source code of the plugin and can not choose the path of the binaries.
Understand CSI
Representing CSI, the Kubernetes team has released external components that are not part of the kernel and are designed to interact with other external components sold by manufacturers. Between themselves, they communicate through domain sockets (UNIX domain sockets - approx. Transl.) Using gRPC .
Kubernetes external components
They are fully implemented and maintained by the Kubernetes team and extend the actions of Kubernetes outside Kubernetes. Manufacturers can not worry about the features of their implementation. Consist of three parts:
- Driver registrar is a sidecar container that registers a CSI driver with a kubelet and adds
NodeIddrivers to the label of the node object in the Kubernetes API. To do this, it interacts with the service of the CSI-driver Identity (for more information about it, see below - approx. Transl.) And callsGetNodeIdthe CSI; - Provisioner the External - the sidecar-container, watching objects PersistentVolumeClaim in Kubernetes API and invoke commands
CreateVolumeandDeleteVolumeto endpoint'a driver; - Attacher the External - the sidecar-container, watching objects VolumeAttachment in Kubernetes API and invoke commands
ControllerPublishandControllerUnpublishto endpoint'a driver.
External component from storage / third party manufacturer
Vendor specific implementation. Each manufacturer implements the necessary APIs as part of the functions of the gRPC service. For example, the implementation of GCE PD or Ceph , etc. They also consist of three components:
- CSI Identity - mainly for plugin identification: make sure that it is functioning, return basic information about the plugin;
service Identity { // вернуть версию и название плагина rpc GetPluginInfo(GetPluginInfoRequest) returns (GetPluginInfoResponse) {} // сообщить, в состоянии ли плагин обслуживать интерфейс Controller rpc GetPluginCapabilities(GetPluginCapabilitiesRequest) returns (GetPluginCapabilitiesResponse) {} // вызывается системой оркестровки, чтобы проверить, запущен ли плагин rpc Probe (ProbeRequest) returns (ProbeResponse) {} }
( kubernetes-csi-identity.proto ) - CSI Controller is responsible for controlling and managing volumes: creating, deleting, attaching / detaching, snapshots, etc .;
service Controller { // выполнить provisioning тома rpc CreateVolume (CreateVolumeRequest) returns (CreateVolumeResponse) {} // удалить выделенный ранее том rpc DeleteVolume (DeleteVolumeRequest) returns (DeleteVolumeResponse) {} // сделать том доступным на требуемом узле rpc ControllerPublishVolume (ControllerPublishVolumeRequest) returns (ControllerPublishVolumeResponse) {} // сделать том недоступным на требуемом узле rpc ControllerUnpublishVolume (ControllerUnpublishVolumeRequest) returns (ControllerUnpublishVolumeResponse) {} // например, может использоваться для одновременного чтения/записи с нескольких узлов rpc ValidateVolumeCapabilities (ValidateVolumeCapabilitiesRequest) returns (ValidateVolumeCapabilitiesResponse) {} // вернуть все доступные тома rpc ListVolumes (ListVolumesRequest) returns (ListVolumesResponse) {} // полная емкость доступного пространства в пуле rpc GetCapacity (GetCapacityRequest) returns (GetCapacityResponse) {} // например, плагины могут не иметь реализации GetCapacity и Snapshotting rpc ControllerGetCapabilities (ControllerGetCapabilitiesRequest) returns (ControllerGetCapabilitiesResponse) {} // сделать снапшот rpc CreateSnapshot (CreateSnapshotRequest) returns (CreateSnapshotResponse) {} // удалить снапшот rpc DeleteSnapshot (DeleteSnapshotRequest) returns (DeleteSnapshotResponse) {} // вывести список снапшотов rpc ListSnapshots (ListSnapshotsRequest) returns (ListSnapshotsResponse) {} }
( kubernetes-csi-controller.proto ) - CSI Node is responsible for controlling the volume actions on the Kubernetes node.
service Node { // временно примонтировать том к staging-пути rpc NodeStageVolume (NodeStageVolumeRequest) returns (NodeStageVolumeResponse) {} // отмонтировать том от staging-пути rpc NodeUnstageVolume (NodeUnstageVolumeRequest) returns (NodeUnstageVolumeResponse) {} // примонтировать том из staging к целевому пути rpc NodePublishVolume (NodePublishVolumeRequest) returns (NodePublishVolumeResponse) {} // отмонтировать том от целевого пути rpc NodeUnpublishVolume (NodeUnpublishVolumeRequest) returns (NodeUnpublishVolumeResponse) {} // статистика по тому rpc NodeGetVolumeStats (NodeGetVolumeStatsRequest) returns (NodeGetVolumeStatsResponse) {} // вернуть уникальный ID узла rpc NodeGetId (NodeGetIdRequest) returns (NodeGetIdResponse) { option deprecated = true; } // вернуть возможности (capabilities) узла rpc NodeGetCapabilities (NodeGetCapabilitiesRequest) returns (NodeGetCapabilitiesResponse) {} // схоже с NodeGetId rpc NodeGetInfo (NodeGetInfoRequest) returns (NodeGetInfoResponse) {} }
( kubernetes-csi-node.proto )
Conclusion
The emergence of CSI brought an obvious plus to orchestration systems and repository manufacturers. In addition, well-defined interfaces help the simple implementation and testing of CSI to both developers and future orchestration systems. If you decide to start implementing your CSI after reading this material, a good starting point will be the article “ How to write a Container Storage Interface (CSI) plugin ” from Fatih Arslan.
References to the literature
- CSI Specification ;
- Sidecar containers at Kubernetes ;
- Report on CSI from Jie Yu at KubeCon EU: CloudNativeCon EU 2018 (and here is the video from this performance available - approx. Transl.) ;
- CSI Architecture Document ;
- Current CSI documentation from Kubernetes ;
- Outdated CSI documentation from Kubernetes .
PS from translator
Read also in our blog:
- “ We are getting acquainted with the alpha version of the snapshot volumes in Kubernetes ”;
- “ Kubernetes 1.9: review of major innovations ”;
- “ Understand RBAC in Kubernetes ”;
- “ What happens in Kubernetes when starting the kubectl run? Part 1 ";
- “ How does the Kubernetes scheduler actually work? ";
- " Behind the scenes of the network in Kubernetes ";
- “ Rook -“ self-service ”data store for Kubernetes ”;
- " Our experience with Kubernetes in small projects " (video of the report, which includes an introduction to the technical device Kubernetes) .