Let's get acquainted with the alpha version of the snapshot volumes in Kubernetes
- Transfer

Note trans. : The original article was recently published on the Kubernetes blog and written by Google and Huawei (Jing Xu, Xing Yang, Saad Ali), whose activities you will certainly see in the GitHub project if you have ever been interested in the K8s features and problems with data storage. Engineers talk about the purpose of volume snapshots, their current capabilities and the basics of working with them.
Kubernetes v1.12 presented an alpha version of snapshot support for volumes. This feature allows you to create and delete snapshots of volumes, as well as create new volumes from snapshots using the native means of the system through the Kubernetes API.
What is snapshot?
Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage and multiple on-premise storage systems) offer the ability to create a snapshot (snapshot) for a persistent volume. A snapshot is a copy of a volume at a specific point in time. It can be used to provision a new volume (already filled with data from snapshot) or restore an existing volume to the previous state (which is represented in snapshot).
Why add snapshots to Kubernetes?
A powerful abstraction is available in the plugin system of the Kubernetes volumes that automates the provisioning, mounting and mounting of block and file storages.
Providing all of these capabilities is part of Kubernetes workload portability goals: Kubernetes strives to create a level of abstraction between applications running as distributed systems and underlying clusters in such a way that applications do not depend on the specifics of the cluster on which they are running and the application does not need to be deployed. any cluster-specific knowledge. Kubernetes Storage SIG
Groupidentified snapshot operations as critical opportunities for a multitude of stateful workloads. For example, a database administrator may want to snapshot his database before performing any operation with it.
Having received a standard way to invoke snapshot operations in Kubernetes API, Kubernetes users can work with them without the need for workarounds (and manual invocation of operations that are specific to the storage system). Instead, users were given the opportunity to embed snapshot operations into their tools and policies with a calm understanding that everything will work with any Kubernetes clusters, regardless of the underlying storage.
In addition, these Kubernetes primitives function as basic building blocks, opening up possibilities for developing more advanced enterprise-level storage management features — for example, data protection, replication, and migration.
What volume plug-ins support Kubernetes snapshots?
Kubernetes supports three types of plugin volumes: in-tree, Flex and CSI. See the Kubernetes Volume Plugin FAQ for details .
Snapshots are supported only for CSI drivers (they are not supported for in-tree or for Flex). To take advantage of this opportunity, make sure that the Kubernetes cluster has a CSI driver deployed that supports snapshots.
By the time of this blog post (October 9, 2018 - approx. Transl. ) , Snapshots are supported by the following CSI drivers:
Support for snapshots for other drivers is in development and should be available soon. More details on CSI and how to deploy CSI drivers are described in the “ Container Storage Interface (CSI) for Kubernetes Goes Beta ” publication (also see our translation of the note “ Understand the Container Storage Interface (in Kubernetes and beyond) ” - approx. Transl. ) .
Kubernetes API for snapshots
To manage snapshots in Kubernetes Volume Snapshots, three new API objects are presented in the same way as in the Kubernetes Persistent Volumes API:
VolumeSnapshot
- Created by the user Kubernetes to request to create a snapshot for the specified volume. Contains information about snapshot operations, such as timestamp snapshot removal and whether it is ready for use.
- Like an object
PersistentVolumeClaim
, the creation and deletion of this object is the desire of the user to create or delete a cluster resource (snapshot).
VolumeSnapshotContent
- Created by the CSI driver when the snapshot was successfully created. Contains information about snapshot including its ID.
- Like an object
PersistentVolume
, it represents a resource already serviced by a cluster (snapshot). - Like objects
PersistentVolumeClaim
andPersistentVolume
when the snapshot is created, the object isVolumeSnapshotContent
attached to theVolumeSnapshot
one for which it was created (one-to-one mapping is used).
VolumeSnapshotClass
- It is set by cluster administrators to describe which snapshots can be created. Includes information about the driver, secrets for accessing snapshots, etc.
It is important to note that - unlike the main Persistent Volume objects in Kubernetes, these snapshots objects are defined as CustomResourceDefinitions (CRDs) . The Kubernetes project is gradually moving away from the previously defined types of resources in the API Server, approaching the model in which the API Server does not depend on API objects. This approach allows you to reuse the server API in other projects (besides Kubernetes), and consumer'y (like the same Kubernetes) can set the types of resources they need as CRD.
CSI drivers supporting snapshots will automatically install the necessary CRDs. The end users of Kubernetes will only need to verify that the CSI driver supporting snapshots is deployed in a cluster.
In addition to these new facilities, an existing
PersistentVolumeClaim
a new field has appeared DataSource
:type PersistentVolumeClaimSpec struct {
AccessModes []PersistentVolumeAccessMode
Selector *metav1.LabelSelector
Resources ResourceRequirements
VolumeName string
StorageClassName *string
VolumeMode *PersistentVolumeMode
DataSource *TypedLocalObjectReference
}
This field (in the status of alpha version) allows you to automatically fill it with data from an existing snapshot when creating a new volume.
Kubernetes snapshots requirements
Before using snapshots of volumes in Kubernetes, you must:
- make sure that the CSI driver that implements snapshots is deployed and running on the cluster;
- enable Kubernetes Volume Snapshotting via a new feature gate (disabled by default for the alpha version):
- set the following flag for the executable API Server file:
--feature-gates=VolumeSnapshotDataSource=true
- set the following flag for the executable API Server file:
Before creating a snapshot, you must also determine which CSI driver to use, which is done by creating an object
VolumeSnapshotClass
and specifying the CSI driver in the field snapshotter
. In the example below, with VolumeSnapshotClass
this driver is com.example.csi-driver
. Each supplier of snapshots requires at least one object VolumeSnapshotClass
. It is also possible to define VolumeSnapshotClass
the default for each CSI driver - this is done by setting the annotation snapshot.storage.kubernetes.io/is-default-class: "true"
in the class definition:apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotClass
metadata:
name: default-snapclass
annotations:
snapshot.storage.kubernetes.io/is-default-class: "true"
snapshotter: com.example.csi-driver
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotClass
metadata:
name: csi-snapclass
snapshotter: com.example.csi-driver
parameters:
fakeSnapshotOption: foo
csiSnapshotterSecretName: csi-secret
csiSnapshotterSecretNamespace: csi-namespace
All required parameters must be set in accordance with the CSI driver documentation. In the example above, the CSI driver during the creation and deletion of the snapshot will be passed the parameter
fakeSnapshotOption: foo
and all the mentioned secrets. CSI external-snapshotter defaults to the parameter keys csiSnapshotterSecretName
and csiSnapshotterSecretNamespace
. Finally, before creating a snapshot, you need to create a volume through the CSI driver and fill it with the data you want to see there (see this publication for details on how to use CSI volumes).
Creating a new snapshot in Kubernetes
Once an object is
VolumeSnapshotClass
defined and there is a volume from which you want to take a snapshot, you can perform this operation by creating an object VolumeSnapshot
. The source for snapshot is determined by two parameters:
kind
- is indicated herePersistentVolumeClaim
;name
- the actual name of the PVC object.
It is understood that the namespace of the volume for which the snapshot is created is determined by the object namespace
VolumeSnapshot
.apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
name: new-snapshot-demo
namespace: demo-namespace
spec:
snapshotClassName: csi-snapclass
source:
name: mypvc
kind: PersistentVolumeClaim
The specification
VolumeSnapshot
can be defined VolumeSnapshotClass
, which contains information about which CSI driver will be used to create snapshots. As previously reported, after the object has been created, the VolumeSnapshot
parameter fakeSnapshotOption: foo
and all the secrets mentioned VolumeSnapshotClass
are transferred to the CSI plugin com.example.csi-driver
in the call CreateSnapshot
. In response to such a request, the CSI driver takes a snapshot of the volume and then automatically creates an object
VolumeSnapshotContent
that represents the new snapshot, and binds this object to VolumeSnapshot
, making it ready for use. If the CSI driver cannot create snapshots and returns an error, the snapshot controller reports this error in the object status VolumeSnapshot
and does notmakes new attempts (this behavior differs from other controllers in Kubernetes - it is implemented in order not to create snapshots in unpredictable times). If the snapshot class is not set, external-snapshotter will try to find the default class and use it for the snapshot being created. At the same time, the CSI driver pointed
snapshotter
to in the default class must correspond to the CSI driver pointed to provisioner
in the PVC storage class. Please note that the alpha release snapshots for Kubernetes does not guarantee consistency. To ensure complete data in the snapshot, it is necessary to prepare the application accordingly (stop the application, freeze the file system, etc.) before removing it.
Make sure that the object
VolumeSnapshot
Created and linked to VolumeSnapshotContent
, you can command kubectl describe volumesnapshot
:Ready
must be settrue
toStatus
, which will indicate the readiness of the snapshot volume for use.- The field
Creation Time
indicates when the snapshot was actually made. - Field
Restore Size
- the minimum size of the volume to restore snapshot. - The field
Snapshot Content Name
in the specification points to the objectVolumeSnapshotContent
created for this snapshot.
Importing an existing snapshot into Kubernetes
You can import an existing snapshot into Kubernetes by manually creating an object
VolumeSnapshotContent
that will represent this snapshot. Since it VolumeSnapshotContent
is an API object that is not tied to a namespace, only a system administrator has the right to create it. When the object is
VolumeSnapshotContent
created, the user can create another object - VolumeSnapshot
, - which will point to it. The external-snapshotter controller will mark the snapshot as ready after checking for the existence and correctness of the connection between objects VolumeSnapshot
and VolumeSnapshotContent
. Snapshot is ready for use in Kubernetes when this connection is established. The object
VolumeSnapshotContent
must be created with the following fields representing the pre-provisioned snapshot:csiVolumeSnapshotSource
- snapshot identification information:snapshotHandle
- name / identifier for snapshot. Obligatory field;driver
- CSI driver used to work with this volume. Obligatory field. Must match the namesnapshotter
in the controller (snapshot controller);creationTime
andrestoreSize
- for pre-prepared volumes, these fields are optional. The external-snapshotter controller will automatically update them after creating the snapshot.
volumeSnapshotRef
- pointer to the objectVolumeSnapshot
to which this object (i.e.VolumeSnapshotContent
) should be tied:name
andnamespace
- the name and namespace of the objectVolumeSnapshot
, the contents of which is bound;UID
- optional (for pre-prepared volumes) field. The external-snapshotter controller will automatically update this field after binding. If the user specifies this field, you need to make sure that it corresponds to the UID of the snapshot for which the binding occurs. If there is no match, the content is considered irrelevant (orphan-object) and therefore the controller will delete both it and its associated snapshot.
snapshotClassName
- optional field. The external-snapshotter controller will automatically update it after binding.
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotContent
metadata:
name: static-snapshot-content
spec:
csiVolumeSnapshotSource:
driver: com.example.csi-driver
snapshotHandle: snapshotcontent-example-id
volumeSnapshotRef:
kind: VolumeSnapshot
name: static-snapshot-demo
namespace: demo-namespace
The object
VolumeSnapshot
must be created so that the user can work with snapshots. In him:snapshotClassName
- class name snapshot volume. Optional field. If set, the fieldsnapshotter
in the snapshot class must match the name of the snapshot controller. If not set, the controller will search for the default snapshot class;snapshotContentName
- the name of the contents of the snapshot volume. Required field for pre-prepared volumes.
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
name: static-snapshot-demo
namespace: demo-namespace
spec:
snapshotClassName: csi-snapclass
snapshotContentName: static-snapshot-content
When these objects are created, the snapshot controller will link them, set the field
Ready
(c Status
) as True
, indicating that the snapshot is ready for use.Preparing a new snapshot volume in Kubernetes
To create a new volume that is pre-filled with data from the snapshot object, use the new field
dataSource
in PersistentVolumeClaim
. It has three parameters:name
- the name of the objectVolumeSnapshot
representing the snapshot source;kind
- must be specified asVolumeSnapshot
;apiGroup
- should besnapshot.storage.k8s.io
.
It is assumed that the namespace of the source object
VolumeSnapshot
corresponds to the namespace of the object PersistentVolumeClaim
.apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-restore
Namespace: demo-namespace
spec:
storageClassName: csi-storageclass
dataSource:
name: new-snapshot-demo
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
When the object
PersistentVolumeClaim
is created, it will cause the provisioning of a new volume, pre-filled with data from the specified snapshot.How can I add snapshots support to my CSI driver if I am a storage developer?
To support snapshots in the CSI-driver additional capabilities of the controller should be added:
CREATE_DELETE_SNAPSHOT
and LIST_SNAPSHOTS
- as well as implemented additional RPC Controller: CreateSnapshot
, DeleteSnapshot
, ListSnapshots
. See the CSI specification for details . Although Kubernetes provides the most minimal guidelines for packaging and deploying the CSI Volume Driver, there is a recommended mechanism for deploying an arbitrary containerized CSI driver to Kubernetes to simplify this process.
As part of the recommended deployment process, the Kubernetes team suggests using multiple sidecar- (i.e., ancillary) containers, including a sidecar-container with an external-snapshotter .
Said external-snapshotter in API Server monitors the objects
VolumeSnapshot
and VolumeSnapshotContent
causing operation CreateSnapshot
and DeleteSnapshot
for CSI endpoint. The sidecar-container with CSI external-provisioner has also been updated to support recovery of the volume from snapshot using a new field dataSource
in the PVC. To support snapshot capabilities, storage vendors are recommended to deploy sidecar containers with an external-snapshotter in addition to an external provisioner, and place the CSI driver in
StatefulSet
, as shown in the diagram below: 
In this example, DeploymentThere are two sidecar containers, the external provisioner and an external snapshotter, and the CSI drivers are included with the CSI hostpath plugin within the StatefulSet pod. The CSI hostpath is an example of a plugin that is not intended for use in production.
What are the limitations of the alpha version?
The alpha version of the implementation of snapshots in Kubernetes has the following limitations:
- It does not support rollback of the existing volume to the previous state represented by snapshot (only provisioning of the new volume from snapshot is supported).
- The in-place restore of the existing
PersistentVolumeClaim
snapshot is not supported : i.e. The provisioning of a new volume from snapshot works, but not updating the existing onePersistentVolumeClaim
so that it points to a new volume and the PVC rolls back to an earlier state (only the use of a new volume created from snapshot through a new PV / PVC is supported). - The snapshot consistency guarantees do not go beyond the guarantees provided by the storage system (for example, integrity in a fall).
What's next?
The Kubernetes team plans to bring the implementation of snapshots for CSI to a beta version in releases 1.13 or 1.14, depending on the feedback received and the adaptation of the technology.
How to find out more details?
See additional snapshots documentation on k8s.io/docs/concepts/storage/volume-snapshots and kubernetes-csi.github.io/docs .
PS from translator
Read also in our blog:
- “ Kubernetes 1.12: a review of major innovations ”;
- “We understand Container Storage Interface (in Kubernetes and not only) ”;
- “ Rook is a“ self-service ”data store for Kubernetes ”;
- “ Our experience with Kubernetes in small projects ” (review and video of the report) ;
- “ Monitoring and Kubernetes ” (review and video of the report) .