Let's get acquainted with the alpha version of the snapshot volumes in Kubernetes

Original author: Jing Xu, Xing Yang, Saad Ali

Transfer

Note trans. : The original article was recently published on the Kubernetes blog and written by Google and Huawei (Jing Xu, Xing Yang, Saad Ali), whose activities you will certainly see in the GitHub project if you have ever been interested in the K8s features and problems with data storage. Engineers talk about the purpose of volume snapshots, their current capabilities and the basics of working with them.

Kubernetes v1.12 presented an alpha version of snapshot support for volumes. This feature allows you to create and delete snapshots of volumes, as well as create new volumes from snapshots using the native means of the system through the Kubernetes API.

What is snapshot?

Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage and multiple on-premise storage systems) offer the ability to create a snapshot (snapshot) for a persistent volume. A snapshot is a copy of a volume at a specific point in time. It can be used to provision a new volume (already filled with data from snapshot) or restore an existing volume to the previous state (which is represented in snapshot).

Why add snapshots to Kubernetes?

A powerful abstraction is available in the plugin system of the Kubernetes volumes that automates the provisioning, mounting and mounting of block and file storages.

Providing all of these capabilities is part of Kubernetes workload portability goals: Kubernetes strives to create a level of abstraction between applications running as distributed systems and underlying clusters in such a way that applications do not depend on the specifics of the cluster on which they are running and the application does not need to be deployed. any cluster-specific knowledge. Kubernetes Storage SIG

Groupidentified snapshot operations as critical opportunities for a multitude of stateful workloads. For example, a database administrator may want to snapshot his database before performing any operation with it.

Having received a standard way to invoke snapshot operations in Kubernetes API, Kubernetes users can work with them without the need for workarounds (and manual invocation of operations that are specific to the storage system). Instead, users were given the opportunity to embed snapshot operations into their tools and policies with a calm understanding that everything will work with any Kubernetes clusters, regardless of the underlying storage.

In addition, these Kubernetes primitives function as basic building blocks, opening up possibilities for developing more advanced enterprise-level storage management features — for example, data protection, replication, and migration.

What volume plug-ins support Kubernetes snapshots?

Kubernetes supports three types of plugin volumes: in-tree, Flex and CSI. See the Kubernetes Volume Plugin FAQ for details .

Snapshots are supported only for CSI drivers (they are not supported for in-tree or for Flex). To take advantage of this opportunity, make sure that the Kubernetes cluster has a CSI driver deployed that supports snapshots.

By the time of this blog post (October 9, 2018 - approx. Transl. ) , Snapshots are supported by the following CSI drivers:

Support for snapshots for other drivers is in development and should be available soon. More details on CSI and how to deploy CSI drivers are described in the “ Container Storage Interface (CSI) for Kubernetes Goes Beta ” publication (also see our translation of the note “ Understand the Container Storage Interface (in Kubernetes and beyond) ” - approx. Transl. ) .

Kubernetes API for snapshots

To manage snapshots in Kubernetes Volume Snapshots, three new API objects are presented in the same way as in the Kubernetes Persistent Volumes API:

VolumeSnapshot
- Created by the user Kubernetes to request to create a snapshot for the specified volume. Contains information about snapshot operations, such as timestamp snapshot removal and whether it is ready for use.
- Like an object PersistentVolumeClaim, the creation and deletion of this object is the desire of the user to create or delete a cluster resource (snapshot).
VolumeSnapshotContent
- Created by the CSI driver when the snapshot was successfully created. Contains information about snapshot including its ID.
- Like an object PersistentVolume, it represents a resource already serviced by a cluster (snapshot).
- Like objects PersistentVolumeClaimand PersistentVolumewhen the snapshot is created, the object is VolumeSnapshotContentattached to the VolumeSnapshotone for which it was created (one-to-one mapping is used).
VolumeSnapshotClass
- It is set by cluster administrators to describe which snapshots can be created. Includes information about the driver, secrets for accessing snapshots, etc.

It is important to note that - unlike the main Persistent Volume objects in Kubernetes, these snapshots objects are defined as CustomResourceDefinitions (CRDs) . The Kubernetes project is gradually moving away from the previously defined types of resources in the API Server, approaching the model in which the API Server does not depend on API objects. This approach allows you to reuse the server API in other projects (besides Kubernetes), and consumer'y (like the same Kubernetes) can set the types of resources they need as CRD.

CSI drivers supporting snapshots will automatically install the necessary CRDs. The end users of Kubernetes will only need to verify that the CSI driver supporting snapshots is deployed in a cluster.

In addition to these new facilities, an existingPersistentVolumeClaima new field has appeared DataSource:

type PersistentVolumeClaimSpec struct {
	AccessModes []PersistentVolumeAccessMode
	Selector *metav1.LabelSelector
	Resources ResourceRequirements
	VolumeName string
	StorageClassName *string
	VolumeMode *PersistentVolumeMode
	DataSource *TypedLocalObjectReference
}

This field (in the status of alpha version) allows you to automatically fill it with data from an existing snapshot when creating a new volume.

Kubernetes snapshots requirements

Before using snapshots of volumes in Kubernetes, you must:

make sure that the CSI driver that implements snapshots is deployed and running on the cluster;
enable Kubernetes Volume Snapshotting via a new feature gate (disabled by default for the alpha version):
- set the following flag for the executable API Server file: --feature-gates=VolumeSnapshotDataSource=true

Before creating a snapshot, you must also determine which CSI driver to use, which is done by creating an object VolumeSnapshotClassand specifying the CSI driver in the field snapshotter. In the example below, with VolumeSnapshotClassthis driver is com.example.csi-driver. Each supplier of snapshots requires at least one object VolumeSnapshotClass. It is also possible to define VolumeSnapshotClassthe default for each CSI driver - this is done by setting the annotation snapshot.storage.kubernetes.io/is-default-class: "true"in the class definition:

apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotClass
metadata:
  name: default-snapclass
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
snapshotter: com.example.csi-driver
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotClass
metadata:
  name: csi-snapclass
snapshotter: com.example.csi-driver
parameters:
  fakeSnapshotOption: foo
  csiSnapshotterSecretName: csi-secret
  csiSnapshotterSecretNamespace: csi-namespace

All required parameters must be set in accordance with the CSI driver documentation. In the example above, the CSI driver during the creation and deletion of the snapshot will be passed the parameter fakeSnapshotOption: fooand all the mentioned secrets. CSI external-snapshotter defaults to the parameter keys csiSnapshotterSecretNameand csiSnapshotterSecretNamespace.

Finally, before creating a snapshot, you need to create a volume through the CSI driver and fill it with the data you want to see there (see this publication for details on how to use CSI volumes).

Creating a new snapshot in Kubernetes

Once an object is VolumeSnapshotClassdefined and there is a volume from which you want to take a snapshot, you can perform this operation by creating an object VolumeSnapshot.

The source for snapshot is determined by two parameters:

kind- is indicated here PersistentVolumeClaim;
name - the actual name of the PVC object.

It is understood that the namespace of the volume for which the snapshot is created is determined by the object namespace VolumeSnapshot.

apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
  name: new-snapshot-demo
  namespace: demo-namespace
spec:
  snapshotClassName: csi-snapclass
  source:
    name: mypvc
    kind: PersistentVolumeClaim

The specification VolumeSnapshotcan be defined VolumeSnapshotClass, which contains information about which CSI driver will be used to create snapshots. As previously reported, after the object has been created, the VolumeSnapshotparameter fakeSnapshotOption: fooand all the secrets mentioned VolumeSnapshotClassare transferred to the CSI plugin com.example.csi-driverin the call CreateSnapshot.

In response to such a request, the CSI driver takes a snapshot of the volume and then automatically creates an object VolumeSnapshotContentthat represents the new snapshot, and binds this object to VolumeSnapshot, making it ready for use. If the CSI driver cannot create snapshots and returns an error, the snapshot controller reports this error in the object status VolumeSnapshotand does notmakes new attempts (this behavior differs from other controllers in Kubernetes - it is implemented in order not to create snapshots in unpredictable times).

If the snapshot class is not set, external-snapshotter will try to find the default class and use it for the snapshot being created. At the same time, the CSI driver pointed snapshotterto in the default class must correspond to the CSI driver pointed to provisionerin the PVC storage class.

Please note that the alpha release snapshots for Kubernetes does not guarantee consistency. To ensure complete data in the snapshot, it is necessary to prepare the application accordingly (stop the application, freeze the file system, etc.) before removing it.

Make sure that the objectVolumeSnapshotCreated and linked to VolumeSnapshotContent, you can command kubectl describe volumesnapshot:

Readymust be set trueto Status, which will indicate the readiness of the snapshot volume for use.
The field Creation Timeindicates when the snapshot was actually made.
Field Restore Size- the minimum size of the volume to restore snapshot.
The field Snapshot Content Namein the specification points to the object VolumeSnapshotContentcreated for this snapshot.

Importing an existing snapshot into Kubernetes

You can import an existing snapshot into Kubernetes by manually creating an object VolumeSnapshotContentthat will represent this snapshot. Since it VolumeSnapshotContentis an API object that is not tied to a namespace, only a system administrator has the right to create it.

When the object is VolumeSnapshotContentcreated, the user can create another object - VolumeSnapshot, - which will point to it. The external-snapshotter controller will mark the snapshot as ready after checking for the existence and correctness of the connection between objects VolumeSnapshotand VolumeSnapshotContent. Snapshot is ready for use in Kubernetes when this connection is established.

The object VolumeSnapshotContentmust be created with the following fields representing the pre-provisioned snapshot:

csiVolumeSnapshotSource - snapshot identification information:
- snapshotHandle- name / identifier for snapshot. Obligatory field;
- driver- CSI driver used to work with this volume. Obligatory field. Must match the name snapshotterin the controller (snapshot controller);
- creationTimeand restoreSize- for pre-prepared volumes, these fields are optional. The external-snapshotter controller will automatically update them after creating the snapshot.
volumeSnapshotRef- pointer to the object VolumeSnapshotto which this object (i.e. VolumeSnapshotContent) should be tied:
- nameand namespace- the name and namespace of the object VolumeSnapshot, the contents of which is bound;
- UID- optional (for pre-prepared volumes) field. The external-snapshotter controller will automatically update this field after binding. If the user specifies this field, you need to make sure that it corresponds to the UID of the snapshot for which the binding occurs. If there is no match, the content is considered irrelevant (orphan-object) and therefore the controller will delete both it and its associated snapshot.
snapshotClassName- optional field. The external-snapshotter controller will automatically update it after binding.

apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotContent
metadata:
  name: static-snapshot-content
spec:
  csiVolumeSnapshotSource:
    driver: com.example.csi-driver
    snapshotHandle: snapshotcontent-example-id
  volumeSnapshotRef:
    kind: VolumeSnapshot
    name: static-snapshot-demo
    namespace: demo-namespace

The object VolumeSnapshotmust be created so that the user can work with snapshots. In him:

snapshotClassName- class name snapshot volume. Optional field. If set, the field snapshotterin the snapshot class must match the name of the snapshot controller. If not set, the controller will search for the default snapshot class;
snapshotContentName- the name of the contents of the snapshot volume. Required field for pre-prepared volumes.

apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
  name: static-snapshot-demo
  namespace: demo-namespace
spec:
  snapshotClassName: csi-snapclass
  snapshotContentName: static-snapshot-content

When these objects are created, the snapshot controller will link them, set the field Ready(c Status) as True, indicating that the snapshot is ready for use.

Preparing a new snapshot volume in Kubernetes

To create a new volume that is pre-filled with data from the snapshot object, use the new field dataSourcein PersistentVolumeClaim. It has three parameters:

name- the name of the object VolumeSnapshotrepresenting the snapshot source;
kind- must be specified as VolumeSnapshot;
apiGroup- should be snapshot.storage.k8s.io.

It is assumed that the namespace of the source object VolumeSnapshotcorresponds to the namespace of the object PersistentVolumeClaim.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-restore
  Namespace: demo-namespace
spec:
  storageClassName: csi-storageclass
  dataSource:
    name: new-snapshot-demo
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

When the object PersistentVolumeClaimis created, it will cause the provisioning of a new volume, pre-filled with data from the specified snapshot.

How can I add snapshots support to my CSI driver if I am a storage developer?

To support snapshots in the CSI-driver additional capabilities of the controller should be added: CREATE_DELETE_SNAPSHOTand LIST_SNAPSHOTS- as well as implemented additional RPC Controller: CreateSnapshot, DeleteSnapshot, ListSnapshots. See the CSI specification for details .

Although Kubernetes provides the most minimal guidelines for packaging and deploying the CSI Volume Driver, there is a recommended mechanism for deploying an arbitrary containerized CSI driver to Kubernetes to simplify this process.

As part of the recommended deployment process, the Kubernetes team suggests using multiple sidecar- (i.e., ancillary) containers, including a sidecar-container with an external-snapshotter .

Said external-snapshotter in API Server monitors the objects VolumeSnapshotand VolumeSnapshotContentcausing operation CreateSnapshotand DeleteSnapshotfor CSI endpoint. The sidecar-container with CSI external-provisioner has also been updated to support recovery of the volume from snapshot using a new field dataSourcein the PVC.

To support snapshot capabilities, storage vendors are recommended to deploy sidecar containers with an external-snapshotter in addition to an external provisioner, and place the CSI driver in StatefulSet, as shown in the diagram below:

In this example, DeploymentThere are two sidecar containers, the external provisioner and an external snapshotter, and the CSI drivers are included with the CSI hostpath plugin within the StatefulSet pod. The CSI hostpath is an example of a plugin that is not intended for use in production.

What are the limitations of the alpha version?

The alpha version of the implementation of snapshots in Kubernetes has the following limitations:

It does not support rollback of the existing volume to the previous state represented by snapshot (only provisioning of the new volume from snapshot is supported).
The in-place restore of the existing PersistentVolumeClaimsnapshot is not supported : i.e. The provisioning of a new volume from snapshot works, but not updating the existing one PersistentVolumeClaimso that it points to a new volume and the PVC rolls back to an earlier state (only the use of a new volume created from snapshot through a new PV / PVC is supported).
The snapshot consistency guarantees do not go beyond the guarantees provided by the storage system (for example, integrity in a fall).

What's next?

The Kubernetes team plans to bring the implementation of snapshots for CSI to a beta version in releases 1.13 or 1.14, depending on the feedback received and the adaptation of the technology.

How to find out more details?

See additional snapshots documentation on k8s.io/docs/concepts/storage/volume-snapshots and kubernetes-csi.github.io/docs .