A workload on Kubernetes typically requires two types of storage:
-
Ephemeral Storage
-
Persistent Volume Storage
Ephemeral storage
Ephemeral storage, by its name, is ephemeral in the sense that it is cleaned up when the workload is deleted or the container crashes. For example, the following are examples of ephemeral storage provided by Kubernetes:
EmptyDir volume. | Managed by kubelet under /var/lib/kubelet. |
Container logs. | Typically under /var/logs/containers. |
Container image layers. | Managed by container runtime (e.g., under /var/lib/containerd). |
Container writable layers. | Managed by container runtime (e.g., under /var/lib/containerd). |
Ephemeral storage is automatically managed by Kubernetes, and typically does not require explicit settings. You may need to express the capacity requests for ephemeral storage so that kubelet
can use that information to make sure it does not run out of ephemeral storage space on each node.
Persistent Volume
Persistent Volumes are storage resources that can be used by the cluster. Persistent Volumes are volume plug-ins that have lifecycle capabilities that are independent of any Kubernetes Pod or Deployment.
You may have stateful workloads requiring persistent storage whose lifecycle is longer than that of Pods or containers. For instance, a database server needs to recover database files after it crashes. For those cases, the workloads need to use PersistentVolumes (PV).
Persistent Volumes are resources that represent storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. Unlike ephemeral storage, the lifecycle of a PersistentVolume is independent of that of the workload that uses it.
The Persistent Volume API objects capture the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system. In order to use a Persistent Volume, your application needs to invoke a Persistent Volume Claim.
Create a Persistent Volume
Create a Persistent Volume using NFS as an example.
Requirements: This procedure assumes you have access to an NFS shared storage in your environment and accessible to your cluster.
-
name: The name of the persistent volume you want.
-
server: Your FQDN server name or IP of NFS Server.
-
path: Path to your NFS server volume.
-
Create a file called
nfs-share.yaml
.apiVersion: v1 kind: PersistentVolume metadata: name: nfs-share namespace: default labels: storage: nfs spec: capacity: storage: 5Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain nfs: server: 192.168.86.252 path: /volume1/nfs-01/nfs-share
-
Apply the file to create an NFS Persistent Volume (PV).
kubectl apply -f nfs-share.yaml
Once that is deployed, validate that the status is available. You should receive a return value of
persistentvolume/nfs-share created.
-
Validate that the Persistent Volume (PV) is available.
kubectl get pv nfs-share
This is the output that shows that the cluster has accepted your Persistent Volume and is in status available:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE nfs-share 5Gi RWX Retain Bound
Your Persistent Volume is now available for consumption. Next, create a Persistent Volume Claim so your Pod can use the storage.
Persistent Volume Claim
A PersistentVolumeClaim is a request for storage. For a workload that requires persistent volumes, the workload should use PersistentVolumeClaim (PVC) to express its request on persistent storage. A PersistentVolumeClaim can request specific size and Access Modes (for example, they can be mounted once read/write or many times read-only).
Any workload can specify a PersistentVolumeClaim. For example, a Pod may need a volume that is at least 4Gi large or a volume mounted under /data
in the container’s filesystem. If there is a PersistentVolume (PV) that satisfies the specified requirements in the PersistentVolumeClaim (PVC), it will be bound to the PVC before the Pod starts.
Create a Persistent Volume Claim to be used with your Pod
Create a PersistentVolumeClaim (PVC) and leverage the existing PersistentVolume (PV) that was created in the previous example under Persistent Volumes. Remember that a Pod must use the PVC to invoke the use of a PV.
Requirements: This assumes you created a Persistent Volume as outlined in the Persistent Volume section.
-
name: The name of the persistent volume claim you want.
-
storage: The size of your storage claim. This must not exceed the persistent volume capacity. In the example above for Persistent Volume we used 5Gi.
-
matchLabels: This must match the Persistent Volume
labels:
in the (PV)
-
Create a file called
nfs-share.yaml
.apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nfs-share namespace: default spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi storageClassName: "" selector: matchLabels: storage: "nfs"
-
After configuration, invoke the NFS Volume as a Persistent Volume Claim (PVC).
kubectl apply -f nfs-share.yaml
-
After applying the file, validate the status of the PVC:
kubectl get pv nfs-share
You should receive a return value of
persistentvolumeclaim/nfs-share created
. After that is deployed, validate that the status isPending
. You should receive a return value ofpersistentvolume/nfs-share created
.
Configure your Workload to use the Persistent Volume Claim
Your Persistent Volume Claim is Pending
because no workload has claimed it. Next, create an example workload that will claim and use the Persistent Volume Claim (PVC). We will also validate that the workload can access the volume.
-
Create a file called
nfs-app.yaml
.- mountPath: This is the path in the container that will map to your NFS Share. You can change this to any path you want in your container.
kind: Pod apiVersion: v1 metadata: name: pod-nfs spec: containers: - name: nfs-app image: alpine volumeMounts: - name: data mountPath: /var/nfs command: ["/bin/sh"] args: ["-c", "sleep 500000"] volumes: - name: data persistentVolumeClaim: claimName: nfs-pvc
-
Next apply the file.
kubectl apply -f nfs-app.yaml
After the file is deployed, you should receive a return value of
pod/pod-nfs created
. -
Enter the following command to ensure that it is fully deployed.
kubectl get pod pod-nfs
When the container is fully running and the
READY STATUS
isRunning
, you should see output similar to the following:NAME READY STATUS RESTARTS AGE pod-nfs 1/1 Running 0 2m27s
-
Use the following command to validate that the Persistent Volume Claim (PVC) has mounted the volume to your container
pod-nfs
:kubectl describe pod pod-nfs
Here you can see under the describe conditions that the Persistent Volume Claim has been mounted to your container under the
ClaimName: nfs-pvc
which is the (PVC).Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: nfs-pvc ReadOnly: false
-
Enter the this command to validate that inside the container we can access the volume and write data:
kubectl exec -it pod-nfs sh
-
This is where the
mountPath: /var/nfs
was labeled in thepod-nfs.yaml
. If you change the valuecd to your path
.cd /var/nfs
If you were able to access your
mountPath
you have successfully mounted the NFS volume to your container. Now try and write a file into the volume. -
Enter this command to create a file in the directory and list all files in that same directory.
touch nfs.txt ls
You should see a file called nfs.txt
. If you do, you have a fully functional NFS volume accessible to your container.
Storage Classes
Container Storage Interface (CSI)
Storage Method | Persistent Volume | Persistent Volume Claim |
NFS | Create NFS PV | Create NFS PVC |
Open-ebs | Create open-ebs PV | Create open-ebs PVC |