3 Using Oracle Cloud Infrastructure Storage
The Oracle Cloud Infrastructure Cloud Controller Manager module implements a Container Storage Interface (CSI) plugin for Kubernetes clusters that provides you with the following storage services for creating persistent data storage:
- Oracle Cloud Infrastructure Block Volume service
-
The Oracle Cloud Infrastructure Block Volume service provides dynamically provisioned and managed block storage volumes. The block storage volumes can be created in sizes ranging from 50 GB to 32 TB.
After you attach a volume to a Kubernetes application, the volume can be used in a similar way to a regular hard drive. The volume can also be disconnected and attached to another instance without the loss of data.
Block Volume volumes are automatically replicated to help protect against data loss.
To read about example use cases, and for more information about the Oracle Cloud Infrastructure Block Volume service, see the Oracle Cloud Infrastructure documentation.
- Oracle Cloud Infrastructure File Storage service
-
The Oracle Cloud Infrastructure File Storage service provides a scalable and durable enterprise-grade network file system. The File Storage service uses the Network File System version 3.0 (NFSv3) protocol. The service uses the Network Lock Manager (NLM) protocol for file locking functionality.
The File Storage service transparently manages storage provisioning as data storage needs increase from the order of bytes to exabytes.
Large compute clusters of thousands of instances can use the File Storage service for high-performance shared storage.
Oracle Cloud Infrastructure File Storage uses 5-way replicated storage, in different fault domains, to provide redundancy for resilient data protection.
To read about example use cases, and for more information about the Oracle Cloud Infrastructure File Storage service, see the Oracle Cloud Infrastructure documentation
The following sections provide examples to illustrate how to use each of the storage services to set up persistent storage for Kubernetes applications in Oracle Cloud Native Environment on Oracle Cloud Infrastructure instances.
Creating Block Storage
This section contains a basic test to verify you can create Oracle Cloud Infrastructure block storage to provide persistent storage to applications running on Kubernetes.
To create a test application to use Oracle Cloud Infrastructure block storage:
-
Create a Kubernetes PersistentVolumeClaim file. On a control plane node, create a file named
pvc.yaml. Copy the following into the file.apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myoci-pvc spec: accessModes: - ReadWriteOnce storageClassName: oci-bv resources: requests: storage: 50GiNote that the
accessModessetting for Oracle Cloud Infrastructure storage must beReadWriteOnce. The minimum Oracle Cloud Infrastructure block size is 50Gi. -
Create the Kubernetes PersistentVolumeClaim.
kubectl apply -f pvc.yaml -
You can see the PersistentVolumeClaim is created using the
kubectl get pvccommand:kubectl get pvcThe output looks similar to:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myoci-pvc Pending oci-bv 15sThe
STATUSisPendingand means the claim is waiting for an application to claim it.You can get more details about the PersistentVolumeClaim using the
kubectl describe pvccommand. For example:kubectl describe pvc myoci-pvcThe output looks similar to:
Name: myoci-pvc Namespace: default StorageClass: oci-bv Status: Pending Volume: Labels: <none> Annotations: <none> Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Used By: <none> Events: Type Reason Age From ... ---- ------ ---- ---- Normal WaitForFirstConsumer 2m18s (x26 over 8m29s) persistentvolume-controller ... -
Create a Kubernetes application that uses the PersistentVolumeClaim. Create a file named
nginx.yamland copy the following into the file.apiVersion: apps/v1 kind: Deployment metadata: labels: run: mynginx name: mynginx spec: replicas: 1 selector: matchLabels: run: mynginx template: metadata: labels: run: mynginx spec: containers: - image: container-registry.oracle.com/olcne/nginx:1.17.7 name: mynginx ports: - containerPort: 80 volumeMounts: - name: nginx-pvc mountPath: /usr/share/nginx/html volumes: - name: nginx-pvc persistentVolumeClaim: claimName: myoci-pvc -
Start the application:
kubectl apply -f nginx.yaml -
You can see the application is running using the
kubectl get deploymentcommand:kubectl get deploymentThe output looks similar to:
NAME READY UP-TO-DATE AVAILABLE AGE mynginx 1/1 1 1 63s -
You can see the application is using the PersistentVolumeClaim to provide persistent storage on Oracle Cloud Infrastructure using the
kubectl describe deploymentcommand:kubectl describe deployment mynginxThe output looks similar to:
... Pod Template: Labels: run=mynginx Containers: mynginx: Image: container-registry.oracle.com/olcne/nginx:1.17.7 Port: 80/TCP Host Port: 0/TCP Environment: <none> Mounts: /usr/share/nginx/html from nginx-pvc (rw) Volumes: nginx-pvc: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: myoci-pvc ReadOnly: false ...Note the
ClaimNameismyoci-pvc, which is the name of the PersistentVolumeClaim created earlier.You can see the PersistentVolumeClaim is now bound to this application using the
kubectl get pvccommand:kubectl get pvcThe output looks similar to:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myoci-pvc Bound csi-84175067-... 50Gi RWO oci-bv 1m
Tip:
If you sign in to Oracle Cloud Infrastructure, you can see a block volume created with the name listed in the
VOLUMEcolumn. The block volume is attached to the compute instance on which the Kubernetes application is running. -
You can delete the test application using:
kubectl delete deployment mynginx -
You can delete the PersistentVolumeClaim using:
kubectl delete pvc myoci-pvcThe storage is deleted.
Tip:
If you sign in to Oracle Cloud Infrastructure, you can see the block volume is terminated.
Creating File Storage
This section contains a basic example to verify you can use an Oracle Cloud Infrastructure File Storage file system to provide persistent file storage to applications running on Kubernetes. The example involves:
-
Creating a File Storage file system in Oracle Cloud Infrastructure.
-
Using YAML files to create a PersistentVolume and a PersistentVolumeClaim for the File Storage file system.
-
Using a YAML file to create a pod with a mount to the volume to write a test file in the File Storage file system.
Setting up a File System
Create a File Storage file system in a Virtual Cloud Network (VCN) in Oracle Cloud Infrastructure.
Tip:
For a step-by-step guide on provisioning a File System in Oracle Cloud Infrastructure, see the Oracle Luna Lab Provision Persistent Volumes Using File Storage Service on Oracle Cloud Native Environment.To create a file system:
-
Configure a VCN for the File Storage file system. The precise configuration of the VCN varies. For example scenarios, see the Oracle Cloud Infrastructure documentation.
-
Create a File Storage file system. For more information, see the Oracle Cloud Infrastructure documentation.
-
Create a File Storage mount target to enable network access to the file system. For more information, see the Oracle Cloud Infrastructure documentation.
-
In Oracle Cloud Infrastructure, find and make a note of the following file system attributes. These are needed when you create YAML files or mount the file system in this example:
-
The OCID of the file system.
-
The export path of the file system. This example assumes you're setting the export path to
/my-fss-export. -
The mount commands. These commands are provided on the Mount Commands page of the File System. You use these commands later in the example when mounting the exported file system.
-
The IP address of the mount target.
-
Using a File System
To create a test application to use an Oracle Cloud Infrastructure File Storage file system:
-
Create a PersistentVolume file. On a control plane node, create a file called
fss-pv.yamlwith the following contents:apiVersion: v1 kind: PersistentVolume metadata: name: fss-pv spec: capacity: storage: 50Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain csi: driver: fss.csi.oraclecloud.com volumeHandle: <filesystem_OCID>:<mount_target_IP>:<mount_target_export_path>The
accessModesfor the file system must beReadWriteMany.The
storageoption is a Kubernetes requirement and must be included. The File Storage service ignores this value and creates a new file system with a default size, regardless of the value you specify forstorage.The
volumeHandleconsists of a colon separated list of File Storage file system attributes, noted in an earlier step. For example:ocid1.filesystem.oc1.iad.aaaa...:10.0.0.200:/my-fss-export -
Create the PersistentVolume.
kubectl apply -f fss-pv.yaml -
Get a list of the PersistentVolumes.
kubectl get pvThe output looks similar to:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE fss-pv 50Gi RWX Retain Available 47s -
Create a PersistentVolumeClaim file. Create a file called
fss-pvc.yaml. Copy the following into the file:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: fss-pvc spec: accessModes: - ReadWriteMany storageClassName: "" resources: requests: storage: 5Gi volumeName: fss-pvThe
accessModesfor the file system must beReadWriteMany.The
storageoption is a Kubernetes requirement and must be included. The File Storage service ignores this value and creates a new file system with a default size, regardless of the value you specify forstorage.The
volumeNameis set to the name attribute in the PersistentVolume filefss-pv.yaml. -
Create the PersistentVolumeClaim.
kubectl apply -f fss-pvc.yaml -
Get a list of PersistentVolumeClaims.
kubectl get pvcThe output looks similar to:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE fss-pvc Bound fss-pv 50Gi RWX 33s - Create a pod manifest file. Create a file called
fss-pod.yaml. Copy the following into the file:apiVersion: v1 kind: Pod metadata: name: app spec: containers: - name: app image: container-registry.oracle.com/os/oraclelinux:9-slim command: ["/bin/sh"] args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"] volumeMounts: - name: persistent-storage mountPath: /data volumes: - name: persistent-storage persistentVolumeClaim: claimName: fss-pvcThe
imageoption specifies the registry location to an Oracle Linux container image.The
mountPathoption specifies the directory mounted to the persistent storage.The
commandoption specifies a command to write to fileout.txtin the/datadirectory. -
Create the pod.
kubectl apply -f fss-pod.yaml -
Get a list of the pods.
kubectl get podsThe output looks similar to:
NAME READY STATUS RESTARTS AGE app 1/1 Running 0 21s -
Open a shell to the container within the pod:
kubectl exec -i -t app --container app -- /bin/bash -
From the container's shell, confirm the container is writing to the
/data/out.txtfile by using thetailcommand:tail -f /data/out.txtThe date and time output of the
date -ucommand is listed to the terminal and written to the/data/out.txtfile on the container. Exit thetailprogram by usingCTRL+C. -
Exit the container shell using the
exitcommand:exit -
Verify that the
/datadirectory within the container is mounted to the file system export path. This step uses the mount commands noted when you created the file system.-
On the control plane node, install the NFS client.
sudo dnf install nfs-utils -
Create and mount a local directory to the file system's export path.
sudo mkdir -p /mnt/my-fss-export sudo mount 10.0.0.200:/my-fss-export /mnt/my-fss-exportReplace
10.0.0.200with the IP address of the mount target. -
Confirm the
output.txtfile the container is writing to is in the file system's export path. You can use thetailcommand for this:sudo tail -f /mnt/my-fss-export/out.txtThe date and time being written to the file by the pod is listed. Exit the
tailprogram by usingCTRL+C.
-
-
Delete the pod. If you don't see the command prompt after the pod is deleted you can use the
CTRL+Ckey combination to get it back:kubectl delete pod app -
Delete the PersistentVolumeClaim:
kubectl delete pvc fss-pvc -
Confirm that the status of the PersistentVolume is
Released:kubectl get pvThe output looks similar to:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE fss-pv 50Gi RWX Retain Released default/fss-pvc 57m -
Delete the PersistentVolume:
kubectl delete pv fss-pv -
Confirm the file is still present:
ls -l /mnt/my-fss-export/out.txt -
Remove the mount:
sudo umount /mnt/my-fss-export -
Confirm the
out.txtfile is no longer available and the mount is removed:ls -l /mnt/my-fss-export -
Confirm the
/mnt/my-fss-exportis no longer mounted:mount | grep my-fss-export