Manage Ephemeral Storage on Kubernetes

Kubernetes pods require ephemeral (temporary) local storage.

Kubernetes pods use ephemeral storage for scratch space, caching, and logs. This storage is temporary and specific to the life cycle of the pod. Ephemeral storage is not shared across pods and it goes away when the pod is deleted.

For more general information about Kubernetes ephemeral storage, see Local ephemeral storage in the Kubernetes documentation.

The following steps apply to Oracle Kubernetes Engine, but the concepts are similar in other Kubernetes environments.

In a node pool, the nodes use their boot volumes for pod storage. Because images are stored in the /var directory, most of the ephemeral storage is occupied by images in the root partition. The space needed in the boot volume increases every time that you install an instance of Oracle Blockchain Platform Enterprise Edition and create chaincodes on that node.

You can use the following kubelet API call to check the total ephemeral storage and the amount to assigned to each pod in a given node:

kubectl get --raw "/api/v1/nodes/<node IP>/proxy/stats/summary"

The rootfs and fs sections of the JSON result show the capacity and the available bytes.

You can update the amount of ephemeral storage by resizing the boot volume while a node is running. For more information, see Updating a Node Pool for Oracle Kubernetes Engine.

Complete the following steps to resize the ephemeral storage on an Oracle Kubernetes Engine cluster with running nodes.

On your Oracle Kubernetes Engine cluster, under Resources, select Node pools.
Click Edit. On the Edit node pool page, select Specify a custom boot volume size and then enter a Boot volume size value in GB. Any nodes that are created will use this value for ephemeral storage.
For each worker node in the node pool, complete the following steps to resize the boot volume.
1. Click the down arrow beside the node to see detailed information about the node.
2. Navigate to the Boot volume for the node and then click Edit.
3. On the Edit volume page under Volume size and performance, specify a Volume size value in GB and then click Save changes.
Complete the following steps to set up a Bastion session and then use it to connect to the private worker nodes.
1. On the instance details page, click the Oracle Cloud Agent tab, and then enable the Bastion plugin.
2. Search for bastion in the search bar, and then click Bastion Identity & Security under Services in the results.
3. Click Create bastion.
4. On the Create bastion page, for the Target virtual cloud network (VCN) specify the Oracle Kubernetes Engine VCN followed by the cluster name. For Target subnet, specify the Kubernetes API endpoint. For CIDR block allowlist, enter 0.0.0.0/0, and then click Create bastion.
5. Click the bastion to open it, and then click Create session.
6. Enter opc for the Username value and select your node from the Compute instance list.
7. Paste your SSH key under Add SSH Key.
8. Click Show advanced options and then select the IP address of the node or instance from the Target compute instance IP address list. This is the private IPv4 address of the node or instance, which is available in the information section for the instance.
9. Click Create session.
10. From the context menu for the session, click Copy SSH command.
11. You can now log in to the node via SSH by providing your private key with the -i parameter in the SSH command.
12. Repeat the previous steps for each worker node in the cluster.
For each node, log in to the node via SSH and then run the following commands, which scan for new block storage devices added to instances or nodes, and then expand the file system when storage is available.
```
sudo dd iflag=direct if=/dev/oracleoci/oraclevda of=/dev/null count=1
echo "1" | sudo tee /sys/class/block/`readlink /dev/oracleoci/oraclevda | cut -d'/' -f 2`/device/rescan
sudo /usr/libexec/oci-growfs -y
```

Determining Ephemeral Storage Usage

You can run the following script to see the ephemeral storage usage of an instance running on Oracle Kubernetes Engine. The script uses the Kubernetes API to retrieve ephemeral storage usage for each pod that is running on each node in the cluster.

#!/usr/bin/env bash
  
kubectl proxy --append-server-path &
  
set -eo pipefail
  
{
    echo "NODE NAMESPACE POD EPHEMRAL_USED"
    for node in $(kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
        curl -fsSL "http://127.0.0.1:8001/api/v1/nodes/$node/proxy/stats/summary" |
            yq '.pods[] | [.podRef.namespace, .podRef.name, .ephemeral-storage.usedBytes] | join(" ")' |
            while read -r namespace name usedBytes; do
                # A pod might have no running containers and consequently no ephemeral-storage usage.
                echo "$node" "$namespace" "$name" "$(numfmt --to iec "${usedBytes:-0}")"
            done
    done | sort -k4,4rh
} | column -t