Scale an Oracle Cloud Native Environment Cluster

Introduction

Scaling up a Kubernetes cluster means adding nodes; likewise, scaling down occurs by removing nodes. These nodes can be either control plane or worker nodes. Oracle recommends against scaling the cluster up and down simultaneously but instead performing a scale up and down in two separate commands.

Scaling the Kubernetes cluster control plane or worker nodes in odd numbers is recommended to avoid split-brain scenarios and maintain the quorum. For example, 3, 5, or 7 control plane nodes or worker nodes ensure the cluster’s reliability.

Objectives

In this tutorial, you’ll learn how to use kubectl scale and kubectl edit to:

• Add a new control plane and worker nodes to a Kubernetes cluster
• Remove control plane nodes and worker nodes from a Kubernetes cluster

Prerequisites

• Minimum of one Oracle Linux instance

  • Runs ocne and libvirt ephemeral cluster

• OCI cluster creation requires access to the following resources in an Oracle Cloud Infrastructure tenancy:

  • Virtual cloud network with four subnets
  • Network load balancer
  • Object Storage bucket with minimum 5 GiB available
  • Compute Custom Image
  • Compute Arm Shape for the control plane node
    • VM.Standard.A1.Flex with two OCPU and 12 memory
  • Compute for each additional control plane and worker node
    • VM.Standard.E4.Flex with four OCPU and 64 memory

Configure Oracle Cloud Native Environment

Note: If running in your tenancy, read the linux-virt-labs GitHub project README.md and complete the prerequisites before deploying the lab environment.

1. Open a terminal on the Luna Desktop.

2. Clone the linux-virt-labs GitHub project.

   git clone https://github.com/oracle-devrel/linux-virt-labs.git
   

3. Change into the working directory.

   cd linux-virt-labs/ocne2
   

4. Install the required collections.

   ansible-galaxy collection install -r requirements.yml
   

5. Increase the Boot volume size.

   cat << EOF | tee instances.yml > /dev/null
   compute_instances:
     1:
       instance_name: "ocne"
       type: "server"
       boot_volume_size_in_gbs: 128
   install_ocne_rpm: true
   install_type: "oci"
   ocne_cluster_name: "mycluster"
   create_ocne_oci_cluster: true
   EOF
   

6. Deploy the lab environment.

   ansible-playbook create_instance.yml -e localhost_python_interpreter="/usr/bin/python3.6" -e "@instances.yml"
   

   The free lab environment requires the extra variable local_python_interpreter, which sets ansible_python_interpreter for plays running on localhost. This variable is needed because the environment installs the RPM package for the Oracle Cloud Infrastructure SDK for Python, located under the python3.6 modules.

   The default deployment shape uses the AMD CPU and Oracle Linux 8. To use an Intel CPU or Oracle Linux 9, add -e instance_shape="VM.Standard3.Flex" or -e os_version="9" to the deployment command.

   Important: Wait for the playbook to run successfully and reach the pause task. At this stage of the playbook, the installation of the Oracle Cloud Native Environment is complete, and the instances are ready. Take note of the previous play, which prints the public and private IP addresses of the nodes it deploys and any other deployment information needed while running the lab.

Confirm the Cluster is Running

1. Open a terminal and connect via SSH to the ocne instance.

   ssh oracle@<ip_address_of_node>
   

2. Get a list of known clusters using the CLI.

   ocne cluster list
   

   Example Output:

   [oracle@ocne ~]$ ocne cluster list
   ocne-ephemeral
   mycluster
   

   The output shows two clusters are present, the one you just created (mycluster) and the bootstrap cluster (ocne-ephemeral).

3. Get the location of the kube configuration.

   ocne cluster show -C mycluster
   

   We use the -C to specify a specific cluster from the cluster list.

4. Set the KUBECONFIG environment variable.

   export KUBECONFIG=$(ocne cluster show -C mycluster)
   

5. Wait for the cluster to stabilize and all nodes to report in a ready state.

   watch kubectl get nodes -A
   

   Example Output:

   Every 2.0s: kubectl get nodes -A                               ocne: Wed Nov 13 16:40:08 2024
   
   NAME                            STATUS   ROLES           AGE     VERSION
   mycluster-control-plane-8dq98   Ready    control-plane   11m     v1.30.3+1.el8
   mycluster-control-plane-dkscq   Ready    control-plane   8m      v1.30.3+1.el8
   mycluster-control-plane-jr9gz   Ready    control-plane   9m      v1.30.3+1.el8
   mycluster-md-0-8jr9k-8gnvw	    Ready    <none>          9m      v1.30.3+1.el8
   mycluster-md-0-8jr9k-fn9mk	    Ready    <none>          9m      v1.30.3+1.el8
   mycluster-md-0-8jr9k-kr9p6	    Ready    <none>          9m      v1.30.3+1.el8
   

   Wait for the three worker and control plane nodes to show a STATUS of Ready, then type ctrl-c to exit the watch command.

Use the Scale Command to Scale the Control Plane and Worker Nodes

Kubernetes administrators typically use the kubectl scale command to scale resources deployed onto your Kubernetes cluster (e.g.: Deployments, StatefulSets, ReplicaSets, etc.). However, it can also scale the number of control and worker nodes in your Kubernetes cluster.

Scale Up the Control Plane and Worker Nodes

1. Confirm the bootstrap cluster is present.

   sudo virsh list
   

   Example Output:

   [oracle@ocne ~]$ sudo virsh list
    Id   Name                             State
   ------------------------------------------------
    1    ocne-ephemeral-control-plane-1   running
   

2. Set the kubectl context to reference the bootstrap cluster.

   export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local
   

3. Use the Cluster API Provider for OCI to confirm the number of nodes present.

   watch kubectl get ocimachine -n ocne
   

   Example Output:

   Every 2.0s: kubectl get ocimachine -n ocne                                ocne: Wed Dec 18 11:29:08 2024
   
   [oracle@ocne ~]$ kubectl get ocimachine -n ocne
   NAME                            AGE
   mycluster-control-plane-wl7n8   9m58s
   mycluster-control-plane-dkscq   12m
   mycluster-control-plane-jr9gz   15m
   mycluster-md-0-fvtvm-6nzwb      10m
   mycluster-md-0-fvtvm-qs4dl      10m
   mycluster-md-0-fvtvm-ttpl5      10m
   

   Wait until the output matches the cluster’s structure from the kubectl get nodes -A command. Then type ctrl-c to exit the watch command.

4. Get the name of the machinedeployments Custom Resource.

   kubectl get machinedeployments -n ocne
   

   Example Output:

   [oracle@ocne ~]$ kubectl get machinedeployments -n ocne
   NAME             CLUSTER     REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE    VERSION
   mycluster-md-0   mycluster   3          3       3         0             Running   10m    v1.30.3
   

   The NAME column shows the MachineDeployments, which reflect Kubernetes objects representing the worker nodes in the cluster.

5. Scale up a worker node from 3 to 4.

   kubectl scale machinedeployment mycluster-md-0 --replicas=4 --namespace ocne
   

6. Scale up the control plane nodes from 3 to 5.

   kubectl scale kubeadmcontrolplane mycluster-control-plane --replicas=5 --namespace ocne
   

7. Set the context back to the Workload cluster and confirm the new control plane nodes are present.

   export KUBECONFIG=$(ocne cluster show -C mycluster)
   watch kubectl get nodes -A
   

   Wait for this to complete, but check for the existence of five control plane nodes and four worker nodes. Once they show as running, type ctrl-c to exit the watch command.

Scale Down the Control Plane and Worker Nodes

1. Set the kubectl context to reference the bootstrap cluster.

   export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local
   

2. Scale down a worker node.

   kubectl scale machinedeployment mycluster-md-0 --replicas=3 --namespace ocne
   

3. Scale down the control plane nodes.

   kubectl scale kubeadmcontrolplane mycluster-control-plane --replicas=3 --namespace ocne
   

4. Set the context back to the Workload cluster and confirm the new control plane nodes are present.

   export KUBECONFIG=$(ocne cluster show -C mycluster)
   watch kubectl get nodes -A
   

   Wait for the cluster to stabilize and the scale down to complete by leaving only three worker and three control plane nodes running. Then type ctrl-c to exit the watch command.

Use the Edit Command to Scale the Control Plane and Worker Nodes

The kubectl edit command provides both Kubernetes administrators and developers with a way to apply real-time direct edits and modification to a deployed Kubernetes API configuration or resource. By default, it opens the referenced Kubernetes API resource in vi on Linux or notepad on Windows. To use your favorite editor, set the KUBE_EDITOR or EDITOR variables within the same shell as you run the edit command. However, it is generally not recommended for use on production systems to avoid altering something accidentally. It is, however, a flexible way to test changes in development or test environments.

You can use the edit command to accomplish the same scale up and down actions as the scale command performs.

Scale Up a Worker Node

1. Confirm the bootstrap cluster is present.

   sudo virsh list
   

   Example Output:

   [oracle@ocne ~]$ sudo virsh list
    Id   Name                             State
   ------------------------------------------------
    1    ocne-ephemeral-control-plane-1   running
   

2. Set the kubectl context to reference the bootstrap cluster.

   export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local
   

3. Use the Cluster API Provider for OCI to confirm the number of nodes present.

   kubectl get ocimachine -n ocne
   

   Example Output:

   [oracle@ocne ~]$ kubectl get ocimachine -n ocne
   NAME                            AGE
   mycluster-control-plane-dbrkt   5m45s
   mycluster-control-plane-nrlh9   4m13s
   mycluster-control-plane-wm7rg   8m34s
   mycluster-md-0-9cczf-kspf9      9m11s
   mycluster-md-0-9cczf-ngjgk      9m11s
   mycluster-md-0-9cczf-z62zl      9m11s
   

4. Get the name of the machinedeployments Custom Resource.

   kubectl get machinedeployments -n ocne
   

   Example Output:

   [oracle@ocne ~]$ kubectl get machinedeployments -n ocne
   NAME             CLUSTER     REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE    VERSION
   mycluster-md-0   mycluster   3          3       3         0             Running   10m    v1.30.3
   

5. Edit the Custom Resource.

   kubectl edit machinedeployments --namespace ocne mycluster-md-0
   

   This command opens the Custom Resource for editing.

6. Set the Replicas field to the number of the worker nodes you want to deploy.

   For this example, change the replicas: 3 field to: replicas: 4. Then save and exit. You enter the total number of worker nodes, and the cluster will scale up or down accordingly.

   Example Output:

   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be
   # reopened with the relevant failures.
   #
   apiVersion: cluster.x-k8s.io/v1beta1
   kind: MachineDeployment
   metadata:
     annotations:
   ...
   ...
   spec:
     clusterName: mycluster
     minReadySeconds: 0
     progressDeadlineSeconds: 600
     replicas: 3
     revisionHistoryLimit: 1
   ...
   ...
   

   Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

7. Confirm the creation of the new worker node.

   kubectl get ocimachine --namespace ocne
   

   Example Output:

   [oracle@ocne ~]$ kubectl get ocimachine --namespace ocne
   NAME                            AGE
   mycluster-control-plane-lj9m9   11m
   mycluster-control-plane-t6gc8   14m
   mycluster-control-plane-w427z   12m
   mycluster-md-0-fvtvm-bzfnc      8s
   mycluster-md-0-fvtvm-qs4dl      38m
   mycluster-md-0-fvtvm-ttpl5      38m
   mycluster-md-0-fvtvm-xf4ch      22m
   

8. Set the context back to the Workload cluster and confirm the new worker node is present.

   export KUBECONFIG=$(ocne cluster show -C mycluster)
   watch kubectl get nodes -A
   

   Wait for the new worker node to show a STATUS of Ready. Then type ctrl-c to exit the watch command.

Scale Up Control Plane Nodes

1. Set the kubectl context to reference the bootstrap cluster.

   export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local
   

2. Get the name of the kubeadmcontrolplane Custom Resource.

   kubectl get kubeadmcontrolplane -n ocne
   

   Example Output:

   [oracle@ocne ~]$ kubectl get kubeadmcontrolplane -n ocne
   NAME                      CLUSTER     INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
   mycluster-control-plane   mycluster   true          true                   3          3       3         0             13m   v1.30.3
   

3. Edit the Custom Resource.

   kubectl edit kubeadmcontrolplane --namespace ocne mycluster-control-plane 
   

4. Set the Replicas field to the new number of control plane nodes to deploy.

   Change the replicas: 3 field to: replicas: 5, then save and exit.

   Example Output:

   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be   
   # reopened with the relevant failures.
   #
   apiVersion: controlplane.cluster.x-k8s.io/v1beta1
   kind: KubeadmControlPlane
   metadata:
     creationTimestamp: "2024-11-05T17:03:52Z"
     finalizers:
     - kubeadm.controlplane.cluster.x-k8s.io
     generation: 3
     labels:
       cluster.x-k8s.io/cluster-name: mycluster
   ...
   ...
     machineTemplate:
      infrastructureRef:
         apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
         kind: OCIMachineTemplate
         name: mycluster-control-plane
         namespace: ocne
       metadata: {}
     replicas: 5
     rolloutStrategy:
   ...
   ...
   

   Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

   Important: The number of control plane nodes set must always be an odd number or the scale request will not be completed.

5. Confirm the new control plane nodes deployed.

   watch kubectl get ocimachine --namespace ocne
   

   Example Output:

   Every 2.0s: kubectl get ocimachine --namespace ocne                    ocne: Wed Dec 18 11:15:03 2024
   
   NAME                            AGE
   mycluster-control-plane-8wq8l   16s
   mycluster-control-plane-lj9m9   16m
   mycluster-control-plane-qzwmb   97s
   mycluster-control-plane-t6gc8   19m
   mycluster-control-plane-w427z   18m
   mycluster-md-0-fvtvm-bzfnc      5m14s
   mycluster-md-0-fvtvm-qs4dl      43m
   mycluster-md-0-fvtvm-ttpl5      43m
   mycluster-md-0-fvtvm-xf4ch      27m
   

   Note: It will take a few minutes for both of the new control plane nodes to show. Then type ctrl-c to exit the watch command.

6. Set the context back to the Workload cluster and confirm the new control plane nodes are present.

   export KUBECONFIG=$(ocne cluster show -C mycluster)
   watch kubectl get nodes -A
   

   Wait for both of the new control plane nodes to display under the NAME column with a ready STATUS. Then type ctrl-c to exit the watch command.

Scale Down a Worker Node

1. Set the kubectl context to reference the bootstrap cluster.

   export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local
   

2. Edit the Custom Resource.

   kubectl edit machinedeployments --namespace ocne mycluster-md-0
   

3. Set the Replicas field to reduce the number of worker nodes.

   Change the replicas: 4 field to: replicas: 3. Then save and exit.

   Example Output:

   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be
   # reopened with the relevant failures.
   #
   apiVersion: cluster.x-k8s.io/v1beta1
   kind: MachineDeployment
   metadata:
     annotations:
   ...
   ...
   spec:
     clusterName: mycluster
     minReadySeconds: 0
     progressDeadlineSeconds: 600
     replicas: 3
     revisionHistoryLimit: 1
   ...
   ...
   

   Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

Scale Down the Control Plane Nodes

1. Edit the Custom Resource (using the NAME noted in the previous step).

   kubectl edit kubeadmcontrolplane --namespace ocne mycluster-control-plane 
   

2. Set the Replicas field to a lower number to reduce the number of control plane nodes.

   Change the replicas: 5 field to: replicas: 3. Then save and exit.

   Example Output:

   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be   
   # reopened with the relevant failures.
   #
   apiVersion: controlplane.cluster.x-k8s.io/v1beta1
   kind: KubeadmControlPlane
   metadata:
     creationTimestamp: "2024-11-05T17:03:52Z"
     finalizers:
     - kubeadm.controlplane.cluster.x-k8s.io
     generation: 3
     labels:
       cluster.x-k8s.io/cluster-name: mycluster
   ...
   ...
     machineTemplate:
      infrastructureRef:
         apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
         kind: OCIMachineTemplate
         name: mycluster-control-plane
         namespace: ocne
       metadata: {}
     replicas: 5
     rolloutStrategy:
   ...
   ...
   

   Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

   Important: The number of control plane nodes set must always be an odd number or the scale request will not be completed.

3. Confirm the removal of the control plane nodes.

   watch kubectl get ocimachine --namespace ocne
   

   Example Output:

   Every 2.0s: kubectl get ocimachine --namespace ocne                    ocne: Wed Dec 18 11:35:43 2024
   
   NAME                            AGE
   mycluster-control-plane-8wq8l   10m
   mycluster-control-plane-qzwmb   12m
   mycluster-control-plane-t6gc8   30m
   mycluster-md-0-fvtvm-qs4dl	    54m
   mycluster-md-0-fvtvm-ttpl5	    54m
   mycluster-md-0-fvtvm-xf4ch	    37m
   

   When the removal process is complete, you can type ctrl-c to exit the watch command.

4. Set the context back to the Workload cluster and confirm the removal of the control plane nodes.

   export KUBECONFIG=$(ocne cluster show -C mycluster)
   kubectl get nodes -A
   

Next Steps

That completes the demonstration of adding and removing nodes from a Kubernetes cluster. While this exercise demonstrated updating the control plane and worker nodes simultaneously, this is not the recommended approach to scaling up or down an Oracle Cloud Native Environment Kubernetes cluster. In production environments, administrators should undertake these tasks separately.

Related Links

• Oracle Cloud Native Environment Documentation
• Oracle Cloud Native Environment Track
• Oracle Linux Training Station

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.