Example: Installing Calico and Setting Up Network Policies

Having created an enhanced cluster using Kubernetes Engine (using either the Console or the API) and selected flannel overlay as the Network type, you can subsequently install Calico on the cluster to support network policies.

Before you install Calico in place of the flannel CNI plugin, you must first disable the flannel CNI plugin. Note that changing the CNI plugin from flannel to Calico only applies to new nodes that are subsequently created in the cluster. Therefore, we recommend that you do not replace flannel with Calico on clusters that already have existing worker nodes.

For convenience, Calico installation instructions are included below. Note that Calico installation instructions vary between Calico versions. For information about installing different versions of Calico, always refer to the Calico installation documentation.

1. Disable the flannel CNI plugin cluster add-on. See Disabling (and Removing) a Cluster Add-on.

2. If you haven't already done so, follow the steps to set up the cluster's kubeconfig configuration file and (if necessary) set the KUBECONFIG environment variable to point to the file. Note that you must set up your own kubeconfig file. You cannot access a cluster using a kubeconfig file that a different user set up. See Setting Up Cluster Access.
3. Remove the flannel daemonset from the kube-system namespace by entering:

   kubectl delete -n kube-system daemonset kube-flannel-ds

4. In a terminal window, download the Calico manifest for the Kubernetes API datastore by entering:

   curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml -o calico.yaml

   Note that the url differs, according to the version of Calico that you want to install.

5. If you have selected an Oracle Linux 8 image for worker nodes in the cluster, set an additional environment variable in the calico.yaml file as follows:
   1. Open the calico.yaml file in a text editor of your choice.

   2. Add the following environment variable to the env section for the calico-node container in the manifest of the calico-node DaemonSet manifest:

                  - name: FELIX_IPTABLESBACKEND
                    value: "NFT"

   3. Save and close the modified calico.yaml file.

6. Install and configure Calico by entering the following command:

   kubectl apply -f calico.yaml

Having created a cluster using Kubernetes Engine (using either the Console or the API) and selected VCN-native pod networking as the Network type, you can subsequently install Calico on the cluster alongside the OCI VCN-Native Pod Networking CNI plugin to support network policies.

For convenience, Calico installation instructions are included below. Note that Calico installation instructions vary between Calico versions. For information about installing different versions of Calico, always refer to the Calico installation documentation.

1. If you haven't already done so, follow the steps to set up the cluster's kubeconfig configuration file and (if necessary) set the KUBECONFIG environment variable to point to the file. Note that you must set up your own kubeconfig file. You cannot access a cluster using a kubeconfig file that a different user set up. See Setting Up Cluster Access.

2. In a terminal window, download the Calico policy-only manifest for the Kubernetes API datastore by entering:

   curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico-policy-only.yaml -o calico-policy-only.yaml

   Note that the url differs, according to the version of Calico that you want to install.

3. The calico-policy-only.yaml file includes Calico components that are not required when using Calico alongside the OCI VCN-Native Pod Networking CNI plugin, so you have to remove these components. You also have to set some additional environment variables.
   1. Open the calico-policy-only.yaml file in a text editor of your choice.
   2. Remove the initContainers section from the manifest of the calico-node DaemonSet.
   3. Remove the following from the env section for the calico-node container from the manifest of the calico-node DaemonSet:

                # Typha support: controlled by the ConfigMap.
                - name: FELIX_TYPHAK8SSERVICENAME
                    valueFrom:
                        configMapKeyRef:
                            name: calico-config
                            key: typha_service_name

   4. Remove the following envFrom section for the calico-node container from the manifest of the calico-node DaemonSet:

                envFrom:
                - configMapRef:
                    # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                    name: kubernetes-services-endpoint
                    optional: true

   5. Remove the following volumes from the volumes section of the manifest of the calico-node DaemonSet:
      • cni-bin-dir
      • cni-net-dir
      • cni-log-dir

      Before you make the change, the volumes section of the calico-node DaemonSet manifest looks like this:

            ...
            volumes:
              # Used by calico-node.
              - name: lib-modules
                hostPath:
                  path: /lib/modules
              - name: var-run-calico
                hostPath:
                  path: /var/run/calico
              - name: var-lib-calico
                hostPath:
                  path: /var/lib/calico
              - name: xtables-lock
                hostPath:
                  path: /run/xtables.lock
                  type: FileOrCreate
              - name: sys-fs
                hostPath:
                  path: /sys/fs/
                  type: DirectoryOrCreate
              - name: bpffs
                hostPath:
                  path: /sys/fs/bpf
                  type: Directory
              # mount /proc at /nodeproc to be used by mount-bpffs initContainer to mount root cgroup2 fs.
              - name: nodeproc
                hostPath:
                  path: /proc
              # Used to install CNI.
              - name: cni-bin-dir
                hostPath:
                  path: /opt/cni/bin
              - name: cni-net-dir
                hostPath:
                  path: /etc/cni/net.d
              # Used to access CNI logs.
              - name: cni-log-dir
                hostPath:
                  path: /var/log/calico/cni
              # Used to create per-pod Unix Domain Sockets
              - name: policysync
                hostPath:
                  type: DirectoryOrCreate
                  path: /var/run/nodeagent
                ...

      After you have made the change, check that the volumes section of the calico-node DaemonSet manifest looks like this:

            ...
            volumes:
              # Used by calico-node.
              - name: lib-modules
                hostPath:
                  path: /lib/modules
              - name: var-run-calico
                hostPath:
                  path: /var/run/calico
              - name: var-lib-calico
                hostPath:
                  path: /var/lib/calico
              - name: xtables-lock
                hostPath:
                  path: /run/xtables.lock
                  type: FileOrCreate
              - name: sys-fs
                hostPath:
                  path: /sys/fs/
                  type: DirectoryOrCreate
              - name: bpffs
                hostPath:
                  path: /sys/fs/bpf
                  type: Directory
              # mount /proc at /nodeproc to be used by mount-bpffs initContainer to mount root cgroup2 fs.
              - name: nodeproc
                hostPath:
                  path: /proc
              # Used to create per-pod Unix Domain Sockets
              - name: policysync
                hostPath:
                  type: DirectoryOrCreate
                  path: /var/run/nodeagent
                ...

   6. Remove the following volume mounts from the volumeMounts section for the calico-node container in the manifest of the calico-node DaemonSet:
      • cni-net-dir, including the associated comment # For maintaining CNI plugin API credentials.
      • cni-log-dir

      Before you make the change, the volumeMounts section looks like this:

                ...
                volumeMounts:
                  # For maintaining CNI plugin API credentials.
                  - mountPath: /host/etc/cni/net.d
                    name: cni-net-dir
                    readOnly: false
                  - mountPath: /lib/modules
                    name: lib-modules
                    readOnly: true
                  - mountPath: /run/xtables.lock
                    name: xtables-lock
                    readOnly: false
                  - mountPath: /var/run/calico
                    name: var-run-calico
                    readOnly: false
                  - mountPath: /var/lib/calico
                    name: var-lib-calico
                    readOnly: false
                  - name: policysync
                    mountPath: /var/run/nodeagent
                  # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
                  # parent directory.
                  - name: bpffs
                    mountPath: /sys/fs/bpf
                  - name: cni-log-dir
                    mountPath: /var/log/calico/cni
                    readOnly: true
                    ...

      After you have made the change, check that the volumeMounts section looks like this:

                ...
                volumeMounts:
                  - mountPath: /lib/modules
                    name: lib-modules
                    readOnly: true
                  - mountPath: /run/xtables.lock
                    name: xtables-lock
                    readOnly: false
                  - mountPath: /var/run/calico
                    name: var-run-calico
                    readOnly: false
                  - mountPath: /var/lib/calico
                    name: var-lib-calico
                    readOnly: false
                  - name: policysync
                    mountPath: /var/run/nodeagent
                  # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
                  # parent directory.
                  - name: bpffs
                    mountPath: /sys/fs/bpf
                    ...

   7. Add the following environment variables for the calico-node container in the manifest of the calico-node DaemonSet:
      • FELIX_INTERFACEPREFIX="oci"
      • NO_DEFAULT_POOLS="true"
      • FELIX_CHAININSERTMODE="Append"
      • FELIX_IPTABLESMANGLEALLOWACTION="Return"
      • FELIX_IPTABLESBACKEND="NFT" Note: Only add this environment variable if you have selected an Oracle Linux 8 image for worker nodes in the cluster.

      Before you make the change, the calico-node container environment variables (env:) section of the calico-node DaemonSet manifest looks like this:

            ...
            containers:
              # Runs calico-node container on each Kubernetes node. This
              # container programs network policy and routes on each
              # host.
              - name: calico-node
                image: docker.io/calico/node:v3.28.0
                imagePullPolicy: IfNotPresent
                env:
                  # Use Kubernetes API as the backing datastore.
                  - name: DATASTORE_TYPE
                    value: "kubernetes"
                  # Configure route aggregation based on pod CIDR.
                  - name: USE_POD_CIDR
                    value: "true"
                  # Wait for the datastore.
                  - name: WAIT_FOR_DATASTORE
                    value: "true"
                  # Set based on the k8s node name.
                  - name: NODENAME
                    valueFrom:
                      fieldRef:
                        fieldPath: spec.nodeName
                  # Don't enable BGP.
                  - name: CALICO_NETWORKING_BACKEND
                    value: "none"
                  # Cluster type to identify the deployment type
                  - name: CLUSTER_TYPE
                    value: "k8s"
                  # Non-calico CNI, disable credential management.
                  - name: CALICO_MANAGE_CNI
                    value: "false"
                  # The default IPv4 pool to create on startup if none exists. Pod IPs will be
                  # chosen from this range. Changing this value after installation will have
                  # no effect. This should fall within `--cluster-cidr`.
                  # - name: CALICO_IPV4POOL_CIDR
                  #   value: "192.168.0.0/16"
                  # Disable file logging so `kubectl logs` works.
                  - name: CALICO_DISABLE_FILE_LOGGING
                    value: "true"
                  # Set Felix endpoint to host default action to ACCEPT.
                  - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
                    value: "ACCEPT"
                  # Disable IPv6 on Kubernetes.
                  - name: FELIX_IPV6SUPPORT
                    value: "false"
                  - name: FELIX_HEALTHENABLED
                    value: "true"         
                  ...

      After you have made the change, check that the calico-node container environment variables (env:) section of the calico-node DaemonSet manifest looks like this:

            ...
            containers:
              # Runs calico-node container on each Kubernetes node. This
              # container programs network policy and routes on each
              # host.
              - name: calico-node
                image: docker.io/calico/node:v3.28.0
                imagePullPolicy: IfNotPresent
                env:
                  # Use Kubernetes API as the backing datastore.
                  - name: DATASTORE_TYPE
                    value: "kubernetes"
                  # Configure route aggregation based on pod CIDR.
                  - name: USE_POD_CIDR
                    value: "true"
                  # Wait for the datastore.
                  - name: WAIT_FOR_DATASTORE
                    value: "true"
                  # Set based on the k8s node name.
                  - name: NODENAME
                    valueFrom:
                      fieldRef:
                        fieldPath: spec.nodeName
                  # Don't enable BGP.
                  - name: CALICO_NETWORKING_BACKEND
                    value: "none"
                  # Cluster type to identify the deployment type
                  - name: CLUSTER_TYPE
                    value: "k8s"
                  # Non-calico CNI, disable credential management.
                  - name: CALICO_MANAGE_CNI
                    value: "false"
                  # The default IPv4 pool to create on startup if none exists. Pod IPs will be
                  # chosen from this range. Changing this value after installation will have
                  # no effect. This should fall within `--cluster-cidr`.
                  # - name: CALICO_IPV4POOL_CIDR
                  #   value: "192.168.0.0/16"
                  # Disable file logging so `kubectl logs` works.
                  - name: CALICO_DISABLE_FILE_LOGGING
                    value: "true"
                  # Set Felix endpoint to host default action to ACCEPT.
                  - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
                    value: "ACCEPT"
                  # Disable IPv6 on Kubernetes.
                  - name: FELIX_IPV6SUPPORT
                    value: "false"
                  - name: FELIX_HEALTHENABLED
                    value: "true"
                  - name: FELIX_INTERFACEPREFIX
                    value: "oci"
                  - name: NO_DEFAULT_POOLS
                    value: "true"
                  - name: FELIX_CHAININSERTMODE
                    value: "Append"
                  - name: FELIX_IPTABLESMANGLEALLOWACTION
                    value: "Return"   
                  - name: FELIX_IPTABLESBACKEND
                    value: "NFT"         
                  ...

   8. Save and close the modified calico-policy-only.yaml file.

4. Install and configure Calico by entering the following command:

   kubectl apply -f calico-policy-only.yaml

Having installed Calico on a cluster you've created with Kubernetes Engine, you can create Kubernetes NetworkPolicy resources to isolate pods as required.

For NetworkPolicy examples and how to use them, see the Calico documentation and specifically:

• Kubernetes policy, demo
• Kubernetes policy, basic tutorial
• Kubernetes policy, advanced tutorial

Note that the examples vary, according to the Calico version you've installed.