Software Requirements

Oracle Real Application Clusters (Oracle RAC) on Docker is currently supported only with the following releases:

• Oracle Grid Infrastructure Release 21c (21.3 or later release updates)
• Oracle Database Release 21c (21.3 or later)
• Oracle Container Runtime for Docker Release 18.09 or later
• Oracle Linux 7-slim Docker image (oraclelinux:7-slim)
• Oracle Linux for Docker host on Oracle Linux 7.4 (Linux-x86-64) or later updates.

• Unbreakable Enterprise Kernel Release 5 (UEKR5), Unbreakable Enterprise Kernel 6 (UEKR6), and their updates. Refer to Oracle Linux Oracle Container Runtime for Docker User's Guide for the supported kernel versions.

  In this example, we use Oracle Linux 7.9 (Linux-x86-64) with the Unbreakable Enterprise Kernel 5: 4.14.35-2047.501.2el7uek.x86_64 .

Storage Requirements

Database storage for Oracle RAC on Docker must use Oracle Automatic Storage Management (Oracle ASM) configured either on block storage, or on a network file system (NFS).

Oracle Automatic Storage Management Cluster File System (Oracle ACFS) and Oracle ASM Filter Driver (Oracle ASMFD) are not supported.

Oracle RAC Node 1

• Container: racnode1
• CPU core IDs: 0 and 1
• Memory
  • RAM memory: 16 GB
  • Swap memory: 16 GB
• Oracle Automatic Storage Management (Oracle ASM) Disks
  • /dev/asm-disk1
  • /dev/asm-disk2
• Host names and IP addresses
  • racnode1, 10.0.20.150
  • racnode1-vip, 10.0.20.160
  • racnode1-priv, 192.168.17.150
  • racnode-scan, 10.0.20.170/171/172
  • Domain: example.info

• The racnode1 Docker volumes are mounted using Docker host directory paths and appropriate permissions.

  To ensure that the mount is performed as a directory, each mount is specified using -v or --volume. A volume consists of three fields, separated by colon characters (:). When you set up volumes, you must set the volume fields in the following order: source-path:target-path:options. For example:

  # docker container create ... --volume /boot:/boot:ro ... -name racnode1

  Note that /boot must be readonly inside the container.

  Each of the following volumes are mounted:

  • /boot : read-only (/boot:ro)
  • /dev/shm read-write (/dev/shm)
  • /software/stage read-write (/scratch/software/stage)
  • /u01 read-write (/scratch/rac/cluster01/node1)
  • /sys/fs/cgroup read-only (/sys/fs/cgroup)
  • /etc/localtime read-only (/etc/localtime)

  Note:

  After this procedure is completed, to confirm mounts are set up, you can run the Docker command docker container inspect racnode1. For more information about using this command to check mounts for your configuration, refer to the Docker documentation.

• Oracle Database configuration
  • Release 21.3
  • CDB name: orclcdb
  • PDB name: orclpdb
  • Instance: orclcdb1
  • SGA size: 3 GB
  • PGA size: 2 GB

Oracle RAC Node 2

• Container: racnode2
• CPU core IDs: 2 and 3
• Memory
  • RAM memory: 16 GB
  • Swap memory: 16 GB
• Oracle Automatic Storage Management (Oracle ASM) Disks
  • /dev/asm/disk1
  • /dev/asm-disk2
• Host names and IP addresses
  • racnode2 10.0.20.151
  • racnode2-vip 10.0.20.161
  • racnode2-priv 192.168.17.151
  • racnode-scan 10.0.20.170/171/172
  • Domain: example.info

• The racnode2 Docker volumes are mounted using Docker host directory paths and appropriate permissions.

  To ensure that the mount was performed as a directory, mounts are created for Docker using -v or --volume. A volume consists of three fields, separated by colon characters (:). When you configure your volumes, you must set the volume fields in the following order: source-path:target-path:options. For example:

  # docker container create ... --volume /boot:/boot:ro ... -name racnode2

  Each of the following volumes are created:

  • /boot : read-only (/boot:ro)
  • /dev/shm read-write (/dev/shm)
  • /software/stage read-write (/scratch/software/stage)
  • /u01 read-write (/scratch/rac/cluster01/node2)
  • /sys/fs/cgroup read-only (/sys/fs/cgroup)
  • /etc/localtime read-only (/etc/localtime)

  Note:

  After this procedure is completed, to confirm mounts are set up, you can run the Docker command docker container inspect racnode2. For more information about using this command to check mounts for your configuration, refer to the Docker documentation.

• Oracle Database
  • Instance: orclcdb2

In that installation, the Docker storage is in /var/lib/docker, and used the Docker OverlayFS storage driver option overlay2, with an xfs backing file system, with d_type=true enabled. After the setup, the Docker engine looks like the following example:

# docker info

Client:
 Debug Mode: false

Server:
 Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
 Images: 0
 Server Version: 19.03.1-ol
 Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: c4444445cb9z30000f49999ed999e6d4zz11c7c39
 runc version: 
 init version: fec3683
 Security Options:
 seccomp
 Profile: default
 Kernel Version: 4.14.35-1902.300.11.el7uek.x86_64
 Operating System: Oracle Linux Server 7.7
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 58.71GiB
 Name: docker-inst-1-99242
 ID: ZZZ2:YYYG:VVVV:VVVV:RRRR:EZI2:XYRX:RJPS:7ZZZ:4KZ4:GFJ5:P26O
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
 127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Registries:

To run processes inside a container in real time mode, you must update the OPTIONS value in /etc/sysconfig/docker.

1. Log in as root, and open /usr/lib/systemd/system/docker.service in an editor.
2. Append the following parameter setting for the line starting with entry ExecStart= under the [Service] section:

   --cpu-rt-runtime=950000

   After appending the parameter, the ExecStart value should appear similar to the following example:

   ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --cpu-rt-runtime=950000

3. Save the /usr/lib/systemd/system/docker.service file.
4. Run the following commands to reload the daemon and container with the changes:

   # systemctl daemon-reload
   # systemctl stop docker
   # systemctl start docker

After the changes, running a Docker service status command must return a result similar to the following:

# systemctl status docker

 docker.service - Docker Application Container Engine
 Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
 Active: active (running) since Mon 2020-04-13 23:45:39 GMT; 17h ago
 Docs: https://docs.docker.com
 Main PID: 12892 (dockerd)
 CGroup: /system.slice/docker.service
 12892 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --cpu-rt-runtime=950000

When you restart Docker containers, device permissions, default routes, and /etc/hosts entries that were previously configured for the containers are reset. To maintain your configuration between Docker and Docker container restarts, use this procedure to build a script that you can configure to run on each container to set up the environment after restarts.

Because Oracle Real Application Clusters (Oracle RAC) uses containers based on systemd, you can run the container environment script on every restart by adding the script to the /etc/rc.local folder inside the container when you create the Oracle RAC slim image. After restarts, the script you add to rc.local can ensure that your Docker container environments are restored. Complete each of these steps in sequence. Create the files and script in the docker image build directory, which in this example is /stage/image.

We will add this script to run at container startup by using the Oracle RAC containers /etc/rc.local. This step is described in the section "Build Oracle RAC Database on Docker Image." After you add the line to /etc/rc.local to call the setupContainer.Env.sh script, if the Docker Container is reset, then when a Docker container is started and init loads, setupContainer.Env.sh runs the following operations on every container restart:

1. Sets the default gateway
2. Sets the correct device permissions on ASM devices
3. Sets up the /etc/hosts file.
4. Sets up the /etc/resolv.conf file.

If you plan to deploy containers on multiple hosts, then you must copy setupContainerEnv.sh on all the Docker hosts where the image is built.

1. Create a file named Dockerfile under /scratch/image.

2. Open the Dockerfile with an editor, and paste the following lines into the Dockerfile:

   
   # Pull base image
   # ---------------
   FROM oraclelinux:7-slim
   # Environment variables required for this build (do NOT change)
   # -------------------------------------------------------------
   ## Environment Variables
   ## ---
   ENV container=true \
       SCRIPT_DIR=/opt/scripts/startup \
       RESOLVCONFENV="resolv.conf" \
       HOSTFILEENV="hostfile" \
       SETUPCONTAINERENV="setupContainerEnv.sh"
         
   ### Copy Files
   # ----
   
   COPY $RESOLVCONFENV $HOSTFILEENV $SETUPCONTAINERENV $SCRIPT_DIR/
   
   ### RUN Commands
   # -----
   RUN yum -y install systemd oracle-database-preinstall-21c vim passwd openssh-server && \
    yum clean all && \
    sync && \
    groupadd -g 54334 asmadmin && \
    groupadd -g 54335 asmdba && \
    groupadd -g 54336 asmoper && \
    useradd -u 54332 -g oinstall -G oinstall,asmadmin,asmdba,asmoper,racdba,dba grid && \
    usermod -g oinstall -G oinstall,dba,oper,backupdba,dgdba,kmdba,asmdba,racdba,asmadmin oracle && \
    cp /etc/security/limits.d/oracle-database-preinstall-21c.conf /etc/security/limits.d/grid-database-preinstall-21c.conf && \
    sed -i 's/oracle/grid/g' /etc/security/limits.d/grid-database-preinstall-21c.conf && \
    rm -f /etc/rc.d/init.d/oracle-database-preinstall-21c-firstboot && \
    rm -f /etc/sysctl.conf && \
    echo "$SCRIPT_DIR/$SETUPCONTAINERENV" >> /etc/rc.local && \
    chmod +x $SCRIPT_DIR/$SETUPCONTAINERENV && \
    chmod +x /etc/rc.d/rc.local && \
    sync
   
   USER root
   WORKDIR /root
   VOLUME ["/software/stage"]
   VOLUME ["/u01"]
   CMD ["/usr/sbin/init"]
   # End of the Dockerfile
   

   If you require additional packages for your application, then you can add them to the RUN yum command.

1. Log in as root, and move to the directory for image creation that you have previously prepared:

   # cd /stage/image

2. Run the procedure for your use case:

   • Your server is behind a proxy firewall:
     • Run the following commands, where:
       • localhost-domain is the local host and domain for the internet gateway in your network
       • http-proxy is the HTTP proxy server for your network environment
       • https-proxy is the HTTPS proxy server for your network environment
       • 21.3 is the Oracle Database release that you are planning to install inside the container.

     # export NO_PROXY=localhost-domain 
     # export http_proxy=http-proxy
     # export https_proxy=https-proxy
     # export version=21.3
     # docker build --force-rm=true --no-cache=true --build-arg \
     http_proxy=${http_proxy} --build-arg https_proxy=${https_proxy} \
     -t oracle/database-rac:$version-slim  -f Dockerfile .

   • Your server is not behind a proxy firewall:
     • Run the following commands, where version is the Oracle Database release that you are planning to install inside the Docker Container (for example, 21.3.0):

     # export version=dbrelease
     # docker build --force-rm=true --no-cache=true -t oracle/database-rac:$version-slim -f Dockerfile .

3. After the image builds successfully, you should see the image oracle/database-rac:21.3.0-slim created on your Docker host:

   # docker images
   REPOSITORY TAG IMAGE ID CREATED SIZE
   oracle/database-rac 21.3.0-slim 7d0a89984725 11 hours ago 359MB
   oraclelinux 7-slim f23503228fa1 3 days ago 120MB

4. Save the image into a tar file, and transfer it to the other Docker host:

   # docker image save -o /var/tmp/database-rac.tar oracle/database-rac
   # scp /var/tmp/database-rac.tar docker-host-2:/var/tmp/database-rac.tar
   

5. On the other Docker host load the image from the tar file and check that it is loaded

   # docker image load –i /var/tmp/database-rac.tar
   # docker images
   REPOSITORY TAG IMAGE ID CREATED SIZE
   oracle/database-rac 21.3.0-slim 7d0a89984725 20s ago 359MB
   

After the NFS volume is created with the supported mount options for its usage, it can be used by the container.

docker volume create --driver local \
--opt type=nfs \
--opt o=addr=nfs-server,rw,bg,hard,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0,noac \
--opt device=:/oradata \
racstorage

In this case, you then provide the following argument in the Docker docker container create command:

  --volume racstorage:/oradata \

After these example commands are run, and the docker container is created with the volume argument, it can access the NFS file system at /oradata when the container is up and running.

If you want to use multiple private networks for Oracle Real Application Clusters (Oracle RAC), then you must set the rp_filter tunable to 0 or 2. To set the rp_filter value, you add the arguments for that tunable to the docker create container command when you create your containers:

--sysctl 'net.ipv4.conf.private_interface_name.rp_filter=2'

Based on the order of connection to the public and private networks that you previously configured, you can anticipate the private interface names that you define with this command.

--sysctl 'net.ipv4.conf.eth1.rp_filter=2' --sysctl 'net.ipv4.conf.eth2.rp_filter=2'. 

# docker network create -d macvlan --subnet=10.0.20.0/24 --gateway=10.0.20.1 -o parent=eth0 rac_eth0pub1_nw 
# docker network create -d macvlan --subnet=192.168.17.0/24 -o parent=eth1 rac_eth1priv1_nw

If you cannot use the MACVLAN driver in your environment, then you can still create a Docker bridge in your environment. However, this bridge will not be reachable from an external network. In addition, the Oracle RAC Docker containers for a given cluster will be limited to be in the same Docker host.

# docker network create --driver=bridge --subnet=10.0.20.0/24 rac_eth0pub1_nw
# docker network create --driver=bridge --subnet=192.168.17.0/24 rac_eth1priv1_nw

Use this example of a container configuration as a guideline. The containers that you create for Oracle RAC must have explicit assignment of compute, memory and storage resources to support the database workload you expect for your Oracle RAC cluster. Accordingly, change the values for --cpuset-cpu, --memory, and --device --dns to the values that you require for your workload environment. Ensure that the domain name servers (DNS) that you specify with --dns can resolve the host names and single client access names (SCANs) that you plan to use for Oracle Grid Infrastructure. Also, the default Docker container stop uses the SIGTERM signal to send to the container when a container stop command is issued. However, the Docker SIGTERM signal does not shut down the systemd services inside the container. Oracle Clusterware supports graceful shutdown of a cluster node with a systemd service shutdown. To enable the Docker container stop to trigger a systemd service shutdown in Oracle Clusterware, include the argument --stop-signal=SIGRTMIN+5 in the docker create command. To understand all of the options mentioned in the following command, refer to the Docker documentation for the Oracle Docker version installed on the Docker host. You can also shut down the container gracefully with manual commands. See "How to Gracefully Shut Down a RAC container" in this document.

# docker create -t -i \
  --hostname racnode1 \
  --volume /boot:/boot:ro \
  --volume /dev/hugepages:/dev/hugepages \
  --volume /dev/shm \
  --tmpfs /dev/shm:rw,exec,size=2G \
  --dns-search=example.info \
  --dns=162.88.2.6 \
  --device=/dev/sdd:/dev/asm-disk1  \
  --device=/dev/sde:/dev/asm-disk2  \
  --privileged=false  \
  --volume /scratch/software/stage:/software/stage \
  --volume /scratch/rac/cluster01/node1:/u01 \
  --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --volume /etc/localtime:/etc/localtime:ro \
  --cpuset-cpus 0-3 \
  --memory 16G \
  --memory-swap 32G \
  --sysctl kernel.shmall=2097152  \
  --sysctl "kernel.sem=250 32000 100 128" \
  --sysctl kernel.shmmax=8589934592  \
  --sysctl kernel.shmmni=4096 \
  --sysctl 'net.ipv4.conf.eth1.rp_filter=2' \
  --sysctl 'net.ipv4.conf.eth2.rp_filter=2' \
  --cap-add=SYS_NICE \
  --cap-add=SYS_RESOURCE \
  --cap-add=NET_ADMIN \
  --restart=always \
  --tmpfs=/run \
  --cpu-rt-runtime=95000 \
  --stop-signal=SIGRTMIN+5 \
  --ulimit rtprio=99  \
  --name racnode1 \
oracle/database-rac:21.3.0-slim

To use this example on your Docker host, change the values for --cpuset-cpu, --memory, and --device --dns values to the correct values for your environment. Ensure that the domain name servers (DNS) that you specify with --dns can resolve the host names and single client access names (SCANs) that you plan to use for Oracle Grid Infrastructure. Also, the default Docker container stop uses the SIGTERM signal to send to the container when a container stop command is issued. However, the Docker SIGTERM signal does not shut down the systemd services inside the container. Oracle Clusterware supports graceful shutdown of a cluster node with a systemd service shutdown. To enable the Docker container stop to trigger a systemd service shutdown in Oracle Clusterware, include the argument --stop-signal=SIGRTMIN+5 in the docker create command. To understand all of the options mentioned in the following command, refer to the Docker documentation for the Oracle Docker version installed on the Docker host. You can also shut down the container gracefully with manual commands. See "How to Gracefully Shut Down a RAC container" in this document.

# docker create -t -i \
  --hostname racnode2 \
  --volume /boot:/boot:ro \
  --volume /dev/hugepages:/dev/hugepages \
  --volume /dev/shm \
  --tmpfs /dev/shm:rw,exec,size=2G \
  --dns-search=example.info \
  --dns=162.88.2.6 \
  --device=/dev/sdd:/dev/asm-disk1  \
  --device=/dev/sde:/dev/asm-disk2  \
  --privileged=false  \
  --volume /scratch/rac/cluster01/node2:/u01 \
  --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --volume /etc/localtime:/etc/localtime:ro \
  --cpuset-cpus 0-3 \
  --memory 16G \
  --memory-swap 32G \
  --sysctl kernel.shmall=2097152  \
  --sysctl "kernel.sem=250 32000 100 128" \
  --sysctl kernel.shmmax=8589934592  \
  --sysctl kernel.shmmni=4096 \
  --sysctl 'net.ipv4.conf.eth1.rp_filter=2' \
  --sysctl 'net.ipv4.conf.eth2.rp_filter=2' \
  --cap-add=SYS_NICE \
  --cap-add=SYS_RESOURCE \
  --cap-add=NET_ADMIN \
  --restart=always \
  --tmpfs=/run \
  --cpu-rt-runtime=95000 \
  --ulimit rtprio=99  \
  --stop-signal=SIGRTMIN+5 \
  --name racnode2 \
oracle/database-rac:21.3.0-slim

To ensure that the network interface name used by each node for a given network is the same, each node must use the exact same order of the disconnect and connect commands to the associated networks. For example, consistently across all nodes, the eth0 interface is public. The eth1 interface is the first private network interface, and the eth2 interface is the second private network interface.

# docker network disconnect bridge racnode1
# docker network connect rac_eth0pub1_nw --ip 10.0.20.150 racnode1
# docker network connect rac_eth1priv1_nw --ip 192.168.17.150  racnode1
# docker network connect rac_eth2priv2_nw --ip 192.168.18.150  racnode1

# docker network disconnect bridge racnode2
# docker network connect rac_eth0pub1_nw --ip 10.0.20.151 racnode2
# docker network connect rac_eth1priv1_nw --ip 192.168.17.151  racnode2
# docker network connect rac_eth2priv2_nw --ip 192.168.18.151  racnode2

In this procedure, you log in to the Docker container, evaluate the total container memory, and then derive the 90 percent value recommended for the memlock limit. The containers in this example are racnode1 and racnode2.

If a network for a container running Oracle Real Application Clusters (Oracle RAC) is disconnected, then reconnecting the same network can result in the network interface name in the Oracle RAC node to disappear. In the place of the previous network interface, a different interface name is created. When this scenario happens, the result is a network configuration that is inconsistent with the network configuration in Oracle Clusterware. If that network interface was the sole private interface for the cluster node, then that node can be evicted from the cluster.

To correct this problem, use this procedure to restore the network interface names on the containers to the same network interface names that were originally configured with Oracle Clusterware.

After you complete this procedure, the network interface names for the Oracle RAC node on the container are restored to their original configuration, and consistent with the Oracle Clusterware configuration.

When an Oracle RAC public or private network is connected to a physical network outside of the Docker host, the corresponding Docker network uses the Macvlan mode of bypassing the Linux bridge, using a NIC adapter in the Docker host as the parent adapter.

After configuration, if you need to disconnect and replace a NIC card used with a network for a container running Oracle Real Application Clusters (Oracle RAC), then reconnecting the same network can result in the network interface name in the Oracle RAC node to disappear. In the place of the previous network interface name, a different interface name is created. When this scenario happens, the result is a network configuration that is inconsistent with the network configuration in Oracle Clusterware. If that network interface was the sole private interface for the cluster node, then that node can be evicted from the cluster.

To resolve that issue, use the following procedure:

1. Disable and then stop the Oracle Clusterware on the node:

   # crsctl disable crs
   # crsctl stop crs

2. Disconnect the container from the network corresponding to the host NIC being replaced. For example, where the container name is mycontainer, and the network name is mypriv1, enter the following command:

   
   # docker network disconnect mycontainer mypriv1

3. Replace the host NIC, and ensure that the new NIC is discovered, is available to the host, and the host operating system device name is found. The NIC name can be different from what it was previously. Depending on the host environment, this step can require restarting the host operating system. For example, if the NIC device name was previously eth1, it can be eth3 after replacement.
4. "Recreate the network using the NIC device name that you find. For example, where the network name is mypriv1, and the NIC that previously was eth1 and now is eth3:

   # docker network rm mypriv1
   # docker network create .... -o parent=eth3 mypriv1

5. Stop the container.
6. Disconnect the other networks from the container.
7. Reconnect to the container all of the networks, using the same order that you used when originally creating the container. Connecting networks in the same order ensures that the interface names in the container are the same as before the replacement.
8. Start the container, and verify that network interface names and IP addresses are as before.
9. Restart and enable Clusterware in the node:

   # crsctl enable crs
   # crsctl start crs

To remove Oracle RAC on Docker, first stop and deinstall Oracle software, and then deconfigure Docker. .

After you delete the container that you used for an Oracle RAC node, if you recreate that container again, then it is not an Oracle RAC node, even though you use the same volume for the mount point (/u01). To make the container an Oracle RAC node again, use the node delete and node add procedures in Oracle Grid Infrastructure Installation and Upgrade Guide for Linux, and in Oracle Real Application Clusters Administration and Deployment Guide.

You can shut down containers gracefully either by stopping the Cluster Ready Services (CRS) stack inside the container, and then stopping the container, or by stopping the container directly from the host, using a grace period sufficient for the graceful shutdown of Oracle RAC inside the container.

# docker stop -t 600 racnode1