Exadata Secure RDMA Fabric Isolation enables strict network isolation for virtual machine (VM) clusters on Oracle Exadata systems that use RDMA over Converged Ethernet (RoCE).

Secure Fabric provides critical infrastructure for secure consolidation of multiple tenants on Oracle Exadata, where each tenant resides in a dedicated VM cluster. Using this feature ensures that:

• Database servers in separate clusters cannot communicate with each other. They are completely isolated from each other on the network.
• Database servers in multiple clusters can share all of the storage server resources. However, even though the different clusters share the same storage network, no cross-cluster network traffic is possible.

With Secure Fabric, each database cluster uses a dedicated network partition and VLAN ID for cluster networking between the database servers, which supports Oracle Real Application Clusters (Oracle RAC) inter-node messaging. In this partition, all of the database servers are full members. They can communicate freely within the partition but cannot communicate with database servers in other partitions.

Another partition, with a separate VLAN ID, supports the storage network partition. The storage servers are full members in the storage network partition, and every database server VM is also a limited member. By using the storage network partition:

• Each database server can communicate with all of the storage servers.
• Each storage server can communicate with all of the database servers that they support.
• Storage servers can communicate directly with each other to perform cell-to-cell operations.

The following diagram illustrates the network partitions that support Exadata Secure RDMA Fabric Isolation. In the diagram, the line connecting the Sales VMs illustrates the Sales cluster network. The Sales cluster network is the dedicated network partition that supports cluster communication between the Sales VMs. The line connecting the HR VMs illustrates the HR cluster network. The HR cluster network is another dedicated network partition that supports cluster communication between the HR VMs. The lines connecting the database server VMs (Sales and HR) to the storage servers illustrate the storage network. The storage network is the shared network partition that supports communications between the database server VMs and the storage servers. But, it does not allow communication between the Sales and HR clusters.

As illustrated in the diagram, each database server (KVM host) can support multiple VMs in separate database clusters. However, Secure Fabric does not support configurations where one database server contains multiple VMs belonging to the same database cluster. In other words, using the preceding example, one database server cannot support multiple Sales VMs or multiple HR VMs.

To support the cluster network partition and the storage network partition, each database server VM is plumbed with 4 virtual interfaces:

• clre0 and clre1 support the cluster network partition.

• stre0 and stre1 support the storage network partition.

  Corresponding stre0 and stre1 interfaces are also plumbed on each storage server.

On each server, the RoCE network interface card acts like a switch on the hypervisor, which performs VLAN tag enforcement. Since this is done at the KVM host level, cluster isolation cannot be bypassed by any software exploits or misconfiguration on the database server VMs.

You can only enable Secure Fabric as part of the initial system deployment using Oracle Exadata Deployment Assistant (OEDA). You cannot enable Secure Fabric on an existing system without wiping the system and re-deploying it using OEDA. When enabled, Secure Fabric applies to all servers and clusters that share the same RoCE Network Fabric.

To use Secure Fabric you must:

1. Configure the RoCE Network Fabric switch hardware to enable Secure Fabric. After you complete the switch configuration, the leaf switch ports become trunk ports, which can carry network traffic with multiple VLAN IDs.

   The switch configuration must occur before initial system deployment using OEDA. See Configuring the RoCE Network Fabric Switches to Enable Exadata Secure RDMA Fabric Isolation.

2. As part of initial system deployment using OEDA, select the option to enable Secure Fabric and specify VLAN IDs for the cluster and storage network partitions associated with each VM cluster.

   In the OEDA Web user interface, the option to enable Secure Fabric is one of the advanced options associated with the Cluster Networks page. When the option to enable Secure Fabric is selected, the Cluster Networks page automatically displays additional fields to specify the VLAN IDs required to configure Secure Fabric.

   Commencing with the October 2024 Oracle Exadata System Software release updates (24.1.5, 23.1.19, and 22.1.28), the option to enable Secure Fabric is selected by default for all new configurations using VM clusters.

   See Using the Browser-based Version of Oracle Exadata Deployment Assistant.