Configuring the RDMA Network Fabric Switch

You must perform an initial configuration of the RDMA Network Fabric switch.

Configuring the InfiniBand Switches

The two Sun Datacenter InfiniBand Switch 36 leaf switches are identified in Recovery Appliance as iba and ibb. Complete these configuration procedures for both switches:

Configuring an InfiniBand Switch

The default identifier for leaf switch 1 in U20 is iba, and for leaf switch 2 in U22 is ibb.

To configure a Sun Datacenter InfiniBand Switch 36 switch:

For a one-rack installation, unplug the InfiniBand cable from Port 8B on the InfiniBand leaf switches. Use hook-and-loop tape to hang it out of the way.

This cable is preinstalled for a connection to the spine switch in a multirack installation. However, in a one-rack installation, the monitoring software might show it as a down link.
Connect a serial cable between your laptop and the InfiniBand switch USB serial adapter. Use these terminal settings, as needed:
```
TERM=vt100; export TERM
```

localhost: ilom-admin
password: welcome1

Ensure that the firmware version is 2.1.5-1 or later:
```
-> version
```
If the switch has a lower version than 2.1.5-1, then contact Oracle Support Services.
Set the switch host name, without the domain name. The following example assigns the name ra1sw to the first gateway switch (iba):
```
-> set /SP hostname=ra1sw-iba
-> show /SP hostname
/SP
Properties:
hostname = ra1sw-iba
```
See the Installation Template for the name of the switch.
Set the DNS server and domain names. In the following syntax, IP_addresses can have up to three IP addresses, separated by commas, in the preferred search order.
```
-> set /SP/clients/dns auto_dns=enabled
-> set /SP/clients/dns nameserver=IP_addresses
-> set /SP/clients/dns searchpath=domain_name
```

Verify the settings:

-> show /SP/clients/dns
/SP/clients/dns
Targets:
Properties:
auto_dns = enabled
nameserver = 10.196.23.245, 138.2.202.15
retries = 1
searchpath = example.com
timeout = 5
     .
     .
     .

Configure the switch management network settings. In the following commands, pending_ip, pending_gw, and pending_nm are IP addresses defined by the network administrator:

-> cd /SP/network
-> set pendingipaddress=pending_ip
-> set pendingipgateway=pending_gw
-> set pendingipnetmask=pending_nm
-> set pendingipdiscovery=static
-> set commitpending=true

Verify the settings:

-> show
/SP/network
Targets:
test
Properties:
commitpending = (Cannot show property)
dhcp_server_ip = none
ipaddress = 10.196.16.152
ipdiscovery = static
ipgateway = 10.196.23.254
ipnetmask = 255.255.248.0
macaddress = 00:E0:4B:38:77:7E
pendingipaddress = 10.196.16.152
pendingipdiscovery = static
pendingipgateway = 10.196.23.254
pendingipnetmask = 255.255.248.0
state = enabled
     .
     .
     .

If any of the values are wrong, repeat the set pendingipparameter command, and then the commitpending=true command.

Setting the Time on an InfiniBand Switch

To set the time on an InfiniBand switch:

Set the time zone, using the value shown in the Installation Template. The following commands display the current setting, change the time zone, and verify the new setting:
```
-> show /SP/clock 
-> set /SP/clock timezone=zone identifier
-> show /SP/clock
```
The Oracle Exadata Deployment Assistant generates the Installation Template. See Using Oracle Exadata Deployment Assistant.
Set the SP clock to the current time. Use the time format MMddHHmmCCyy, indicating the month, day, hour, minute, century, and year. The following commands display the current setting, change the time, and verify the new setting:
```
-> show /SP/clock
-> set /SP/clock datetime=MMddHHmmCCyy
-> show /SP/clock
```
Configure NTP. The following commands configure both the primary (1) and the secondary (2) NTP servers:
```
-> set /SP/clients/ntp/server/1 address=IP_address
-> set /SP/clients/ntp/server/2 address=IP_address
-> set /SP/clock usentpserver=enabled
```
Note:

If the network does not use NTP, then configure the first compute server (U16) as an NTP server before you install the software in Installing the Recovery Appliance Software.

Verify the IP address of the primary NTP server:

-> show /SP/clients/ntp/server/1
/SP/clients/ntp/server/1
   Targets:

   Properties:
      address = 10.204.74.2

   Commands:
      cd
      set
      show

Verify the IP address of the secondary NTP server:

-> show /SP/clients/ntp/server/2
/SP/clients/ntp/server/2
   Targets:

   Properties:
      address = 10.196.16.1
     .
     .
     .

Verify the time:

-> show /SP/clock
/SP/clock
   Targets:

   Properties:
      datetime = Mon Nov 04 11:53:19 2013
      timezone = EST (US/Eastern)
      usentpserver = enabled
     .
     .
     .

Setting the Serial Number on a Spine Switch

In a multirack configuration, set the rack master serial number in the ILOM of the spine switch. Skip this procedure when configuring the leaf switches.

To set the serial number on the spine switch:

Set the system identifier to 40 characters or fewer:
```
-> set /SP system_identifier="Oracle ZDLRA X5 serial_number"
```
An invalid property value error indicates too many characters.

Verify that the value is set:

-> show /SP system_identifier
     /SP
       Properties:
         system_identifier = Oracle ZDLRA X5 AK012345678

Checking the Health of an InfiniBand Switch

To check the health of an InfiniBand switch:

Open the fabric management shell:

-> show /SYS/Fabric_Mgmt
NOTE: show on Fabric_Mgmt will launch a restricted Linux shell.
User can execute switch diagnosis, SM Configuration and IB
monitoring commands in the shell. To view the list of commands,
use "help" at rsh prompt.
Use exit command at rsh prompt to revert back to
ILOM shell.
FabMan@hostname->

The prompt changes from -> to FabMan@hostname->

Check the general health of the switch:

FabMan@ra1sw-iba-> showunhealthy
OK - No unhealthy sensors

Check the general environment.

FabMan@ra1sw-iba-> env_test
NM2 Environment test started:
Starting Voltage test:
Voltage ECB OK
Measured 3.3V Main = 3.28 V
Measured 3.3V Standby = 3.42 V
Measured 12V = 12.06 V
     .
     .
     .

The report should show that fans 1, 2, and 3 are present, and fans 0 and 4 are not present. All OK and Passed results indicate that the environment is normal.

Determine the current InfiniBand subnet manager priority of the switch. Leaf switches must have an smpriority of 5, and spine switches must have a smpriority of 8. The sample output shown here indicates the correct priority for a leaf switch.
```
FabMan@ra1sw-iba-> setsmpriority list
Current SM settings:
smpriority 5
controlled_handover TRUE
subnet_prefix 0xfe80000000000000
```

If the priority setting is incorrect, then reset it:

Disable the subnet manager:

FabMan@ra1sw-iba->disablesm
Stopping partitiond daemon.             [ OK ]
Stopping IB Subnet Manager..            [ OK ]

Reset the priority. This example sets the priority on a leaf switch:

FabMan@ra1sw-iba->setsmpriority 5
Current SM settings:
smpriority
5 controlled_handover TRUE
subnet_prefix 0xfe80000000000000

Restart the subnet manager:

FabMan@ra1sw-iba->enablesm
Starting IB Subnet Manager.             [ OK ]
Starting partitiond daemon.             [ OK ]

Log out of the Fabric Management shell and the Oracle ILOM shell:
```
FabMan@ra1sw-iba-> exit
-> exit
```

localhost: root
password: welcome1
[root@localhost ~]# reboot

Disconnect your laptop from the InfiniBand switch.
Repeat these procedures for the second InfiniBand leaf switch.

Setting a Spine Switch as the Subnet Manager Master

The InfiniBand switch located in rack unit 1 (U1) is the spine switch. Recovery Appliance has a spine switch only when it is connected to another Recovery Appliance. It is not included as a basic component of the rack.

Perform these steps after the racks are cabled together

The spine switch is the Subnet Manager Master for the InfiniBand subnet. The Subnet Manager Master has priority 8.

To verify the priority setting of the spine switch:

Log in to the spine switch as the root user.
Run the setsmpriority list command.

The command should show that smpriority has a value of 8. If smpriority has a different value, then do the following:
1. Use the disablesm command to stop the Subnet Manager.
2. Use the setsmpriority 8 command to set the priority to 8.
3. Use the enablesm command to restart the Subnet Manager.

The other two InfiniBand switches are the leaf switches. The leaf switches are located in rack units 20 and 22 (U20 and U22). They are the Standby Subnet Managers with a priority of 5. You can verify the status using the preceding procedure, substituting a value of 5 in the command shown in step 2 2.b.

To determine the Subnet Manager Master:

Log in as the root user on any InfiniBand switch.
Display the location of the Subnet Manager Master.
```
# getmaster
20100701 11:46:38 OpenSM Master on Switch : 0x0021283a8516a0a0 ports 36 Sun DCS 36
QDR switch ra01sw-ib1.example.com enhanced port 0 lid 1 lmc 0
```
The preceding output shows the proper configuration. The Subnet Master Manager is running on spine switch ra01sw-ib1.example.com.

If the spine switch is not the Subnet Manager Master, then reset the Subnet Manager Master:

Use the getmaster command to identify the current location of the Subnet Manager Master.
Log in as the root user on the leaf switch that is the Subnet Manager Master.
Disable Subnet Manager on the switch. The Subnet Manager Master relocates to another switch.

See Also:

"Disable the Subnet Manager" in Sun Datacenter InfiniBand Switch 36 User's Guide at

http://docs.oracle.com/cd/E19197-01/835-0784-05/z4001de61813698.html#z40003f12047367
Use the getmaster command to identify the current location of the Subnet Manager Master. If the spine switch is not Subnet Manager Master, then repeat steps 2 and 3 until the spine switch is the Subnet Manager Master.
Enable Subnet Manager on the leaf switches that were disabled during this procedure.

See Also:

"Enable the Subnet Manager" in Sun Datacenter InfiniBand Switch 36 User's Guide at

http://docs.oracle.com/cd/E19197-01/835-0784-05/z4001de61707660.html#z40003f12047359

Note:

If the InfiniBand network consists of four or more racks cabled together, then only the spine switches run Subnet Manager. Disable the Subnet Manager on the leaf switches.

Configuring the Cisco Nexus 9336C-FX2 Switch

The RoCE Network Fabric switch supplied with the engineered system rack is minimally configured during installation.

During initial system configuration, you can reset and configure the switch.

Connect from the RoCE Network Fabric switch serial console to a laptop or similar device using the available RJ45 cable.
Ensure the terminal session is recorded on the laptop by logging the output.
The output can be used as a reference that the switch has been configured correctly.
Power on the switch.
Log in as the admin user.
```
User Access Verification
dbm0sw-rocea0 login: admin
Password: ********
```
Note:
If you do not have the password for the admin user, then contact Oracle Support Services.

Erase the existing configuration.

dbm0sw-rocea0# write erase

Warning: This command will erase the startup-configuration.

Do you wish to proceed anyway? (y/n)  [n] y

Restart the system so you can perform the automated setup.

dbm0sw-rocea0# reload

This command will reboot the system. (y/n)?  [n] y

2017 Aug 31 01:09:00 dbm0sw-rocea0 %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface


CISCO SWITCH Ver7.59
Device detected on 0:1:2 after 0 msecs  
...

Switch to normal setup and, when asked if you want to enforce secure password standard, enter no, then enter a new password for the admin user.

Running S93thirdparty-script...

Populating conf files for hybrid sysmgr ...
Starting hybrid sysmgr ...
inserting /isan/lib/modules/klm_cisco_nb.o ... done

Abort Auto Provisioning and continue with normal setup ? (yes/no) [n]: yes

         ---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no) [y]: no

  Enter the password for "admin": 
  Confirm the password for "admin":

When the Basic System Configuration Dialog appears, choose to enter the basic configuration dialog.

        ---- Basic System Configuration Dialog VDC: 1 ----

This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.

Please register Cisco Nexus9000 Family devices promptly with your
supplier. Failure to register may affect response times for initial
service calls. Nexus9000 devices must be registered to receive 
entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime
to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): yes

In the basic configuration, you can use the default inputs until asked to enter the switch name.

In this example, the switch has a name of test123sw-rocea0.

  Create another login account (yes/no) [n]: 
  Configure read-only SNMP community string (yes/no) [n]: 
  Configure read-write SNMP community string (yes/no) [n]: 
  Enter the switch name : test123sw-rocea0

Respond yes when asked to configure Out-of-band management configuration, and specify appropriate network addresses when prompted.

Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: yes
     Mgmt0 IPv4 address : 100.104.10.21
     Mgmt0 IPv4 netmask : 255.255.248.0
  Configure the default gateway? (yes/no) [y]:
     IPv4 address of the default gateway : 100.104.10.1

Respond yes when asked to configure advanced IP options.
```
Configure advanced IP options? (yes/no) [n]: yes
```
Respond yes when asked to configure static route (this can be changed later).
```
Configure static route? (yes/no) [n]: yes
```

Enter the destination prefix and mask, and other values as prompted.

   Destination prefix : 10.100.100.0

   Destination prefix mask : 255.255.255.0

   Next hop IPv4 address : 10.100.100.1

Configure the DNS IPv4 addresses.

Configure the DNS IPv4 address? (yes/no) [n]: yes
   DNS IP address: 10.100.100.2

Skip configuring the default domain name (this will be configured later).
```
Configure the default domain name? (yes/no) [n]: no
```

Accept the default responses until asked to configure SSH and the NTP server.

Enable the telnet service? (yes/no) [n]: no
Enable the ssh service? (yes/no) [y]: yes
   Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa
   Number of rsa key bits <1024-2048> [1024]: 1024
 
Configure the NTP server? (yes/no) [n]: yes
     NTP server IPv4 address : 10.100.100.3

Accept the default responses until asked to specify the CoPP system profile. Enter strict.

 Configure default interface layer (L3/L2) [L2]: 
 Configure default switchport interface state (shut/noshut) [noshut]: 
 Configure CoPP system profile (strict/moderate/lenient/dense) [strict]: strict

After reviewing the configuration, save the configuration.

The following configuration will be applied:
   no password strength-check
   switchname test123sw-rocea0
   ip route 100.104.8.0 255.255.248.0 100.104.10.1
   vrf context management
   ip route 0.0.0.0/0 100.104.10.1
   exit
    no feature telnet
    ssh key rsa 1024 force
    feature ssh
    ntp server 100.104.10.1
    system default switchport
    no system default switchport shutdown
    copp profile strict
   interface mgmt0
   ip address 100.104.10.21 255.255.248.0
   no shutdown

Would you like to edit the configuration? (yes/no) [n]: 

Use this configuration and save it? (yes/no) [y]: yes

[########################################] 100%
Copy complete.

Enable the scp server feature on the switch.
```
test123sw-rocea0# feature scp-server
```

Save the running configuration to flash.

test123sw-rocea0# copy running-config startup-config
[########################################] 100%
Copy complete.

Apply the golden configuration on the switch.
- Starting with Oracle Exadata System Software release 20.1.0, use the procedure described in Applying Golden Configuration Settings on RoCE Network Fabric Switches, in Oracle Exadata Database Machine Maintenance Guide.
- Otherwise, use the following procedure to apply the golden configuration on the switch:
1. Delete the configuration file on the switch for the target configuration.
  
  Note:
  If you do not remove the file you are replacing, then when you attempt to overwrite the file you will get a 'permission denied' error.
  Log in to the switch, enter configuration mode, then run a command similar to the following:
```
test123sw-rocea0# delete bootflash:roce_leaf_switch.cfg
Do you want to delete "/roce_leaf_switch.cfg" ? (yes/no/abort) [y] y
test123sw-rocea0# 
```
2. Log in to a server that has SSH access to the switch, and contains the latest RDMA Network Fabric patch ZIP file.
  
  To find the available RDMA Network Fabric patches, search for 'RDMA network switch' in My Oracle Support document 888828.1. Download and use the latest patch for your Oracle Exadata System Software release.
3. Unzip the RDMA Network Fabric patch ZIP file and change directories to the location of the patchmgr utility.
4. Locate the golden configuration files in the RDMA Network Fabric patch bundle.
  The files are located within the roce_switch_templates directory.
  
  The golden configuration files are as follows:
  - Single rack leaf (leaf): roce_leaf_switch.cfg
  - Multi-rack spine (mspine): roce_spine_switch_multi.cfg
  - Multi-rack leaf (mleaf): roce_leaf_switch_multi.cfg
  - Single rack leaf with Secure Fabric support (sfleaf): roce_sf_leaf_switch.cfg
  - Multi-rack leaf with Secure Fabric support (msfleaf): roce_sf_leaf_switch_multi.cfg
  - Single rack leaf configured with 23 host ports (leaf23): roce_leaf_switch_23hosts.cfg
  - Multi-rack leaf configured with 23 host ports (mleaf23): roce_leaf_switch_23hosts_multi.cfg
  - Multi-rack leaf configured with 14 inter-switch links (mleaf_u14): roce_leaf_switch_14uplinks_multi.cfg
  - Multi-rack leaf configured with 14 inter-switch links and with Secure Fabric support (msfleaf_u14): roce_sf_leaf_switch_14uplinks_multi.cfg
  - Multi-rack leaf configured with 23 host ports and 13 inter-switch links (mleaf23_u13): roce_leaf_switch_23hosts_13uplinks_multi.cfg
5. Copy the golden configuration file to the switch.
  In the following example, 100.104.10.21 represents the IP address of the switch you are configuring.
```
# scp roce_leaf_switch.cfg admin@100.104.10.21:/
User Access Verification
Password:
roce_leaf_switch.cfg 100% 23KB 23.5KB/s 00:00
```
6. Apply the golden configuration file on the switch.
  Use the run-script command while connected directly to the switch.
```
test123sw-rocea0# run-script bootflash:roce_leaf_switch.cfg | grep 'none'
```
  Note:
  This command may take up to 1-2 minutes on a single-rack switch and up to 3-4 minutes on a multi-rack switch.
7. Verify the switch configuration.
  Use the patchmgr utility on the server that has SSH access to the switch, and contains the latest RDMA Network Fabric patch bundle.
  In the following command, roceswitch.lst is a file that contains the switch host name or IP address.
```
# ./patchmgr --roceswitches roceswitch.lst --verify-config
```
Backup up the switch configuration.

Follow the steps in Backing Up Settings on the ROCE Switch, in Oracle Exadata Database Machine Maintenance Guide.
Optional: Set the clock, using the same procedure as in Setting the Clock on the Cisco 93108-1G or 9348 Ethernet Switch.

Related Topics

Exadata Database Machine and Exadata Storage Server Supported Versions (My Oracle Support Doc ID 888828.1)