You can configure or unconfigure an IB device on a system by using the cfgadm command. The command manages dynamic reconfiguration (DR) of the entire IB fabric as seen by a host. In addition, the command enables you to do the following:
Display the IB fabric
Manage communication services
Update P_key table databases
For more information, see the cfgadm_ib(1M) man page.
All the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices support cfgadm operations.
This section describes how to display information about IB devices on the system.
To display IB device information, use the following command syntax:
# cfgadm -[a]l
where the –a option adds more information that is displayed than by using the –l option alone.
In the following sample output, information is truncated to show relevant IB information only.
# cfgadm -al Ap_Id Type Receptacle Occupant Condition ib IB-Fabric connected configured ok hca:21280001A0A478 IB-HCA connected configured ok ib::21280001A0A47A,0,ipib IB-PORT connected configured ok ib::21280001A0A479,0,ipib IB-PORT connected configured ok ib::1730000008070,0,hnfs IB-HCA_SVC connected configured ok ib::daplt,0 IB-PSEUDO connected configured ok ib::iser,0 IB-PSEUDO connected configured ok ib::rdsib,0 IB-PSEUDO connected configured ok ib::rdsv3,0 IB-PSEUDO connected configured ok ib::rpcib,0 IB-PSEUDO connected configured ok ib::sdpib,0 IB-PSEUDO connected configured ok ib::sol_umad,0 IB-PSEUDO connected configured ok ib::sol_uverbs,0 IB-PSEUDO connected configured ok
The cfgadm command displays information about attachment points, which are locations in the system where DR operations can occur.
For more information about attachment points, see Attachment Points. For more information about attachment points supported by the cfgadm command, see the cfgadm_ib(1M) man page.
An attachment point consists of a receptacle and an occupant. The following table describes the state of a device depending on the corresponding combined states of the receptacle, occupant, and condition:
|
Devices are listed by their attachment point IDs (Ap_Id). Note that all IB Ap_Ids are shown as connected. The following list explains the information for a specific Ap_Id in the sample output.
Identifies an IB port device that is connected to port GUID and is bound to the ipib service.
Identifies a pseudo device.
Identifies an HCA device.
Identifies an IB HCA_SVC device that is bound to the hnfs service.
Identifies a pseudo device.
You can use other options of the cfgadm command to further customize the information to be displayed. The following examples show the use of selected options.
Example 14 Displaying Information About a Port DeviceThis example shows information about the port device ib::21280001A0A47A,0,ipib.
# cfgadm -al -s "cols=ap_id:info" ib::21280001A0A47A,0,ipib Ap_Id Information ib::21280001A0A47A,0,ipib ipibExample 15 Displaying Port and GUID Information of an HCA Device
This example shows the number of ports and their respective GUIDs for the HCA device hca::1730000008070.
# cfgadm -al -s "cols=ap_id:info" hca::1730000008070 Ap_Id Information hca::1730000008070 VID: 0x15b3, PID: 0x5a44, #ports: 0x2, port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072Example 16 Displaying Kernel Clients of an HCA Device
This example shows the kernel clients of the HCA hca:173000007F50.
$ cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,0,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no
In the sample output, if a kernel IB client uses an HCA other than hca:173000007F50, the entry under the column Alternate HCA would indicate yes. If IB Managers and kernel clients that do not use the HCA, their Ap_Ids would not be listed. Moreover, although EoIB uses the Solaris IB framework, the command does not list EoIB as a kernel IB client.
Example 17 Displaying Supported Communication ServicesThe following example lists IB communication services currently used by the InfiniBand Transport Framework (IBTF).
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs
You can also use the prtconf command to display general information about IB devices. In the following example, pci15b3,673c refers to an IB HCA.
$ prtconf . . . ib, instance #0 rpcib, instance #0 rdsib, instance #0 daplt, instance #0 rdsv3, instance #0 sdpib, instance #0 sol_umad, instance #0 sol_uverbs, instance #0 iser, instance #0 . . . pci15b3,673c, instance #0 ibport, instance #0 ibport, instance #1
You use the following commands to configure or unconfigure IB devices. The commands apply to IB port, HCA_SVC, or VPPA devices.
cfgadm -c configure device configures the IB device.
cfgadm -c unconfigure device unconfigures the IB device.
In both commands, the device is represented by its Ap_Id.
As a useful practice, you should display information about the system devices before and after the actual step of configuring or unconfiguring a device. The information helps you in the following ways:
Properly identify the bus or device to dynamically reconfigure.
Verify from the state of the bus or device that the reconfiguration completed successfully.
The following examples show how to use the cfgadm command to perform DR on IB devices.
Example 18 Configuring an IB Port DeviceThe following example configures the IB port device ib::1730000007F51,*0*,ipib and includes verification of the operation.
# cfgadm -c configure ib::1730000007F51,*0*,ipib
# cfgadm -a ib::1730000007F51,*0*,ipib
Ap_Id Type Receptacle Occupant Condition
ib::1730000007F51,*0*,ipib IB-Port connected configured okThe port device is configured.
Note that if EoIB datalinks were configured over the IB Port when the port was previously unconfigured, you must re-create the vnics corresponding to those EoIB datalinks on the EoIB Gateway with the new IB HCA port GUIDs. You must re-create the vnics before re-configuring the IB Port. For information about how to create vnics on the Gateway, refer to the Sun Network QDR InfiniBand Gateway Switch Administration Guide (https://docs.oracle.com/cd/E36256_01/pdf/E36260.pdf).
Example 19 Unconfiguring an IB Port DeviceThe following example unconfigures the IB port device ib::1730000007F51,*0*,ipib and includes verification of the operation.
# cfgadm -c unconfigure ib::1730000007F51,*0*,ipib
Unconfigure the device: /devices/ib:fabric::1730000007F51,*0*,ipib
This operation will suspend activity on the IB device
Continue (yes/no)? Y
# cfgadm -a ib::1730000007F51,*0*,ipib
Ap_Id Type Receptacle Occupant Condition
ib::1730000007F51,*0*,ipib IB-Port connected unconfigured unknownThe port device is unconfigured.
Example 20 Configuring an IB Pseudo Device
The following example configures the pseudo device ib::ibgen,0 and includes verification of the operation.
# cfgadm -yc configure ib::ibgen,0
# cfgadm -a ib::ibgen,0
Ap_Id Type Receptacle Occupant Condition
ib::ibgen,0 IB-PSEUDO connected configured okThe device is configured.
Example 21 Unconfiguring an IB Pseudo Device
The following example unconfigures the pseudo device ib::ibgen,0 and includes verification of the operation.
# cfgadm -c unconfigure ib::ibgen,0
Unconfigure the device: /devices/ib:fabric::ibgen,0
This operation will suspend activity on the IB device
Continue (yes/no)? Y
# cfgadm -a ib::ibgen,0
Ap_Id Type Receptacle Occupant Condition
ib::ibgen,0 IB-PSEUDO connected unconfigured unknownThe device is unconfigured.
This section provides examples of the use of the following cfgadm command options to modify existing IB configurations:
cfgadm -o hardware-options
cfgadm -x hardware-functions (also used to customize information display. See Displaying IB Device Information: the cfgadm Command.)
When you enable additional P_keys or disable them, the P_key table information of an HCA's ports changes. For consistency, corresponding updates must also occur in the internal P_key databases of the InfiniBand Transport Framework (IBTF) and IBDM.
The following example shows how to update the P_key databases of IBTF and IBDM.
# cfgadm -x update_pkey_tbls -y ib
For more information, see the ibtl(7D) and ibdm(7D) man pages.
Example 23 Adding a Communication ServiceThe following example adds a VPPA communication service to the IB device and also includes verification of the operation.
# cfgadm -o comm=vppa,service=new -x add_service ib
# cfgadm -x list_services ib
Port communication services:
srp
VPPA communication services:
ibd
newThe service has been added.
HCA_SVC communication services:
nfs_service
Example 24 Remove an Existing Communication Service
This example removes the communication service that was added in the previous example. The name of the communication service is new. After the removal, the completion of the operation is verified.
# cfgadm -o comm=vppa,service=new -x delete_service ib # cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs