1. Managing Storage Devices
  2. Working With NVMe Storage Over Fibre Channel

7 Working With NVMe Storage Over Fibre Channel

Oracle Linux uses the Non-volatile Memory Express over Fibre Channel (NVMe/FC) protocol to manage hosts and NVMe solid state storage devices over a fibre channel network. Oracle Linux running on machines that have NVMe/FC enabled host bus adapters (HBAs) can discover and connect to NVMe SSD devices attached to a fibre network. You can establish connectivity from hosts to these SSD devices through these adapters by using various nvme commands which are part of the nvme-cli package. nvme-cli is an NVMe storage CLI utility available on Oracle Linux.

This chapter provides instructions relating to configuring HBA devices such as certain Broadcom Emulex or Marvell Qlogic Fibre Channel adapters to connect to remote NVMe SSD devices over a Fibre channel network.

About NVMe/FC Devices

The NVMe/FC is a protocol for managing NVMe hosts and NVMe SSD storage devices on a Fibre channel network. NVMe/FC requires infrastructure such as fiber-optic cabling, NVMe/FC enabled interface adapters, and NVMe/FC enabled network switches. Fibre channel networks typically provide better throughput, lower latency, and higher performance when compared to iSCSI.

A host (often called the NVMe initiator) accesses the storage targets over a fibre channel network. The host can be a server, a VM, or any device equipped with an NVMe/FC adapter or software stack that can initiate requests to access remote NVMe/FC storage. To an NVMe initiator, the storage appears to be locally attached.

NVMe initiators and targets use dedicated NVMe HBAs which are NVMe adapter with ports through which it connects to a fibre channel network.

The parameters in the nvme connect command that represent the initiator and the target include the following:

  • host-traddr: This parameter specifies the host transport address and refers to the address of the initiator. The address or identifier is the host's fibre channel HBA port through which it connects to the fibre channel fabric. This port is typically called the local port.
  • traddr: This parameter specifies the transport address and refers to the address of the target server or controller hosting one or more NVMe SSD storage device. This port is typically called the remote port.
Local and remote port addresses are a combination of the following elements:
  • World Wide Namespace Name (WWNN): This is a globally unique identifier assigned to each namespace within an NVMe subsystem. The NVMe protocol uses WWNNs when routing commands to correct namespaces.
  • World Wide Namespace Port (WWNP): This is a globally unique identifier that specifies the port through which a namespace is accessed. The NVMe protocol uses WWNP to route commands to the appropriate namespace device.

You can query local and remote port address combinations to discover available target namespaces that represent the NVMe storage devices. After identifying the namepace of a storage device, you can then combine the local and remote port addresses with the device's namespace to connect the host to the target NVMe storage device.

For more information about the NVMe specification, see https://nvmexpress.org/specifications/.

NVMe/FC Storage Adapter Compatibility

The NVMe/FC discovery and disconnect feature is compatible with the Fibre Channel storage adapters listed in the following document: https://linux.oracle.com/Component_Compatibility_Guide.pdf.

Install NVMe-CLI and Identify Host Namespace

The following procedure describes how to install the nvme-cli NVMe CLI utility and identify the host namespace.

  1. On the host open the nvme-cli command line utility: 

    sudo dnf install nvme-cli
  2. Verify that a namespace exists for the host:
    nvme show-hostnqn

    Note:

    Alternatly, you can find the namespace in the /etc/nvme/hostnqn file created when you installed nvme-cli.

Discovering and Connecting to Targets from Broadcom Hosts

The following procedure describes how to discover and connect to an NVMe/FC target from an Oracle Linux host using a Broadcom Emulex Fibre Channel adapter.

  1. Identify the WWNN and WWPN of the local and remote ports in the /sys/class/scsi_host/host*/nvme_info directory. For example, the following shows a host with one local port and two remote ports: 

    cat /sys/class/scsi_host/host*/nvme_info

NVME Initiator Enabled
XRI Dist lpfc1 Total 6144 IO 5894 ELS 250
NVME LPORT lpfc1 WWPN x100000109adcbefh WWNN x200000109adcbefh DID x000000 UNKNOWN

NVME Statistics
LS: Xmt 0000000000 Cmpl 0000000000 Abort 00000000
LS XMIT: Err 00000000  CMPL: xb 00000000 Err 00000000
Total FCP Cmpl 0000000000000000 Issue 0000000000000000 OutIO 0000000000000000
        abort 00000000 noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000
FCP CMPL: xb 00000000 Err 00000000

NVME Initiator Enabled
XRI Dist lpfc0 Total 6144 IO 5894 ELS 250
NVME LPORT lpfc0 WWPN x100000109acfgcad WWNN x200000109acfgcad DID x011700 ONLINE
NVME RPORT       WWPN x2015c146dc13a14b WWNN x2014c146dc13a14b DID x011307 TARGET DISCSRVC ONLINE
NVME RPORT       WWPN x2016c146dc13a14b WWNN x2014c146dc13a14b DID x011105 TARGET DISCSRVC ONLINE

NVME Statistics
LS: Xmt 0000000028 Cmpl 0000000028 Abort 00000000
LS XMIT: Err 00000000  CMPL: xb 00000000 Err 00000000
Total FCP Cmpl 00000000000019fa Issue 00000000000019fa OutIO 0000000000000000
        abort 00000005 noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000
FCP CMPL: xb 00000005 Err 00000005
  2. Apply the following format change to a combination of the WWPN and WWNN from the local port and one of the remote ports.
    • For all WWPNs, append np-0 to the beginning of the address.
    • For all WWNNs, append nn-0 to the beginning of the address.
    For example, 
    LPORT WWNN nn-0x200000109acfgcad WWPN pn-0x100000109acfgcad
RPORT WWNN nn-0x2014c146dc13a14b WWPN pn-0x2015c146dc13a14b
  3. Run a discovery command to identify NVMe SSD devices available on the remote port. Use the following command format: 
    nvme discover --transport fc --traddr <rportwwnn>:<rportwwnp> --host-traddr <lportwwnn>:<lportwwnp>

    In the previous, <rportwwnn> and <rportwwnp> are the remote port world wide namespace name and port, and <rportwwnn> and <rportwwnp> are the local world wide namespace name and port.

    For example, the following command
    # nvme discover --transport fc --traddr nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b --host-traddr nn-0x200000109acfgcad:pn-0x100000109acfgcad

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  fc
adrfam:  fibre-channel
subtype: nvme subsystem
treq:    not specified
portid:  0
trsvcid: none
subnqn:  nqn.1432-01.com.netapp:sn.c44dd3bsce4245edc1adc021cb11a0f6:test_broadcom
traddr:  nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b
  4. Establish a connection to the device represented by the subnqn listed in the discovery response. Use the following command format: 
    nvme connect --transport fc --traddr <rportwwnn>:<rportwwnp> --host-traddr <lportwwnn>:<lportwwnp> -n <subnqn> -k <s>
    In the previous,
    • <rportwwnn> and <rportwwnp> are the remote port world wide namespace name and port,
    • <rportwwnn> and <rportwwnp> are the local world wide namespace name and port,
    • <subnqn> is the value of the subnqn parameter generated by the discovery command,
    • <s> is the keep-alive time in seconds for the command to wait for a response from the remote NVMe device before the command generates a timeout message.
    For example, the following command establishes a connection to a remote port with a 5 second timer. 
    nvme connect --transport fc --traddr nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b --host-traddr nn-0x200000109acfgcad:pn-0x100000109acfgcad -n nqn.1432-01.com.netapp:sn.c44dd3bsce4245edc1adc021cb11a0f6:test_broadcom -k 5
  5. Verify that the storage is now available for the host to use. For example: 
    nvme list -v
Subsystem        Subsystem-NQN                                                                                    Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys1     nqn.1432-01.com.netapp:sn.c44dd3bsce4245edc1adc021cb11a0f6:test_broadcom                             nvme1

Device   SN                   MN                                       FR       TxPort Asdress        Slot   Subsystem    Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------
nvme1    81EYIJSh2VMMAAAAAAAB NetApp ONTAP Controller                  FFFFFFFF fc     traddr nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b,host-traddr nn-0x200000109acfgcad:pn-0x100000109acfgcad    nvme-subsys1 nvme1n1

Device       Generic      NSID       Usage                      Format           Controllers
------------ ------------ ---------- -------------------------- ---------------- ----------------
/dev/nvme1n1 /dev/ng1n1   0x1        107.37  GB / 107.37  GB      4 KiB +  0 B   nvme1

Discovering and Connecting to Targets from Qlogic Hosts

The following procedure describes how to discover and connect to an NVMe/FC target from an Oracle Linux host using a Marvell Qlogic Fibre Channel adapter.

  1. Ensure that the qla2xxx Qlogic module is installed and loaded. 
    modprobe -r qla2xxx
modprobe qla2xxx
  2. Identify the WWNN and WWPN of the local and remote ports. Use the following command: 
    dmesg | grep traddr
    For example, the following shows a host with one local port and one remote ports:
    dmesg | grep traddr

[    6.139862] qla2xxx [0000:04:00.0]-ffff:0: register_localport: host-traddr=nn-0x200000109acfgcad:pn-0x100000109acfgcad on portID:10700
[    6.241762] qla2xxx [0000:04:00.0]-2102:0: qla_nvme_register_remote: traddr=nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b PortID:01050d
  3. Run a discovery command to identify NVMe SSD devices available on the remote port. Use the following command format: 
    nvme discover --transport fc --traddr <rportwwnn>:<rportwwnp> --host-traddr <lportwwnn>:<lportwwnp>

    In the previous, <rportwwnn> and <rportwwnp> are the remote port world wide namespace name and port, and <rportwwnn> and <rportwwnp> are the local world wide namespace name and port.

    For example, the following command
    # nvme discover --transport fc --traddr nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b --host-traddr nn-0x200000109acfgcad:pn-0x100000109acfgcad

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  fc
adrfam:  fibre-channel
subtype: nvme subsystem
treq:    not specified
portid:  0
trsvcid: none
subnqn:  nqn.1432-01.com.netapp:sn.c44dd3bsce4245edc1adc021cb11a0f6:test_qlogic
traddr:  nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b
  4. Establish a connection to the device represented by the subnqn listed in the discovery response. Use the following command format: 
    nvme connect --transport fc --traddr <rportwwnn>:<rportwwnp> --host-traddr <lportwwnn>:<lportwwnp> -n <subnqn> -k <s>
    In the previous,
    • <rportwwnn> and <rportwwnp> are the remote port world wide namespace name and port,
    • <rportwwnn> and <rportwwnp> are the local world wide namespace name and port,
    • <subnqn> is the value of the subnqn parameter generated by the discovery command,
    • <s> is the keep-alive time in seconds for the command to wait for a response from the remote NVMe device before the command generates a timeout message.
    For example, the following command establishes a connection to a remote port with a 5 second timer. 
    nvme connect --transport fc --traddr nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b --host-traddr nn-0x200000109acfgcad:pn-0x100000109acfgcad -n nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b:test_qlogic -k 5
  5. Verify that the storage is now available for the host to use. For example: 
    nvme list -v
Subsystem        Subsystem-NQN                                                                                    Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys1     nqn.1432-01.com.netapp:sn.c44dd3bsce4245edc1adc021cb11a0f6:test_qlogic                             nvme1

Device   SN                   MN                                       FR       TxPort Asdress        Slot   Subsystem    Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------
nvme1    81EYIJSh2VMMAAAAAAAB NetApp ONTAP Controller                  FFFFFFFF fc     traddr nn-0x2014c146dc13a14b:pn-0x2015c146dc13a14b,host-traddr nn-0x200000109acfgcad:pn-0x100000109acfgcad    nvme-subsys1 nvme1n1

Device       Generic      NSID       Usage                      Format           Controllers
------------ ------------ ---------- -------------------------- ---------------- ----------------
/dev/nvme1n1 /dev/ng1n1   0x1        107.37  GB / 107.37  GB      4 KiB +  0 B   nvme1