Create and Deploy on KVM

For complete KVM documentation, refer to https://www.linux-kvm.org/page/Documents.

Set the following options in the BIOS of your Oracle Server:

Under IO Virtualization, enable virtualization (VT-d) and SR-IOV.
Under CPU Power Management, disable C3 and C6 reporting.
Under CPU Power Management, set the Energy option to Performance

Linux Server and BIOS Configuration

Disable irqbalance.
```
sudo systemctl disable --now irqbalance
```
Isolate CPU cores from kernel scheduling.
On a system with 96 cores, if you want to isolate cores 2 through 47 and cores 50 through 95, use this command to add the appropriate boot parameters.
```
sudo grubby --update-kernel=ALL --args="isolcpus=2-47,50-95 nohz_full=2-47,50-95 rcu_nocbs=2-47,50-95 intel_iommu=on iommu=pt tuned.non_isolcpus=00030000,00000003 intel_pstate=disable"
```
Reboot your server to activate these kernel parameters.
Use the command cat /proc/cmdline to verify your kernel parameters.

Host SR-IOV Configuration

In the file /etc/rc.d/rc.local, set the number of VFs needed per media-interface.
This number is proportional to the number of virtual SBCs that can be launched on this host machine. For example, if launching 2 virtual machines with 2 media ports each, then 2 VFs need to be enabled.

Note:
In the examples below, the physical interface enp175s0f0 and enp175s0f1 are on PCI B.D.F. af:00.0 and af:00.1 respectively. Change these values according to your own setup.
```
echo 2 > /sys/bus/pci/devices/0000\:af\:00.0/sriov_numvfs
echo 2 > /sys/bus/pci/devices/0000\:af\:00.1/sriov_numvfs
```

In the file /etc/rc.d/rc.local, set unique MAC addresses per VF.

The MAC addresses should be unique across your network and across all your virtual machines.

ip link set enp175s0f0 vf 0 mac ee:07:94:7a:a7:81
ip link set enp175s0f0 vf 1 mac ee:07:94:7a:a7:82
ip link set enp175s0f1 vf 0 mac 5a:b0:f8:ce:04:51
ip link set enp175s0f1 vf 1 mac 5a:b0:f8:ce:04:52

In the file /etc/rc.d/rc.local, enable trust mode for each VF.

ip link set enp175s0f0 vf 0 trust on
ip link set enp175s0f0 vf 1 trust on
ip link set enp175s0f1 vf 0 trust on
ip link set enp175s0f1 vf 1 trust on

In the file /etc/rc.d/rc.local, enable link state for each VF.

ip link set enp175s0f0 vf 0 state auto
ip link set enp175s0f0 vf 1 state auto
ip link set enp175s0f1 vf 0 state auto
ip link set enp175s0f1 vf 1 state auto

In the file /etc/rc.d/rc.local, disable spoof checking for each VF if you will run these SBCs in an HA environment.
In HA environments, the SBC configures additional MAC addresses on the active node.
```
ip link set enp175s0f0 vf 0 spoof off
ip link set enp175s0f0 vf 1 spoof off
ip link set enp175s0f1 vf 0 spoof off
ip link set enp175s0f1 vf 1 spoof off
```
Make the rc.local file executable.
```
chmod +x /etc/rc.d/rc.local
```
Blacklist the i40evf and iavf drivers.
Addend the following to /etc/modprobe.d/blacklist.conf:
```
blacklist i40evf
blacklist iavf
```
Reboot the host machine.

Virtual Machine Setup

Install the Virtualization Host group and virt-install.

# yum groupinstall "Virtualization Host"
# yum install virt-install

Extract the image.

# tar xvf nnSCZ1000.64-img-vm_kvm.tar
nnSCZ1000.64-img-vm_kvm.ovf
nnSCZ1000.64-img-vm_kvm.qcow2
legal.txt

Create the XML configuration file used to create the virtual machine.

In the configuration file, set the amount of memory you want to use.

In this example, the virtual machine will use 48GB of memory

<domain type='kvm'>
  <name>sbc1</name>
  <memory unit='KiB'>50331648</memory>
  <currentMemory unit='KiB'>50331648</currentMemory>
  . . .
</domain>

Set the CPU pinning.

<vcpu placement='static'>12</vcpu>
<cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='7'/>
    <vcpupin vcpu='6' cpuset='8'/>
    <vcpupin vcpu='7' cpuset='9'/>
    <vcpupin vcpu='8' cpuset='10'/>
    <vcpupin vcpu='9' cpuset='11'/>
    <vcpupin vcpu='10' cpuset='12'/>
    <vcpupin vcpu='11' cpuset='13'/>
    <emulatorpin cpuset='2-13'/>
</cputune>

Set the NUMA node memory pinning.
This NUMA 'nodeset' value depends on the above CPU pinning cores assigned. In this example, CPU cores 2 to 13 belong to NUMA node 0, so here 0 is assigned as the NUMA nodeset value.
```
<numatune>
    <memory mode='strict' nodeset='0'/>
</numatune>
```

Set the source file to the qcow2 image you previously extracted.

<disk type='file' device='disk'>
    <driver name='qemu' type='qcow2' />
    <source file='/home/vsbc_images/nnSCZ1000-img-vm_kvm_node0.qcow2'/>
    <target dev='vda' bus='virtio'/>
    <alias name='disk0'/>
    <address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
</disk>

Modify the SR-IOV VF's PCI identifiers.

<hostdev mode='subsystem' type='pci' managed='yes'>
    <driver name='vfio'/>
    <source>
        <address domain='0x0000' bus='0xaf' slot='0x02' function='0x0'/>
    </source>
    <alias name='hostdev0'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
    <driver name='vfio'/>
    <source>
        <address domain='0x0000' bus='0xaf' slot='0x06' function='0x0'/>
    </source>
    <alias name='hostdev1'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
</hostdev>

Set the MAC addresses for each network interface of type bridge.

<interface type='bridge'>
    <mac address='52:54:00:22:9a:59'/>
    <source bridge='br-mgmt0'/>
    <model type='virtio'/>
    <alias name='net0'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

Save the configuration file.

Create the virtual machine.
```
virsh create <config-file.xml>
```
Connect to the virtual machine.
```
virsh console <domain>
```