1. Authenticate the pcs cluster configuration tool for the hacluster user on each node in your configuration. To do this, run the following command on one of the nodes that will form part of the cluster:

   sudo pcs cluster auth node1node2 -u hacluster

   Replace node1 and node2 with the resolvable hostnames of the nodes that will form part of the cluster. The tool will prompt you to provide a password for the hacluster user. You should provide the password that you set for this user when you installed and configured the Pacemaker software on each node.

2. To create the cluster, use the pcs cluster setup command. You must specify a name for the cluster and the resolvable hostnames for each node in the cluster:

   sudo pcs cluster setup --name pacemaker1 node1 node2

   Replace pacemaker1 with an appropriate name for the cluster. Replace node1 and node2 with the resolvable hostnames of the nodes in the cluster.

3. Start the cluster on all nodes. You can do this manually using the pcs command:

   sudo pcs cluster start --all

   You can also do this by starting the pacemaker and corosync services from systemd:

   sudo systemctl start pacemaker.service
   sudo systemctl start corosync.service

   Optionally, you can enable these services to start at boot time, so that if a node reboots it automatically rejoins the cluster:

   sudo systemctl enable pacemaker.service
   sudo systemctl enable corosync.service

   Some users prefer not to enable these services, so that a node failure resulting in a full system reboot can be properly debugged before it rejoins the cluster.

1. Fencing is an advanced feature that helps protect your data from being corrupted by nodes that may be failing or unavailable. Pacemaker uses the term stonith (shoot the other node in the head) to describe fencing options. Since this configuration depends on particular hardware and a deeper understanding of the fencing process, we recommend disabling the fencing feature for this example.

   sudo pcs property set stonith-enabled=false

   Fencing is an important part of setting up a production level HA cluster and is disabled in this example to keep things simple. If you intend to take advantage of stonith, see Fencing Configuration for more information.

2. Since this example is a two-node cluster, you can disable the no-quorum policy, as quorum requires a minimum of three nodes to make any sense. Quorum is only achieved when more than half of the nodes agree on the status of the cluster. In this example, quorum can never be reached, so configure the cluster to ignore the quorum state:

   sudo pcs property set no-quorum-policy=ignore

3. Configure a migration policy. In this example we configure the cluster to move the service to a new node after a single failure:

   sudo pcs resource defaults migration-threshold=1

Creating a service and testing failover

Services are created and are usually configured to run a resource agent that is responsible for starting and stopping processes. Most resource agents are created according to the OCF (Open Cluster Framework) specification defined as an extension for the Linux Standard Base (LSB). There are many handy resource agents for commonly used processes included in the resource-agents packages, including a variety of heartbeat agents that track whether commonly used daemons or services are still running.

In this example we set up a service that uses a Dummy resource agent created precisely for the purpose of testing Pacemaker. We use this agent because it requires the least possible configuration and does not make any assumptions about your environment or the types of services that you intend to run with Pacemaker.

1. To add the service as a resource, use the pcs resource create command. Provide a name for the service. In the example below, we use the name dummy_service for this resource:

   sudo pcs resource create dummy_service ocf:pacemaker:Dummy op monitor interval=120s

   To invoke the Dummy resource agent, a notation (ocf:pacemaker:Dummy) is used to specify that it conforms to the OCF standard, that it runs in the pacemaker namespace and that the Dummy script should be used. If you were configuring a heartbeat monitor service for an Oracle Database, you might use the ocf:heartbeat:oracle resource agent.

   The resource is configured to use the monitor operation in the agent and an interval is set to check the health of the service. In this example we set the interval to 120s to give the service some time to fail while you are demonstrating failover. By default, this is usually set to 20 seconds, but may be modified depending on the type of service and your own environment.

2. As soon as you create a service, the cluster attempts to start the resource on a node using the resource agent's start command. You can see the resource start and run status by running the pcs status command:

   sudo pcs status

   Cluster name: pacemaker1
   Stack: corosync
   Current DC: node1 (version 1.1.16-12.el7-94ff4df) - partition with quorum
   Last updated: Wed Jan 17 06:35:18 2018
   Last change: Wed Jan 17 03:08:00 2018 by root via cibadmin on node1
   
   2 nodes configured
   1 resource configured
   
   Online: [ node2 node1 ]
   
   Full list of resources:
   
    dummy_service   (ocf::pacemaker:Dummy): Started node2
   
   Daemon Status:
     corosync: active/enabled
     pacemaker: active/enabled
     pcsd: active/enabled

3. Simulate service failure by force stopping the service directly, using crm_resource, so that the cluster is unaware that the service has been manually stopped.

   sudo crm_resource --resource dummy_service --force-stop

4. Run crm_mon in interactive mode so that you can wait until you see the node fail and a Failed Actions message is displayed. You should see the service restart on the alternate node.

   sudo crm_mon

   Stack: corosync
   Current DC: node1 (version 1.1.16-12.el7-94ff4df) - partition with quorum
   Last updated: Wed Jan 17 06:41:04 2018
   Last change: Wed Jan 17 06:39:02 2018 by root via cibadmin on node1
   
   2 nodes configured
   1 resource configured
   
   Online: [ node2 node1 ]
   
   Active resources:
   
   dummy_service    (ocf::pacemaker:Dummy): Started node1
   
   Failed Actions:
   * dummy_service_monitor_120000 on node2 'not running' (7): call=16, status=complete, exitreason='none',
       last-rc-change='Wed Jan 17 06:41:02 2018', queued=0ms, exec=0ms

   You can use the Ctrl-C key combination to exit out of crm_mon at any point.

5. You can try to reboot the node where the service is running to see that failover also occurs in the case of node failure. Note that if you have not enabled the corosync and pacemaker services to start on boot, you may need to start the service on the node that you have rebooted, manually. For example:

   sudo pcs cluster start node1

Intelligent Platform Management Interface (IPMI) is an interface to a subsystem that provides management features of the host system's hardware and firmware and includes facilities to power cycle a system over a dedicated network without any requirement to access the system's operating system. The fence_ipmilan fencing agent can be configured for the cluster so that stonith can be achieved across the IPMI LAN.

If your systems are configured for IPMI, you can run the following commands on one of the nodes in the cluster to enable the ipmilan fencing agent and to configure stonith for both nodes:

sudo pcs stonith create ipmilan_n1_fencing fence_ipmilan pcmk_host_list=node1 delay=5 \
ipaddr=203.0.113.1 login=root passwd=password lanplus=1 op monitor interval=60s

sudo pcs stonith create ipmilan_n2_fencing fence_ipmilan pcmk_host_list=node2 \
ipaddr=203.0.113.2 login=root passwd=password lanplus=1 op monitor interval=60s

In the above example, the host named node1 has an IPMI LAN interface configured on the IP 203.0.113.1. The host named node2 has an IPMI LAN interface configured on the IP 203.0.113.2. The root user password for the IPMI login on both systems is specified here as password. In each instance, you should replace these configuration variables with the appropriate information to match your own environment.

Note that the delay option should only be set to one node. This helps to ensure that in the rare case of a fence race condition only one node is killed and the other continues to run. Without this option set, it is possible that both nodes believe they are the only surviving node and simultaneously reset each other.

The SCSI Fencing agent is used to provide storage level fencing. This protects storage resources from being written to by two nodes at the same time, using SCSI-3 PR (Persistent Reservation). Used in conjunction with a watchdog service, a node can be reset automatically via stonith when it attempts to access the SCSI resource without a reservation.

To configure an environment in this way, install the watchdog service on both nodes and copy the provided fence_scsi_check script to the watchdog configuration before enabling the service:

sudo yum install watchdog
sudo cp /usr/share/cluster/fence_scsi_check /etc/watchdog.d/
sudo systemctl enable --now watchdog

To use this fencing agent, you must also enable the iscsid service provided in the iscsi-initiator-utils package on both nodes:

sudo yum install -y iscsi-initiator-utils
sudo systemctl enable --now iscsid

Once both nodes are configured with the watchdog service and the iscsid service, you can configure the fence_scsi fencing agent on one of the cluster nodes to monitor a shared storage device, such as an iSCSI target. For example:

sudo pcs stonith create scsi_fencing fence_scsi pcmk_host_list="node1 node2" \
 devices="/dev/sdb" meta provides="unfencing"

In the example, node1 and node2 are the hostnames of the nodes in the cluster and /dev/sdb is the shared storage device. You should replace these variables with the appropriate information to match your own environment.

Storage Based Death (SBD) is a daemon that can run on a system and monitor shared storage and that can use a messaging system to track cluster health. SBD can trigger a reset in the event that the appropriate fencing agent determines that stonith should be implemented.

To set up and configure SBD fencing, stop the cluster by running the following command on one of the nodes:

sudo pcs cluster stop --all

On each node, install and configure the SBD daemon:

sudo yum install sbd

Edit /etc/sysconfig/sbd to set the SBD_DEVICE parameter to identify the shared storage device. For example, if your shared storage device is available on /dev/sdc, edit the file to contain the line:

SBD_DEVICE="/dev/sdc"

Enable the SBD service in systemd:

sudo systemctl enable --now sbd

On one of the nodes, create the SBD messaging layout on the shared storage device and confirm that it is in place. For example, to set up and verify messaging on the shared storage device at /dev/sdc, run the following commands:

sudo sbd -d /dev/sdc create
sudo sbd -d /dev/sdc list

Finally, start the cluster and configure the fence_sbd fencing agent for the shared storage device. For example, to configure the shared storage device, /dev/sdc, run the following commands on one of the nodes:

sudo pcs cluster start --all
sudo pcs stonith create sbd_fencing fence_sbd devices=/dev/sdc

IF-MIB fencing takes advantage of SNMP to access the IF-MIB on an Ethernet network switch and to shutdown the port on the switch to effectively take a host offline. This leaves the host running, but disconnects it from the network. It is worth bearing in mind that any FibreChannel or InfiniBand connections could remain intact, even after the Ethernet connection has been terminated, which could mean that data made available on these connections could still be at risk. As a result, it is best to configure this as a fallback fencing mechanism. See Configuring Fencing Levels for more information on how to use multiple fencing agents together to maximise stonith success.

To configure IF-MIB fencing, ensure that your switch is configured for SNMP v2c at minimum and that SNMP SET messages are enabled. For example, on an Oracle Switch, via the ILOM CLI, you could run:

sudo set /SP/services/snmp/ sets=enabled
sudo set /SP/services/snmp/ v2c=enabled

On one of the nodes in your cluster, configure the fence_ifmib fencing agent for each node in your environment. For example:

sudo pcs stonith create ifmib_n1_fencing fence_ifmib pcmk_host_list=node1 \
ipaddr=203.0.113.10 community=private port=1 delay=5 op monitor interval=60s

sudo pcs stonith create ifmib_n2_fencing fence_ifmib pcmk_host_list=node2 \
ipaddr=203.0.113.10 community=private port=2 op monitor interval=60s

In the above example, the switch SNMP IF-MIB is accessible at the IP address 203.0.113.10. The host node1 is connected to port 1 on the switch. The host node2 is connected to port 2 on the switch. You should replace these variables with the appropriate information to match your own environment.

If you have configured multiple fencing agents, you may want to set different fencing levels. Fencing levels allow you to prioritize different approaches to fencing and can provide a valuable mechanism to provide fallback options should a default fencing approach fail.

Each fencing level is attempted in ascending order starting from level 1. If the fencing agent configured for a particular level fails, the fencing agent from the next level is attempted instead.

For example, you may wish to configure IPMI-LAN fencing at level 1, but fallback to IF-MIB fencing as a level 2 option. Using the example configurations from IPMI LAN Fencing and IF-MIB Fencing, you could run the following commands on one of the nodes to set the fencing levels for each configured agent:

sudo pcs stonith level add 1 node1 ipmilan_n1_fencing
sudo pcs stonith level add 1 node2 ipmilan_n2_fencing
sudo pcs stonith level add 2 node1 ifmib_n1_fencing
sudo pcs stonith level add 2 node2 ifmib_n2_fencing