Backup and Restore

To protect your data so that it can be recovered in the event of a disaster, your system and database administrators must create a backup and recovery plan for Oracle Communications Unified Assurance. This plan is part of your organization's overall policies.

This document provides guidance for the Unified Assurance backup and recovery plan, including the full manual offline backup and restore process, and how to set up scheduled online backups for individual components. You may need to modify these guidelines according to your organization's operational and management processes and requirements.

Best Practices and Expectations for Backup and Restore

Make a mount point available on the database server and presentation server for conducting backups. This document uses /mnt/backups, which you should replace with your backup mount point if you use a different one.
On systems with databases, you must shut down the database to back up the database files, or use a database backup utility while the database is active.
The Event Analytics and Metric databases may be too big to back up locally. Oracle recommends backing them up to a mount on an SAN or NFS instead.
The examples in this document assume:
- An external system is mounted at /mnt/backups.
- All commands are run as root or an equally privileged user.
- The $A1BASEDIR environment variable is set to the directory where you installed Unified Assurance. This is typically /opt/assure1.

Manual Full Offline Backup and Restore

This section explains how to perform the simplest backup, where you shut down all applications and perform a full offline backup. You should only do this during a scheduled maintenance window.

Performing a Full Offline Backup

Stop all processes:
1. On all servers, run the following commands:
```
source $A1BASEDIR/.assure1_bashrc
$A1BASEDIR/bin/BrokerControl --batch stopall
service assure1-broker stop
```
2. On presentation servers, also run the following commands:
```
service assure1-web stop
service assure1-bus stop
service assure1-db stop
```
3. On database servers, also run the following commands:
```
service assure1-db stop
```
4. On all servers, verify that all processes are stopped:
  1. Run the following command:
```
ps aux | grep assure1
```
  2. For anything still running, run the following command:
```
kill -9 <processID>
```
    where <processID> is the ID of the process.
Create a full backup of Unified Assurance:
```
tar czpf /mnt/backups/assure1.tgz $A1BASEDIR
```
This may take some time.

Note:

If Unified Assurance is running as a virtual machine, you can instead create a snapshot and store it as a backup.
Start all processes:
1. On database servers, run the following command:
```
service assure1-db start
```
2. On presentation servers, run the following command:
```
service assure1-db start
service assure1-bus start
service assure1-web start
```
3. On all servers, also run the following command:
```
service assure1-broker start
```

Performing a Full Offline Restore

If you are restoring onto a server that is not new, stop all processes:
1. On all servers, run the following commands:
```
$A1BASEDIR/bin/BrokerControl --batch stopall
service assure1-broker stop
```
2. On presentation servers, also run the following commands:
```
service assure1-web stop
service assure1-bus stop
service assure1-db stop
```
3. On database servers, also run the following commands:
```
service assure1-db stop
```
4. On all servers, verify that all processes are stopped:
  1. Run the following command:
```
ps aux | grep assure1
```
  2. For anything still running, run the following command:
```
kill -9 <processID>
```
    where <processID> is the ID of the process.
If you are restoring onto a new server, ensure that the initial installation steps have been performed, up to installing the RPM. You do not need to run SetupWizard.

See Installing Unified Assurance on a Single Server in Unified Assurance Installation Guide for information about getting the collection file, extracting it, and installing the RPM.
Restore Unified Assurance by running the following command:
```
tar xzf /mnt/backups/assure1.tgz
```

If you are restoring onto a new server, restore the startup scripts:

On all servers, run the following commands:

cp $A1BASEDIR/distrib/startup/assure1/init.d/assure1-broker.service.local /etc/systemd/system/assure1-broker.service
systemctl --quiet enable assure1-broker.service

On presentation servers, run the following commands:

cp $A1BASEDIR/distrib/startup/vendorRabbitMQ/init.d/assure1-bus.service.local /etc/systemd/system/assure1-bus.service
systemctl --quiet enable assure1-bus.service
cp $A1BASEDIR/distrib/startup/vendorApache/init.d/assure1-web.service.local /etc/systemd/system/assure1-web.service
systemctl --quiet enable assure1-web.service
p $A1BASEDIR/distrib/startup/vendorMySQL/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service
systemctl --quiet enable assure1-db.service

On database servers, run the following commands:

cp $A1BASEDIR/distrib/startup/vendorMySQL/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service
systemctl --quiet enable assure1-db.service

On all servers, run the following command:
```
systemctl daemon-reload
```

Start all processes:
1. On database servers, run the following command:
```
service assure1-db start
```
2. On presentation servers, run the following command:
```
service assure1-db start
service assure1-bus start
service assure1-web start
```
3. On all servers, also run the following command:
```
service assure1-broker start
```

Scheduling Online Backups and Restores

You can set up scheduled backups for the Unified Assurance home directory and the databases independently, and restore them individually as needed.

The following database schemas can be backed up and restored:

MySQL, with separate Assure1, Event, Vision, and Grafana schemas
InfluxDB for the Metric database
Elasticsearch, with separate Event Analytics and log schemas
Neo4j for the Graph database

In multi-server environments, you must set up backups on each database server and restore the individual servers as needed. In single-server environments, or if you have multiple MySQL database schemas on a single server, the backup and restore processes cover all of them in one pass.

For example, your multi-server environment includes, among the servers for other databases and components, three separate MySQL database servers. The first is for the Assure1 schema, the second is for the Event schema, and the third is for both the Vision and Grafana schemas. You must set up scheduled backups on all three servers, but the backups on the third server include both Vision and Grafana. If the third server goes down, you only have to run through the restore process once to restore both schemas.

Preparing for Scheduled Backups

Create the mount directories for scheduled backups. Some of the directories may not be needed, depending on whether the backup is on a presentation or database server.

For example, run the following commands:

mkdir -p /mnt/backups/assure1/full  /mnt/backups/assure1/inc{1..6}
mkdir -p /mnt/backups/mysql/full  /mnt/backups/mysql/inc{1..6}
mkdir -p /mnt/backups/influxdb/full /mnt/backups/influxdb/inc{1..6}
mkdir -p /mnt/backups/elastic/
mkdir -p /mnt/backups/neo4j/

Backing Up and Restoring the Unified Assurance Home Directory

This section provides examples for backing up and restoring your Unified Assurance home directory.

Backing Up the Unified Assurance Home Directory

To back up your Unified Assurance home directory, set up a cron job to run your full and incremental backups. In this example, the full backup runs on Sunday and incremental backups every subsequent day after that. If your organization has different backup policies, update the schedule accordingly.

The Unified Assurance home directory backup must exclude data from running databases.

For example, create a file named assure1-home in the /etc/cron.d directory with the following content:

0 0 * * 0 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} $A1BASEDIR /mnt/backups/assure1/full && ln -s /mnt/backups/assure1/full /mnt/backups/assure1/latest
0 0 * * 1 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc1 && ln -s /mnt/backups/assure1/inc1 /mnt/backups/assure1/latest
0 0 * * 2 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc2 && ln -s /mnt/backups/assure1/inc2 /mnt/backups/assure1/latest
0 0 * * 3 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc3 && ln -s /mnt/backups/assure1/inc3 /mnt/backups/assure1/latest
0 0 * * 4 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc4 && ln -s /mnt/backups/assure1/inc4 /mnt/backups/assure1/latest
0 0 * * 5 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc5 && ln -s /mnt/backups/assure1/inc5 /mnt/backups/assure1/latest
0 0 * * 6 assure1 rsync -a --delete --exclude={'var/mysql_data','var/influxdb','var/elasticsearch','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc6 && ln -s /mnt/backups/assure1/inc6 /mnt/backups/assure1/latest

Restoring the Unified Assurance Home Directory

If you are restoring onto a server that is not new, stop all processes:
1. On all servers, run the following commands:
```
$A1BASEDIR/bin/BrokerControl --batch stopall
service assure1-broker stop
```
2. On presentation servers, also run the following commands:
```
service assure1-web stop
service assure1-bus stop
service assure1-db stop
```
3. On database servers, also run the following command:
```
service assure1-db stop
```

Restore the Unified Assurance home directory:

rsync -a --delete /mnt/backups/assure1/latest $A1BASEDIR

If you are restoring onto a new server, restore the startup scripts:

On all servers, run the following commands:

cp $A1BASEDIR/distrib/startup/assure1/init.d/assure1-broker.service.local /etc/systemd/system/assure1-broker.service
systemctl --quiet enable assure1-broker.service

On presentation servers, run the following commands:

cp $A1BASEDIR/distrib/startup/vendorRabbitMQ/init.d/assure1-bus.service.local /etc/systemd/system/assure1-bus.service
systemctl --quiet enable assure1-bus.service
cp $A1BASEDIR/distrib/startup/vendorApache/init.d/assure1-web.service.local /etc/systemd/system/assure1-web.service
systemctl --quiet enable assure1-web.service
p $A1BASEDIR/distrib/startup/vendorMySQL/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service
systemctl --quiet enable assure1-db.service

On database servers, run the following commands:

cp $A1BASEDIR/distrib/startup/vendorMySQL/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service
systemctl --quiet enable assure1-db.service

On all servers, run the following command:
```
systemctl daemon-reload
```

Start all processes:
1. On database servers, run the following command:
```
service assure1-db start
```
2. On presentation servers, run the following command:
```
service assure1-db start
service assure1-bus start
service assure1-web start
```
3. On all servers, also run the following command:
```
service assure1-broker start
```

Backing Up and Restoring a MySQL Database

This section provides examples for backing up and restoring your MySQL Assure1 and Event databases.

See MySQL Enterprise Backup User's Guide for complete information about backing up a MySQL database, including command usage and options.

Backing Up a MySQL Database

To back up your MySQL Assure1 and Events databases, set up a cron job to run your full and incremental backups. In this example, the full backup runs on Sunday and incremental backups every subsequent day after that. If your organization has different backup policies, update the schedule accordingly.

For example, create a file named assure1-mysql in the /etc/cron.d directory with the following content:

0 1 * * 0 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full backup
0 1 * * 1 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --incremental --incremental-base=dir:/mnt/backups/mysql/full --incremental-backup-dir=/mnt/backups/mysql/inc1 backup
0 1 * * 2 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --incremental --incremental-base=dir:/mnt/backups/mysql/inc1 --incremental-backup-dir=/mnt/backups/mysql/inc2 backup
0 1 * * 3 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --incremental --incremental-base=dir:/mnt/backups/mysql/inc2 --incremental-backup-dir=/mnt/backups/mysql/inc3 backup
0 1 * * 4 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --incremental --incremental-base=dir:/mnt/backups/mysql/inc3 --incremental-backup-dir=/mnt/backups/mysql/inc4 backup
0 1 * * 5 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --incremental --incremental-base=dir:/mnt/backups/mysql/inc4 --incremental-backup-dir=/mnt/backups/mysql/inc5 backup
0 1 * * 6 assure1 $A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --incremental --incremental-base=dir:/mnt/backups/mysql/inc5 --incremental-backup-dir=/mnt/backups/mysql/inc6 backup

Restoring a MySQL Database

If you are restoring onto a server that is not new, stop the database:
```
service assure1-db stop
```

Apply any changes that were made while the backup was running:

$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full apply-log

Apply each subsequent incremental backup, stopping when you reach the day before the current day:

$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full --incremental-backup-dir=/mnt/backups/mysql/inc1 apply-incremental-backup
$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full --incremental-backup-dir=/mnt/backups/mysql/inc2 apply-incremental-backup
$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full --incremental-backup-dir=/mnt/backups/mysql/inc3 apply-incremental-backup
$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full --incremental-backup-dir=/mnt/backups/mysql/inc4 apply-incremental-backup
$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full --incremental-backup-dir=/mnt/backups/mysql/inc5 apply-incremental-backup
$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full --incremental-backup-dir=/mnt/backups/mysql/inc6 apply-incremental-backup

For example, if you are applying backups on a Thursday, stop after applying inc3.

Move the contents of the MySQL data directory to a temporary location:
```
mkdir $A1BASEDIR/tmp/mysql
mv $A1BASEDIR/var/mysql_data/* $A1BASEDIR/tmp/mysql
```
The restore command requires an empty MySQL data directory.

Restore the database:

$A1BASEDIR/vendor/mysql/bin/mysqlbackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup-dir=/mnt/backups/mysql/full --datadir=$A1BASEDIR/var/mysql_data copy-back

Start the database:
```
service assure1-db start
```
Verify that the data was restored correctly. For example, check to see that the Events list in the UI is displaying the expected data.
Delete the temporary data directory and its contents.
```
rm -r $A1BASEDIR/tmp/mysql
```

Backing Up and Restoring the InfluxDB Database

This section provides examples for backing up and restoring your InfluxDB Metric database.

See Back up and restore data in the InfluxDB documentation for complete information about backing up an InfluxDB database, including command usage and options.

Backing Up an InfluxDB Database

To back up your InfluxDB Metric database, set up a cron job to run your full and incremental backups. In this example, the full backup runs on Sunday and incremental backups every subsequent day after that. If your organization has different backup policies, update the schedule accordingly.

For example, create a file named assure1-influxdb in the /etc/cron.d directory with the following content:

0 2 * * 0 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/full
0 2 * * 1 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc1
0 2 * * 2 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc2
0 2 * * 3 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc3
0 2 * * 4 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc4
0 2 * * 5 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc5
0 2 * * 6 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc6

Restoring an InfluxDB Database

If you are restoring onto a server that is not new, stop the database:
1. Get the ID for the InfluxDB service:
```
$A1BASEDIR/bin/BrokerControl --batch listservices | grep InfluxDB
```
2. Stop the database, using the returned ID:
```
$A1BASEDIR/bin/BrokerControl --batch stopservice <ID>
```

Restore the full backup and each subsequent incremental backup, stopping when you reach the day before the current day:

$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/full
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc1
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc2
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc3
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc4
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc5
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc6

For example, if you are applying backups on a Thursday, stop after applying inc3.

Start the database:

Get the ID for the InfluxDB service:

$A1BASEDIR/bin/BrokerControl --batch listservices | grep InfluxDB

Start the database, using the returned ID:

$A1BASEDIR/bin/BrokerControl --batch startservice <ID>

Backing Up and Restoring the Elasticsearch Databases

This section provides examples for backing up and restoring your Elasticsearch databases for Event Analytics.

See Snapshot and restore in the Elasticsearch documentation for complete information about backing up an Elasticsearch database, including command usage and options.

Preparing the Elasticsearch Backup

Add the mount directory as an Elasticsearch repository path in an override file:
1. In the command line, switch to the assure1 user:
```
su - assure1
```
2. Navigate to the $A1BASEDIR/etc/elasticsearch.d/overrides directory:
```
cd $A1BASEDIR/etc/elasticsearch.d/overrides
```
  If the elasticsearch.d and overrides directories do not already exist, create them:
```
mkdir $A1BASEDIR/etc/elasticsearch.d
mkdir $A1BASEDIR/etc/elasticsearch.d/overrides
cd $A1BASEDIR/etc/elasticsearch.d/overrides
```
3. Add the following line to the elasticsearch.override file:
  
  path.repo: ["/mnt/backups/elastic"]
  
  If the file does not already exist, create it and add the line.
4. Save and close the file.
5. Switch to the root user:
```
su - root
```
6. Run the ConfigHelper application:
```
$A1BASEDIR/bin/ConfigHelper merge-restart Elasticsearch
```
  ConfigHelper merges the override and restarts the ElasticSearch service.
In a browser, log in to the Unified Assurance user interface.
From the Analytics menu, select Events, then Administration, then Console.

Create the backup repository by running the following in the console:

PUT /_snapshot/backup
{
  "type": "fs",
  "settings": {
    "location": "/mnt/backups/elastic"
  }
}

Backing Up the Elasticsearch Databases

In a browser, log in to the Unified Assurance user interface.
From the Analytics menu, select Events, then Administration, then Console.

Run the following in the console:

PUT /_snapshot/backup/<snapshot-{now%2fd}>

Restoring the Elasticsearch Databases

In a browser, log in to the Unified Assurance user interface.
From the Analytics menu, select Events, then Administration, then Console.
Get a list of all indices by running the following command in the console:
```
GET /_cat/indices
```
Identify the indices you want to restore from this list.
Close the indices that are going to be restored by running one of the following in the console:

Caution:

Do not close system indices while running this command.
- To close a single index for Event Analytics, replacing YYYY.MM.DD with the relevant date from the list of indices:
```
POST /eventanalytics-YYYY.MM.DD/_close
```
- To close all indices for Event Analytics:
```
POST /eventanalytics-*/_close
```
- To close a single index for logs, replacing YYYY.MM.DD with the relevant date from the list of indices:
```
POST /filebeat-7.3.1-YYYY.MM.DD/_close
```
- To close all indices for logs:
```
POST /filebeat-*/_close
```
Restore the indices from the snapshot by running the following in the console, replacing and and adding more index names as appropriate:
```
POST /_snapshot/backup/<snapshot_name>/_restore
{
  "indices": "<index1_name>,<index2_name>,..."
}
```

Backing Up and Restoring the Neo4j Database

This section provides examples for backing up and restoring your Neo4j Graph database.

See Backup and restore in Neo4j Operations Manual for complete information about backing up Neo4j database, including command usage and options.

Backing Up a Neo4j Database

To back up your Neo4j database, set up a cron job to run the backup. In this example, the backup runs at 2am every day. If your organization has different backup policies, update the schedule accordingly.

For example, create a file named assure1-neo4j in the /etc/cron.d directory with the following content:

0 2 * * * assure1 env HEAP_SIZE=2G NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin backup --backup-dir=/mnt/backups/neo4j --database=* --pagecache=4G

Restoring a Neo4j Database

To restore your Neo4j database, run the following commands:

NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --from=/mnt/backups/neo4j --database=neo4j --force
NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --from=/mnt/backups/neo4j --database=Graph --force

Title and Copyright Information

System Administrator’s Guide

G10627-01