Database Sharding

Database sharding is a way to organize large sets of data across multiple servers. Sharded data is stored completely separate from other shards and each individual shard has no knowledge of the other, but the Unified Assurance web interface seamlessly correlates and displays data collected from all shards allowing for a completely unified view. The Unified Assurance platform supports database sharding for Events & Historical, and Metric data that is simple to configure and transparent to users.

There are many benefits and use cases when considering sharding a database:

• Load balancing - Unified Assurance applications only talk to the shards needed for lightening the load across the databases.

• Scaling - Big data can be stored in manageable pieces across the multiple database shards supporting larger scale environments.

• Data Storage Compliance - Each shard's data is stored in a standalone fashion and has no knowledge of data from any other shard.

This section details the procedure for installing and configuring new database shards for Events, Historical Events, and Metrics databases within the Unified Assurance platform.

Pre-Planning Shard Data Organization

Event and Historical

Unified Assurance aggregators support different shards but must know at startup what shard it will be communicating with. Within the Unified Assurance platform, events can be sharded in the following ways:

• Collection Point (most common) - Alarm data based on which server received it. This method allows for segregation of data based on geographical regions. For example: Shard 1 is in US and only handles collected events from the US Unified Assurance aggregation servers, while Shard 2 is in Europe and only handles collected events from the Europe Unified Assurance aggregation servers.

• Alarm Type - Alarm data from a certain aggregator, such as SNMP traps, to a certain shard. This method allows for additional analysis or profiling of a certain type of data.

Metric

Unified Assurance metric collectors and pollers support multiple shards based on the device's and metric's shard definition. Shard information can be set at the device or metric level within the Unified Assurance platform and metrics can be sharded in a few different manners:

• Device (most common) - All metrics for certain devices are stored on a specific shard, this includes separation by Device Zone or Device Group. This method allows for segregation of device data by geographical regions or polling clusters. For example: Shard 1 is in US and only handles collected metrics from the US Unified Assurance collection servers, while Shard 2 is in Europe and only handles collected metrics from the Europe Unified Assurance collection servers. (This is the easiest to configure and maintain)

• Metric Type - Metric data of certain types, such as Latency & Packet Loss, are stored on a specific shard. This method allows for additional analysis or profiling of a certain type of data, but additional maintenance is required to keep accurate.

Installation

New Servers

1. Follow the steps in the Preparation section of Installing Unified Assurance on Multiple Servers to get the server ready for Unified Assurance.

2. Follow the steps in the Database Server section of Installing Unified Assurance on Multiple Servers to get Unified Assurance installed.

   Note:

   When asked what roles to install, either one of or both the Database.Event and Database.Metric roles can be entered depending on the type of database being sharded. Additional roles can also be installed, if needed.

3. Wait for the installation to complete.

Existing Servers

1. Log in to the command line of the server that will have the database added to it.

2. Use the Package Manager application to install additional roles with the new database functionality.

   $A1BASEDIR/bin/Package install-role (Comma-separated list of roles)
   

   Note:

   See the All Server Roles section of the Understanding Server Roles guide for a list of roles that can be installed.

3. Wait for the installation to complete.

Sharding Procedure

New Events Shard

1. Navigate to the Databases UI and verify a new shard entry exists for the Event database.

   Configuration -> Databases -> Databases

2. Optionally, run the ApplyEventSchema application with the --Dry-Run option for a preview of changes:

   1. Log the queries that will be run to change the database:

      $A1BASEDIR/bin/ApplyEventSchema --Dry-Run
      

   2. Review the queries in the log file:

      lnav $A1BASEDIR/logs/ApplyEventSchema.log
      

3. Run ApplyEventSchema to update the database table schema, replacing <N> with the correct Event Shard ID:

   $A1BASEDIR/bin/ApplyEventSchema --EventShard <N>
   

4. Organize event aggregators into shards based on pre-planning or policies.

5. Clone necessary aggregators and respective application configurations, updating ShardID to appropriate shard.

6. Verify new events are being inserted into the correct database instance.

New Metric Shard

1. Navigate to the Databases UI and verify a new shard entry exists for the Metric database.

   Configuration -> Databases -> Databases

2. Organize and set Devices' and Metrics' ShardIDs to appropriate shards based on pre-planning or policies.

3. Restart any affected polling clusters.

4. Verify new metrics are being inserted into the correct database instance.

Redundancy

1. Log in to the command line of the server that will have the redundancy added to it.

2. Use the Package Manager application to install the Database.Redundancy role.

   $A1BASEDIR/bin/Package install-role Database.Redundancy
   

3. Wait for the installation to complete.

4. Follow the steps in the Run Redundancy Wizard on Database Servers section of Installing Unified Assurance on Multiple Redundant Servers to configure database redundancy.