12 Data Migration and Dynamic Attribute Mapping between UIM and SmartSearch

This chapter describes how to perform data migration and Dynamic Attribute mapping from UIM to SmartSearch or OpenSearch NoSQL Database. This migration helps you to seamlessly integrate data between UIM and SmartSearch or OpenSearch while designing networks.

Prerequisites

Before you perform SmartSearch data migration, ensure the following are performed:

  • Cancel or complete all BIs that are associated with devices, connectivities, and pipes.
  • Upgrade UIM.
  • Ensure that SmartSearch and OpenSearch are deployed and are running with Authentication disabled. For more information, see "Deploying OpenSearch and OpenSearch Dashboard" and "Deploying SmartSearch".
  • Add lang_pipeline in OpenSearch. For more information, see "Creating Ingest Pipeline for OpenSearch".
  • Create all required indexes or metadata in OpenSearch. For more information, see "Installing OpenSearch" and "Creating SmartSearch Schema".
  • Update the hostAliases section according to your SmartSearch and OpenSearch service setup. Add the target machine details in the /etc/host file as follows:
    <IP ADDRESS>  <instance>.<project>.<hostSuffix> t3.<instance>.<project>.<hostSuffix> admin.<instance>.<project>.<hostSuffix> <instance>.<project>.topology.<hostSuffix>
  • Verify your access to the service URL and database connection as follows:
    1. Open a command line window and login to SQL*Plus for the UIM database.
    2. Verify if UIM schema connection is successful using SQL*Plus.
    3. Ensure that you have admin or system access for UIM schema.
    4. Open a web browser and verify if you can access the SmartSearch API URL: https://<instance>.<project>.topology.<hostSuffix>:<PORT>/<API-VERSION>/index/smartsearch-location.
    5. Verify if JAVA_HOME and PATH are set.
    6. If SmartSearch is SSL enabled, use the Import SSL certificate in JAVA to secure connection.
    7. Download the corresponding .pem file from target machine and run the following command in terminal to import certificate:
      keytool -import -alias common-cert -file commoncert.pem -storetype JKS -keystore $JAVA_HOME/jre/lib/security/cacerts

Running the SmartSearch Migration Script

When you perform ATA migration along with SmartSearch migration, and if you perform data migration from UIM to ATA, you can ignore running the SQL files from migration SQL script as MIG tables are already present in your UIM schema.

If ATA migration is done before you start SmartSearch migration, you must drop all the tables before running the scripts.

If you perform data migration from UIM to SmartSearch, run the migration script as follows:

  1. Go to $WORKSPACEDIR/ata-builder/migration_scripts/Char_Mig_tables directory.

    You can see the following SQL files:

    1. CREATE_CHAR_MIG_TABLE.sql
    2. MIGRATION_CHAR1.sql
    3. MIGRATION_CHAR2.sql
    4. MIGRATION_CHAR3.sql
  2. Run the SQL scripts in UIM schema in the above order.
  3. Make a new directory:
    mkdir $WORKSPACEDIR/<directory_name>
  4. Copy $WORKSPACEDIR/uim-image-builder/staging/cnsdk/uim-db/staging/uim-db-installer/ora_uim_dbtools.jar into the newly created directory.
  5. Extract the jar file as follows:
    jar -xvf ora_uim_dbtools.jar
  6. Modify smartSearchMappings.json for dynamic attribute mapping and save it.

    Note:

    Dynamic attribute mappings support mapping of vendor-specific characteristic for Physical Device. You can map only one characteristic from the Physical Device entity to the vendor field in the OpenSearch index, using the mappings. The equivalent vendor characteristic mappings must be present in topologySpecificationMapping.json.
  7. Modify the required parameters in the smartSearchMigration.sh file. For more information, see "Post-migration Check".
  8. Grant the execute permissions to the smartSearchMigration.sh file. If you do not have modify permissions, update the required parameters and save the file as follows:
    chmod +x smartSearchMigration.sh
  9. Run the migration command using:
    ./smartSearchMigration.sh UIMToSmartSearch

    Note:

    You must wait until the migration script is complete. Or, if you want more options, run ./smartSearchMigration.sh help.

If the migration script fails, and you want to perform the migration again, delete the temp tables as follows:

./smartSearchMigration.sh deleteTable force

After a table is deleted, run the following command to restart the migration:

./smartSearchMigration.sh UIMToSmartSearch

Post-migration Check

Perform the following after a successful data migration to SmartSearch:

  1. Verify or search the following information in the console log:
    **** After Json Data Processed Completed *****
               Total Batch Size: 8
               Total Completed Batch Size: 8
               Total Pending And Failed Batch Size: 0
       **** Data Migration from UIM to Open Search Successfully Completed! ***** 

    Note:

    The Total Pending and Failed Batch Size value is Zero if the data migration is successful.
  2. Contact your administrator to enable Authentication for SmartSearch API and OpenSearch database.
  3. After enabling Authentication, open NPD to verify the search functionality for the corresponding entities such as logical device, physical device, equipment, and so on.

The sample file is as follows:

[
  {
    "name": "Ciena 3928 PD",
    "entityType": "PhysicalDeviceSpecificationDAO",
    "characteristics": [
      {
        "name": "vendor",
        "smartSearchName": "vendor"
      }
    ]
  },
  {
    "name": "Nokia 1800 PD",
    "entityType": "PhysicalDeviceSpecificationDAO",
    "characteristics": [
      {
        "name": "vendor",
        "smartSearchName": "vendor"
      }
    ]
  },
  {
    "name": "Ciena 3928 PD",
    "entityType": "PhysicalDeviceSpecificationDAO",
    "relatedVertices": [
      {
        "vertex": "vendor",
        "value": "Ciena",
        "smartSearchName": "vendor"
      }
    ],
    "characteristics": []
  }
]

The relatedVertices and characteristics sections in the mapping are mutually exclusive. Meaning, you should use only one of them, not both. If both sections are provided, the characteristics section overrides the relatedVertices section.

The relatedVertices section specifies a default value for a characteristic in the mapping, than searching for it from the database. For example, in the above sample, physical devices with the specification Ciena 3928 PD always have Ciena as the value in the vendor field in the index. This ensures that all physical devices with the Ciena 3928 PD specification have Ciena as their vendor in the NPD search.

Table 12-1 Dynamic Attributes

Dynamic Attribute Description
name Name of the entity specification on which the characteristic to be mapped exists. This is a mandatory attribute.
entityType Entity specification DAO. This is a mandatory attribute. The only valid value is PhysicalDeviceSpecificationDAO.
characteristics.name Name of the characteristic that you plan to map with the index attribute.
characteristics.smartSearchName Attribute name of the index where dynamic attribute value is stored in OpenSearch index
relatedVertices.vertex The relatedVertices.vertex value from the ATA mappings. The only supported value is vendor.
relatedVertices.value Manually-entered value for characteristics that will persist in attribute of the index.
relatedVertices.smartSearchName Attribute name of index where dynamic attribute value is stored in OpenSearch index

Open the smartSearchMigration.sh file in terminal or in a text editor, modify the required attributes values, and save the file. The sample file is as follows:

export SMART_SEARCH_BASE_URL="";  #BASE URL format like http or https://<host>:<port>/<api_version>
   export LANG=""; # for example en
   export TENANT_ID=""; # for example tenant1
   export LIMIT=""; #for example 10000
   export BATCH_SIZE_LIMIT=""; # for example 10
   export BULK_LOAD_TIMEOUT="20";   # Maximum processing time per batch operation (seconds)
   export INITIAL_DELAY="1";       # Initial waiting time between batch operations (seconds)
   export BACK_OFF_FACTOR="2.0";    # Multiplier to increase retry delay time
   export MAX_DELAY="180";          # Maximum Delay allowed between retries of one batch (seconds)
   export JAVA_HOME="";
   export DB_HOSTNAME="";
   export DB_PORT="";
   export DB_SERVICE_NAME="";
   export DB_USER_NAME="";    # user details left as blank for production environment once prompt then you can provide
   export DB_PASSWD=""; # password details left as blank for production environment once prompt then you can provide