6 Deploying SmartSearch

This chapter describes how to deploy the SmartSearch service.

About SmartSearch

SmartSearch is a micronaut application, when integrated with OpenSearch, offers a powerful, flexible, and feature-rich search experience that can be tailored to specific business and user needs. Using OpenSearch as the underlying engine, SmartSearch can handle large volumes of data, perform real-time indexing, and support complex querying to enhance search relevancy. Features such as autocomplete, fuzzy matching, synonym recognition, and intelligent ranking make it easier for users to locate precise information, even if search terms are partially matched or misspelled.

Creating a SmartSearch Instance

This section describes how to create a SmartSearch service instance in your cloud native environment using the operational scripts and the configuration provided in the Common cloud native toolkit (Common CNTK).

Before you create a SmartSearch instance, you must validate cloud native environment. See "Planning UIM Installation" for details on prerequisites.

In this section, while creating a basic instance, the project name is considered as sr and instance name is considered as quick.

Note:

The project and instance names cannot contain any special characters.

Creating SmartSearch Schema

The SmartSearch schema in OpenSearch is structured to optimize search performance and relevance, using six main indexes: Location, Logical Device, Physical Device, Equipment, Communication, and Events. Each index is defined by mappings that specify field types such as text for full-text search and keyword for exact matches and may include custom analyzers for handling text processing tasks such as tokenization and synonyms. Additionally, metadata is configured for each index to control search behavior, supporting features such as filters, sorting, and autocomplete to enhance user experience. All indexes are accessed using aliases, providing a flexible way to query and manage data. This setup ensures fast, accurate, and relevant search results tailored to user needs.

Prerequisites

Ensure the following environment variables are set:

  • SMARTSEARCHDOMAIN: Set this to the SmartSearch service endpoint (format: export SMARTSEARCHDOMAIN=<host:port/appVersion>), using the corresponding version .

    Note:

    For SmartSearch appVersion, see UIM Compatibility Matrix.
  • CLIENT_SECRET, CLIENT_ID, SCOPE, and IDP_TOKEN_URL: These are necessary for generating the access token for the SmartSearch service.
  • Ensure SMARTSEARCHDOMAIN and IDP_TOKEN_URL are accessible and included in the NO_PROXY settings or that the proxy is disabled.
  • Ensure the relevant domain entries are present in the /etc/hosts file. For example: <instance>.<project>.topology.<hostSuffix>

Creating Index and Metadata

To create index and metadata:

  1. Navigate to the smartsearch directory:
    cd $COMMON_CNTK/scripts/smartsearch
  2. Run the script to create indexes and metadata for NPD:
    ./refreshIndexes.sh --ssl-cert-smartsearch <path-to-cert> --ssl-cert-oauth <path-to-cert>
  3. (Optional) For non-secure deployments, use the --security-enable false flag.

Validating the Indexes

To validate the indexes:

  1. Access the OpenSearch dashboard and navigate to DevTools.
  2. Run the following command to verify if the required indexes are created:
    GET /_cat/indices
  3. Check if the following indexes with any suffixes are present:
    • smartsearch-location-{*}
    • smartsearch-communication-{*}
    • smartsearch-physicaldevice-{*}
    • smartsearch-logicaldevice-{*}
    • smartsearch-equipment-{*}
    • smartsearch-event-{*}

    Note:

    If any of these indexes are missing, rerun the script.

Accessing the SmartSearch Service

To access SmartSearch service, use the SmartSearch endpoint in the format: http://<topology-instance>.<topology-project>.topology.<hostSuffix>:<port>/<appVersion>/<uri-endpoint>

For example: SmartSearch API endpoint is http://quick.sr.topology.uim.org:30309/20240801/health

Note:

Make sure you add entry in /etc/hosts: <k8s cluster ip or external loadbalancer ip> quick.sr.topology.uim.org.

For SmartSearch appVersion, see UIM Compatibility Matrix.

Validating the SmartSearch Instance

To validate the SmartSearch instance, run the following to the check the status of the deployed SmartSearch instance:

$COMMON_CNTK/scripts/application-status.sh -p sr -i quick -f $SPEC_PATH/sr/quick/applications.yaml -a smartsearch
application-status

This script returns the status of the SmartSearch service deployments and pods status.

Monitoring the SmartSearch Health

Use the following command to monitor the SmartSearch endpoint health:

https://<loadbalancerhost>:<loadbalancerport>/<appVersion>/health

Configure Logging for SmartSearch

To update log level for SmartSearch, check the following steps:

  1. Update SmartSearch.env.LOGLEVEL value with the appropriate Log level (ERROR, INFO, DEBUG, TRACE ) in $SPEC_PATH/sr/quick/applications.yaml.
  2. Upgrade SmartSearch instance as follows:
    $COMMON_CNTK/scripts/upgrade-applications.sh -p sr -i quick -f  $SPEC_PATH/sr/quick/applications.yaml -a smartsearch

Managing SmartSearch SSL Certificates

To renew SmartSearch ingress SSL certificates or to import any new Egress certificate in SmartSearch truststore, see "Managing Certificate Expiry".

Setting up Environment Variables

The SmartSearch service relies on access to certain environment variables to run seamlessly. Ensure the following variables are set in your environment:

  • Path to your common cloud native toolkit
  • Traefik namespace (Optional)

To set the environment variables:

  1. Set the COMMON_CNTK variable to the path of directory where common cloud native toolkit is extracted as follows:
    $ export COMMON_CNTK=$WORKSPACEDIR/common-cntk
  2. In case Traefik is used as ingress, set the TRAEFIK_NS variable for Traefik namespace as follows:
    $ export TRAEFIK_NS=Treafik Namespace
  3. Set the variable for Traefik Helm chart version. For the corresponding version, see UIM Compatibility Matrix. The following is a sample for Traefik chart version 15.1.0.
    $ export TRAEFIK_CHART_VERSION=15.1.0 
  4. Set SPEC_PATH variable to the location where application and database yamls are copied. See "Assembling the Specifications" to copy specification files if not already copied.
    $ export SPEC_PATH=$WORKSPACEDIR/spec_dir

Registering the Namespace

After you set the environment variables, register the namespace if Traefik is used as Ingress Controller.

To register the namespace, run the following command:

$COMMON_CNTK/scripts/register-namespace.sh -p sr -t targets
# For example, $COMMON_CNTK/scripts/register-namespace.sh -p sr -t traefik
# Where the targets are separated by a comma without extra spaces

Note:

traefik is the name of the target for registration of the namespace sr. The script uses TRAEFIK_NS to find these targets. Do not provide the Traefik target if you are not using Traefik.

For Generic Ingress Controller, you do not have to register the namespace. To select the ingress controller, provide the ingressClassName value under the ingress.className field in the applications.yaml file. For more information about ingressClassName, see https://kubernetes.io/docs/concepts/services-networking/ingress/

Creating Secrets

To create secrets:

  1. Run the following command to generate the openSearchCredentials secret and enter the basic auth credentials of OpenSearch as prompted:
    $COMMON_CNTK/scripts/manage-app-credentials.sh -p sr -i quick -a smartsearch -f $SPEC_PATH/samples/applications.yaml create openSearchCredentials  
     
     #Enter the values appropriately against  prompts:
     opensearch username: admin   #opensearch basic oauth credentials
     opensearch password: xxxxx   
  2. Verify if the secrets are created as follows:
    sr-quick-open-search-credentials

Secret for Authentication Details

The SmartSearch service uses configuration values from Kubernetes secret (<namespace>-<instance>-oauth-credentials) and Config Map (<namespace>-<instance>-oauth-config-cm) objects from the same namespace. This Secret and Configuration Map Kubernetes objects have to be created before deploying the SmartSearch service for authentication. See "Adding Common OAuth Secret and ConfigMap" for creating the secret.

Secret to Pass Egress Certificates to SmartSearch

You have to create a commonTrust secret to provide SSL certificates of IdP, OpenSearch, and so on to SmartSearch.

To create commonTrust secret:

  1. Generate truststore.jks that contains the required certificates as follows:
    #generate truststore with idp cert (applicable only if authentication is enabled )
    keytool -importcert -v -alias <param> -file <path to IDP cert file> -keystore <truststorename>.jks -storepass <password>
     
    #add opensearch ssl trust certificate in truststore
    keytool -importcert -v -alias <param> -file <path to opensearch cert file> -keystore <truststorename>.jks -storepass <password>
  2. Run the below command to generate the Kubernetes secret. The system will prompt for the truststore file path and password of truststore. Provide the details accordingly.
    $COMMON_CNTK/scripts/manage-app-credentials.sh -p project -i instance -f $SPEC_PATH/project/instance/applications.yaml create commonTrust

Configuring applications.yaml

In $SPEC_PATH/sr/quick/applications.yml, the SmartSearch image and tag are configured to specify the container image used for the SmartSearch service. The image tag can be updated as new version of the image are released, allowing easy upgrades and ensuring the application runs the latest features and security patches. To apply a new image version, simply change the tag in applications.yml to the desired version, facilitating version control and deployment management.

To configure the applications.yaml file:

  1. Download the Oracle Communications Unified Inventory Management SmartSearch image from Oracle Software Delivery Cloud.
  2. Load the image from tar file and push the image to container repository. Refer "About Container Image Management" in UIM Cloud Native Deploy Guide.
    podman load -I <smart-search-latest.tar>
    podman tag smart-search:latest <conatiner repository>/smart-search:latest
    podman push <conatiner repository>/smart-search:latest
    
  3. Update the corresponding values under the smartsearch section in $SPEC_PATH/<PROJECT>/<INSTANCE>/applications.yaml.
  4. Edit the applications.yaml file and enter the name and tag of the SmartSearch image from your repository as follows:
    smartSearch:
      image:
        name: <conatiner repository>/smart-search
        tag: 0.3.11
  5. You can provide an image pull secret as follows to pull images from your private repository:
    # The image pull access credentials for Private repository, as a Kubernetes secret.
    # uncomment and set if required.
     
    #  imagePullSecret:
    #    imagePullSecrets:
    #      - name: regcred
  6. Set pullPolicy for SmartSearch images in applications.yaml as follows:
    smartSearch:
      #Provide imagePullPolicy i.e Always, IfNotPresent, Never
      imagePullPolicy: IfNotPresent
  7. Update loadbalancerport in applications.yaml as follows:
    # For Generic and Traefik Ingress Controllers:
    # If ssl is enabled this would be loadbalancer's ssl port.
    # IF ssl is disabled this would be loadblancer's non ssl port.
    # For examle ssl and non-ssl ports for external loadbalancer would be 443 and 80 respectively.
    # If loadbalancer is not created, provide nodePort of Nginx/Traefik or any other Generic Ingress Controller
     
    loadBalancerPort: 30505
  8. For Generic ingress controller update as follows:
    • Update applications.yaml as follows to provide the following annotations to enable stickiness through cookies:
      # Valid values are TRAEFIK, GENERIC
      ingressController: "GENERIC"
    • Annotations for NGINX is as follows:
      ingress:
        className: nginx  ##provide ingressClassName value, default value for nginx ingressController is nginx.
        # This annotation is required for nginx ingress controller.
        annotations:
          nginx.ingress.kubernetes.io/affinity: "cookie"
          nginx.ingress.kubernetes.io/affinity-mode: "persistent"
          nginx.ingress.kubernetes.io/session-cookie-name: "nginxingresscookie"
          nginx.ingress.kubernetes.io/proxy-body-size: "50m"
          
      smartSearch:
         #uncomment and provide applications specific annotations if required, these will get added to list of annotations specified in common section.
         ingress:
           annotations:
             nginx.ingress.kubernetes.io/rewrite-target: /$2
  9. Set the authentication.enabled flag to true to enable authentication as follows:
    # The enabled flag is to enable or disable authentication
    authentication:
      enabled: true

Creating a SmartSearch Instance

To create an SmartSearch service instance in your environment using the scripts that are provided with the toolkit:

  1. Run the following command to create an SmartSearch service instance:
    $COMMON_CNTK/scripts/create-applications.sh -p sr -i quick -f $SPEC_PATH/sr/quick/applications.yaml -a smartsearch

    The create-applications script uses the Helm chart located in $COMMON_CNTK/charts/smartsearch-app to create and deploy a SmartSearch service.

  2. (Optional) If the scripts fail, run the following command to review the error message in the log:
    kubectl logs -n sr sr-quick-smartsearch
  3. Clear the failed pod by running the following command:
    helm uninstall sr-quick-smartsearch-n sr
  4. Fix the issues if any and run the script again to deploy the SmartSearch service.

Upgrading the SmartSearch Service

Upgrading SmartSearch is required when there are updates made to applications.yaml or other configuration files.

To upgrade the SmartSearch instance:

  1. Run the following command to upgrade the SmartSearch service:
    $COMMON_CNTK/scripts/upgrade-applications.sh -p sr -i quick -f $SPEC_PATH/sr/quick/applications.yaml -a smartsearch
    
  2. After script is run, validate the SmartSearch service by running the application-status script.

Restarting the SmartSearch Instance

To restart the SmartSearch instance:

  1. Run the following command to restart the SmartSearch service:
    $COMMON_CNTK/scripts/restart-applications.sh -p sr -i quick -f $SPEC_PATH/sr/quick/applications.yaml -a smartsearch
  2. After running the script, validate the SmartSearch service by running the application-status script.

Deleting the SmartSearch Service

To delete the SmartSearch instance, run the following command to delete the SmartSearch service instance:

$COMMON_CNTK/scripts/delete-applications.sh -p <project-name> -i <instance-name> $SPEC_PATH/sr/quick/applications.yaml -a smartsearch

Alternate Configuration Options for SmartSearch

You can configure SmartSearch using the following alternate options.

Setting Up a Secure Communication using TLS

To set up a secure communication using TLS:

  1. Edit the $COMMON_CNTK/samples/applications.yaml and set the TLS enabled to true.
    tls:
      # The enabled flag is to enable or disable the TLS support for the smart search m-s  end points
      enabled: true

    Smartsearch API endpoint: https://quick.sr.topology.uim.org:<LB Port>/<appVersion>/<uri-endpoint>

  2. Generate common certificates commoncert.pem and commonkey.pem. Refer "SSL Certificates" in Appendix.

  3. Create IngressTLS secret to pass the generated certificate and key pem files:

    $COMMON_CNTK/scripts/manage-app-credentials.sh -p sr -i quick -f $SPEC_PATH/sr/quick/applications.yaml -a smartsearch create ingressTLS

    The script prompts for the following detail:

    • Ingress TLS Certificate Path (PEM file): <path_to_cert.pem>
    • Ingress TLS Key file Path (PEM file): <path_to_key.pem>
  4. Verify that the following secrets are created successfully:
    sr-quick-smart-search-ingress-tls-cert-secret
  5. Create a SmartSearch Instance as usual and access the SmartSearch endpoints using hostname <instance>.<instance>.topology.uim.org
  6. Add entry in /etc/hosts <k8s cluster ip or external loadbalancer ip> quick.sr.topology.uim.org.

    The SmartSearch API endpoint is https://quick.sr.topology.uim.org:<LB Port>/<appVersion>/<uri-endpoint>

Supporting Wildcard Certificates

SmartSearch supports wildcard certificates. You can generate the wildcard certificates with the hostSuffix value provided in applications.yaml. The default is uim.org.

Change the subDomainNameSeperator value from period(.) to hyphen(-) so that incoming hostnames match the wildcard DNS pattern.

Make the following modifications to the $SPEC_PATH/project/instance/applications.yaml file.

Uncomment and provide the value of subDomainNameSeparator. The default is ".".

#Value can be changed as "-" to match wild-card pattern of ssl certificates.
#Example hostnames for "-" quick-sr-topology.uim.org
subDomainNameSeparator: "-"

Scaling Up or Down the SmartSearch Service

To scale up or down the SmartSearch service:

  1. Provide the replica count in applications.yaml to scale up or down the SmartSearch pods.

  2. Update applications.yaml to increase replica count to 3 for topology-api deployment as follows:

    smartSearch:
        replicas: 3
  3. Apply the change in replica count to the running Helm release by running the upgrade-applications script as follows:
    $COMMON_CNTK/scripts/upgrade-applications.sh -p sr -i quick -f $SPEC_PATH/sr/quick/applications.yaml -a smartsearch