Deploying OpenSearch and OpenSearch Dashboard

5 Deploying OpenSearch and OpenSearch Dashboard

This chapter describes how to deploy the OpenSearch service and OpenSearch dashboard.

Configuring OpenSearch and OpenSearch Dashboard Images

The Common CNTK has a sample that provides deployment instructions for OpenSearch on Kubernetes cluster using Helm charts. For more information, see https://opensearch.org/docs/latest/install-and-configure/install-opensearch/helm/

Note:

You can install any OpenSearch available in the market. Oracle recommends you to use OCI OpenSearch or OpenSearch with the configuration parameters provided under samples directory in common-cntk.

Installing OpenSearch Helm Charts

To install OpenSearch Helm charts, run the following commands:

helm repo add opensearch https://opensearch-project.github.io/helm-charts/
helm repo update
helm search repo opensearch //you should see 2 charts opensearch/opensearch & opensearch/opensearch-dashboard

Creating an OpenSearch and OpenSearch Dashboard Instance without SSL and Oauth Enablement

This section provides information on creating an OpenSearch and OpenSearch Dashboard instance without having SSL and Oauth enablement.

Setting up Environment Variables

OpenSearch and OpenSearch Dashboard rely on access to certain environment variables to run seamlessly across different environments. The OpenSearch charts are available at $COMMON_CNTK/samples/charts/opensearch.

The environment variables are configured in $COMMON_CNTK/samples/charts/opensearch/scripts/env.sh.

Configure OPENSEARCH_CHART_VERSION, OPENSEARCH_DASHBOARD_CHART_VERSION, and OS_NAMESPACE in env.sh script.

To set up the environmental variables:

Set the OpenSearch namespace as follows:
```
export OS_NAMESPACE=sr
```
Set the OpenSearch chart version as follows:
```
export OPENSEARCH_CHART_VERSION=<chart version>
```

Set the OpenSearch dashboard chart version as follows:

export OPENSEARCH_DASHBOARD_CHART_VERSION=<OpenSearch Dashboard chart version>

Note:

For the corresponding OpenSearch chart versions, see Unified Inventory and Topology Microservices.

Installing OpenSearch

For OpenSearch configuration overrides, see "Assembling the Specifications". To override the default values, copy the $COMMON_CNTK/samples/charts/opensearch/os_board_values.yaml and $COMMON_CNTK/samples/charts/opensearch/os_engine_values.yaml files to the $SPEC_PATH/opensearch directory.

Install OpenSearch as follows:

$COMMON_CNTK/samples/charts/opensearch/scripts/create-opensearch.sh

To configure Persistence and Storage for OpenSearch, provide storage class name under storageClass as follows:

persistence:
  storageClass: <storageclass>

If you use a private or alternative Docker registry, provide the registry URL in the following command. For example: "myregistry.example.com":

global:
  dockerRegistry: ""

Installing OpenSearch Dashboard

Install the OpenSearch dashboard using the following command:

$COMMON_CNTK/samples/charts/opensearch/scripts/create-opensearch-dashboard.sh

Accessing the OpenSearch Dashboard Service

Access the OpenSearch dashboard using the node port of the OpenSearch dashboard service in the namespace.

export NODE_PORT=$(kubectl get --namespace $OPENSEARCH_NS -o jsonpath="{.spec.ports[0].nodePort}" services os-board-opensearch-dashboards)
export NODE_IP=$(kubectl get nodes --namespace $OPENSEARCH_NS-o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT

Installing FluentD

Update the $COMMON_CNTK/samples/charts/fluentd/template/fluentd-config-map.yaml file with OpenSearch details such as type, host, port, scheme, user, password, and ssl_verify as follows:

#Export the kubernetes namespace to be used for OpenSearch installation
helm install fluentd-logging $COMMON_CNTK/samples/charts/fluentd --values $COMMON_CNTK/samples/charts/fluentd/values.yaml

Creating an OpenSearch and OpenSearch Dashboard Instance with SSL and Oauth Enablement

This section provides information on creating an OpenSearch and OpenSearch Dashboard Instance with SSL and Oauth enablement.

Setting up Environment Variables

To set up environmental variables, see Setting up Environment Variables.

Configuring config.yml

While configuring $COMMON_CNTK/samples/charts/opensearch/config.yml for OpenSearch with an Identity Provider (IdP), set up authentication parameters to enable seamless and secure integration.

To configure config.yaml:

Define the openid_connect_idp settings to include the discovery URL, client ID, and client secret of the IdP, which are essential for OpenID Connect (OIDC) authentication.
Specify the authorization and token endpoints provided by the IdP to handle user logins and configure role mapping to assign user roles based on the IdP attributes.

This setup allows OpenSearch to authenticate users through the IdP, easing access management while enhancing the security and compliance using:

openid_connect_url: The OpenID configuration URL of IdP.
client_id and client_secret: To securely authenticate OpenSearch with the Identity Provider (IdP) for OpenID Connect.

Creating Secrets in OpenSearch

To create secrets in OpenSearch:

Create a Kubernetes secret for the IdP trust certificate as follows:

kubectl create secret generic <opensearch-namespace>-opensearch-idp-cert  --from-file idp-chain-cert.pem=<IDP chain cert path> -n <opensearch-namespace>

Create a Kubernetes secret for OpenSearch IdP configuration after providing the required information in config.yml as follows:
```
 kubectl create secret generic <opensearch-namespace>-opensearch-idp-config-secret --from-file config.yml=<path/config.yml> -n <opensearch-namespace>
```

Configure rolesMappingSecret for OpenSearch roles mapping as follows:

 kubectl create secret generic <opensearch-namespace>-opensearch-rolesmapping-secret --from-file roles_mapping.yml=<path/role_mapping.yml> -n <opensearch-namespace>

Configure internalUsersSecret for OpenSearch internal users as follows:

kubectl create secret generic <opensearch-namespace>-opensearch-internalusers-secret --from-file internal_users.yml=<path/internal_users.yml> -n <opensearch-namespace>

Note:

The default username and password are used.

Implement Custom Certificates in OpenSearch

To implement custom certificates in OpenSearch:

Create the common certificates and common keys.

Create a secret from the common certificate and common key as follows:

 kubectl create secret generic <opensearch-namespace>-opensearch-keystore-cert --from-file=commoncert.pem=<path/commoncert.pem> --from-file=commonkey.pem=<path/commonkey.pem> -n sr

Provide the secretName under secretMounts in os_engines_values.yml as follows:

- name: commoncertandkey
  secretName: <opensearch-namespace>-opensearch-keystore-cert
  path: /usr/share/opensearch/config/certs

Provide the Distinguished Names (DNs) for authorized admin users in plugins.security.authcz.admin_dn and for trusted nodes in plugins.security.nodes_dn to control the access and secure node communication in OpenSearch.
Configure disabling of the installation of demo configuration settings that include demo users, roles, and certificates as follows:
```
env:
  name: DISABLE_INSTALL_DEMO_CONFIG
  value: true
```
Update os_engine_values.yaml to enable Security Plugin for OpenSearch. Comment the following section or set the value to false:
```
extraEnvs:
- name: DISABLE_SECURITY_PLUGIN
  value: "false"
```

Create an OpenSearch Instance

To create an OpenSearch instance:

To create an opensearch instance:

$COMMON_CNTK/samples/charts/opensearch/scripts/create-opensearch.sh

Run the runsecurity-admin.sh script to apply the changes that are made while implementing the custom certificates as follows:
```
$COMMON_CNTK/samples/charts/opensearch/scripts/runsecurity-admin.sh
```

Configuring opensearch_dashboards.yml for OpenSearch Dashboards Settings

The opensearch_dashboards.yml configuration file is essential for setting up and customizing the behavior of OpenSearch Dashboards. This file contains various settings that control how the dashboards interact with OpenSearch, including authentication, security, and connection parameters as follows:

opensearch.password: The Basic Authentication password for accessing OpenSearch.
opensearch_security.openid.base_redirect_url: The redirect URL for OpenSearch authentication.
opensearch_security.openid.connect_url: The OpenID configuration URL of IdP.
opensearch_security.openid.client_id: The client ID for OpenSearch authentication with the IdP.
opensearch_security.openid.client_secret: The client secret to secure communication with the IdP.
opensearch_security.openid.scope: The authentication scope.

Creating Secrets in OpenSearch Dashboard

To create secrets for OpenSearch Dashboard:

Create a Kubernetes Secret with the IdP trust Certificate and mount the secret in os_board_values.yml under secretMounts as follows:
```
kubectl create secret generic <opensearch-namespace>-opensearch-idp-cert --from-file idp-chain-cert.pem=<Path/IDP chain cert path> -n <opensearch-namespace>
```
Note:
You may skip this step if you have created a Kubernetes secret for OpenSearch earlier.

Create a Kubernetes secret containing the OpenSearch Dashboards configuration and the corresponding location within the container:

kubectl create secret generic <opensearch-namespace>-opensearch-dashboard-config --from-file=opensearch_dashboards.yml=<path/opensearch_dashboards.yml> -n <opensearch-namespace>

Setting up Ingress Controller for OpenSearch Dashboard

Using nginx as an Ingress Controller for OpenSearch Dashboards provides a highly configurable and stable solution for managing external access to your OpenSearch environment. By deploying nginx as the ingress, you can direct traffic to OpenSearch Dashboards using custom routing rules, including domain-based and path-based routing, allowing a better traffic control. You can use any Generic Ingress Controller that is available in the market. The following samples are for nginx.

The OpenSearch Dashboards integrate seamlessly with nginx ingress in Kubernetes environments to provide secure and scalable access. By leveraging the nginx ingress controller, administrators can share the Dashboards service externally, allowing users to interact with the OpenSearch data through a web-based interface.

To setup nginx for dashboard, configure below values in os_board_values.yml:

ingressClassName: Provides the ingress class name value under the ingress.ingressClassName field.
secretName: Defines the TLS secret that contains the SSL certificate for HTTPS.
tls.host: Specifies the domain for accessing OpenSearch Dashboards over HTTPS.
hosts.host: Specifies the domain for accessing OpenSearch Dashboards over HTTPS.

Run the following to create the secret:

kubectl create secret tls <project>-<instance>-opensearch-ingress-tls-cert-secret --cert=<path/commoncert.pem> --key=<path/commonkey.pem> -n <os-namespace>

Registering OpenSearch or OpenSearch Dashboard in Identity Provider

You must create Oauth 2.0 client in your IDP. You can use the same client created for ATA service. For more information, see "Registering ATA in Identity Provider".

In addition to the steps you perform while registering ATA in IdP, add the redirect URI: https://<opensearch-instance>.<opensearch-namespace>.opensearch.uim.org:<LB-PORT>/auth/openid/login

Add the following Post Logout Redirect URI:

https://<opensearch-instance>.<opensearch-namespace>.opensearch.uim.org:<LB-PORT>

Create OpenSearch Dashboard Instance

To create a OpenSearch dashboard instance, run the following command:

$COMMON_CNTK/samples/charts/opensearch/scripts/create-opensearch-dashboard.sh

Accessing the OpenSearch Dashboard

Access the OpenSearch dashboard URL that should redirect to the IdP login page. After you successfully log in, the OpenSearch dashboard appears.

Dashboard URI:  https://<instance>.<project>.opensearch.<hostSuffix>:<nginx-port>/app/home#/
Example: https://quick.sr.opensearch.uim.org:30543/app/home#/

Note:

Make sure you add the entry in the hosts file:

<k8s cluster ip or external
                                loadbalancer ip>
                                <instance>.<project>.opensearch.<hostSuffix>

Upgrading the OpenSearch and OpenSearch Dashboard Service

This section provides information about upgrading the OpenSearch and OpenSearch dashboard services.

Upgrade OpenSearch Service

Upgrading OpenSearch is required when there are updates in the OpenSearch values yaml file.

To upgrade the OpenSearch instance:

$COMMON_CNTK/samples/charts/opensearch/scripts/upgrade-opensearch.sh

Upgrade OpenSearch Dashboard Service

Upgrading OpenSearch dashboard is required when there are updates in the OpenSearch dashboards values yaml file.

To upgrade OpenSearch dashboard service:

$COMMON_CNTK/samples/charts/opensearch/scripts/upgrade-opensearch-dashboard.sh

After running the script, verify the OpenSearch dashboard service by accessing it on a browser.

Deleting the OpenSearch and OpenSearch Dashboard Service

This section provides information on deleting the OpenSearch and OpenSearch dashboard services.

Deleting OpenSearch Service

To delete the OpenSearch instance:

$COMMON_CNTK/samples/charts/opensearch/scripts/delete-opensearch.sh

Deleting OpenSearch Dashboard Service

To delete the OpenSearch dashboard instance:

$COMMON_CNTK/samples/charts/opensearch/scripts/delete-opensearch-dashboard.sh
helm uninstall fluentd-logging --namespace=$OPENSEARCH_NS

Alternate Configuration Options for OpenSearch and OpenSearch Dashboard

This section provides alternate configuration options for OpenSearch and OpenSearch dashboard.

Setting up Traefik Ingress Controller for OpenSearch Dashboard

Using Traefik as an Ingress Controller for OpenSearch Dashboards provides a robust, flexible way to manage and route external traffic to your OpenSearch environment. By setting up Traefik, you enable seamless access control and traffic routing through HTTP or HTTPS, helping secure and balance traffic flow to your OpenSearch Dashboards. Traefik supports dynamic configuration, allowing you to set up rules based on domain names, paths, or other criteria, making it highly adaptable to different deployment environments.

While using Traefik as an Ingress Controller for OpenSearch Dashboards, you can configure the required strategies in your traefik.yml file based on whether TLS is enabled or disabled. If TLS is enabled, you can specify the TLS certificates for secure HTTPS connections, ensuring encrypted communication between users and OpenSearch Dashboards. If TLS is disabled, Traefik routes the traffic over HTTP, bypassing the encryption step.

To setup Traefik for dashboard, configure below values in traefik.yml:

name: Configure the Traefik Ingress Controller with a specific name. Replace <project> and <instance> with your project and instance identifiers to apply this setting specifically to your OpenSearch Dashboards setup.
match: Replace <instance>, <project>, and <hostSuffix> with your specific instance, project name, and domain suffix to ensure accurate routing within your Traefik setup.
TLSOptions.name: Replace <project> and <instance> with your project and instance names to create a dedicated TLS configuration for your OpenSearch deployment.

To access the OpenSearch Dashboard:

To access OpenSearch dashboard using https://<instance>.<project>.opensearch.<hostSuffix>:<traefik-port>/app/home#/:

Changing the Default OpenSearch Basic Auth Password

To enhance security, update the default admin password in internal_users.yml by generating a hashed password using the OpenSearch hash script.

To change the default OpenSearch basic auth password:

Navigate to the OpenSearch Security plugin directory in opensearch pod as follows:
```
cd /usr/share/opensearch/plugins/opensearch-security/tools
```
Run the following:
```
./hash.sh "<your-password>"
```
Copy the hashed password and update internal_users.yml.
Set the same password in opensearch_dashboards.yml.
Replace the existing password hash with the new hashed value to secure the admin account.
To ensure compatibility, add the same password (in plain text) under opensearch.password in opensearch_dashboards.yml.
This configuration allows OpenSearch Dashboards to authenticate with the updated credentials, aligning password settings across both files.

Creating Ingest Pipeline for OpenSearch

Login to OpenSearch Dashboard and navigate to DevTools on the top-right corner of the Home page and run the following to add the lang_pipeline in OpenSearch:

PUT _ingest/pipeline/lang_pipeline
{
  "description": "Pipeline to ensure documents have a lang field",
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": """
          if (ctx.lang == null) {
            throw new IllegalArgumentException('Document does not have the *lang* field');
          }
        """
      }
    }
  ]
}

Debugging and Troubleshooting

OpenSearch pods restarting with error, related to max virtual memory areas

When you get the error: vm.max_map_count [65530] is too low, increase it to at least [262144]. In OpenSearch pods, you need to set vm.max_map_count to 262144 on each worker node where the OpenSearch pod is getting scheduled. Run the following commands on the nodes:

sudo sysctl -w vm.max_map_count=262144
 
# Alternatively you can run the following to save changes permanently  
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sysctl -p

After logging in to the OAuth-enabled OpenSearch dashboard, seeing user ID instead of username

If you get a user ID instead of a username in the user-info section, you can configure a custom property mapper to map the username to the sub claim of the access token and add it to your scope configured with the OAuth client.

In the case of Keycloak, you can create a user property mapper.