The Java Flight Recorder (JFR) is a tool that collects diagnostic data about running Java applications. OSM cloud native leverages JFR. See Java Platform, Standard Edition Java Flight Recorder Runtime Guide for details about JFR.

You can change the JFR settings provided with the toolkit by updating the appropriate values in the instance specification.

To analyze the output produced by the JFR, use Java Mission Control. See Java Platform, Standard Edition Java Mission Control User's Guide for details about Java Mission Control.

JFR is turned on by default in all managed servers. You can disable this feature by setting the enabled flag to false.

# Java Flight Recorder (JFR) Settings
jfr:
 enabled: true
 max_age: 4h

Persisting JFR Data

JFR data can be persisted outside of the container by re-directing it to persistent storage through the use of a PV-PVC. See "Setting Up Persistent Storage" for details.

Once the storage has been set up, enable storageVolume and set the PVC name. Once enabled, JFR data is re-directed automatically.

# The storage volume must specify the PVC to be used for persistent storage.

storageVolume:
  enabled: true
  pvc: storage-pvc

This section describes how to troubleshoot issues with access to the OSM UI clients, WLST, and WebLogic Administration Console.

It is assumed that Traefik is the Ingress controller being used and the domain name suffix is osm.org. You can modify the instructions to suit any other domain name suffix that you may have chosen.

helm uninstall traefik-operator -n traefik

When the OSM DB Installer fails during an installation, it exits with an error message. You must then find and resolve the issue that caused the failure. You can re-run the DB Installer after the issue (for example, space issue or permissions issue) is rectified. You do not have to rollback the DB.

It is recommended that you always run the DB Installer with the logs directed to a Persistent Volume so that you can examine the log for errors. The log file is located at: filestore/project-instance/db-installer/{yyyy-mm-dd}-osm-db-installer.log.

In addition, to identify the operation that failed, you can look in the filestore/project-instance/db-installer/InstallPlan-OMS-CORE.csv CSV file. This file shows the progress of the DB Installer.

While running OSM cloud native with more than one managed server, sometimes, the incoming orders and the resulting workload may not get distributed evenly across all managed servers.

While there are multiple causes for improper distribution (including the use of an incorrect JMS connection factory to inject order creation messages), one possible cause is the improper assignment of JMS servers to managed servers. For even distribution of workload, each managed server that is running must host its corresponding JMS server.

The following figure shows an example of improper JMS assignment.

Figure 14-1 Example of Improper JMS Assignment

Description of "Figure 14-1 Example of Improper JMS Assignment"

In the figure, osm_jms_server@ms7 is incorrectly running on ms6 even though its native host ms7 is running. It can be normal for more than one JMS server to be running on a managed server as long as the additional JMS servers do not have a native managed server that is online.

Workaround

As a workaround, terminate the Kubernetes pod for the managed server that has been left underutilized. In the above example, the pod for ms7 should be terminated. The WebLogic Operator recreates the managed server pod, and that should trigger the migration of osm_jms_server@ms7 back to ms7.

Resolution

To resolve this issue, tune the time setting for InitialBootDelaySeconds and PartialClusterStabilityDelaySeconds. See the WebLoigic Server documentation for more details.

To tune the time setting:

1. Add the following Clustering fragment to the instance specification:

   Clustering:
     InitialBootDelaySeconds: 10
     PartialClusterStabilityDelaySeconds: 30

2. Increase the value for the following parameters from the base WDT model:

   • InitialBootDelaySeconds. The default value in base WDT is 2.
   • PartialClusterStabilityDelaySeconds. The default value in base WDT is 5.

This section describes some common problems that you may experience because you have run a script or a command erroneously or you have not properly followed the recommended procedures and guidelines regarding setting up your cloud environment, components, tools, and services in your environment. This section provides possible solutions for such problems.

Domain Introspection Pod Does Not Start

There may be a case where introspector doesn't start. This could mean that the operator is not monitoring your namespace or your namespace is not tagged to the correct label which the operator is monitoring.

For more information about operator monitoring, see: https://oracle.github.io/weblogic-kubernetes-operator/managing-operators/common-mistakes/#forgetting-to-configure-the-operator-to-monitor-a-namespace

Domain Introspection Pod Status

kubectl get pods -n namespace

## healthy status looks like this
 
NAME                                       READY   STATUS    RESTARTS   AGE
project-instance-introspect-domain-job-hzh9t   1/1     Running   0          3s

NAME                                           READY   STATUS         RESTARTS   AGE
project-instance-introspect-domain-job-r2d6j   0/1     ErrImagePull   0          5s
 
### OR
 
NAME                                           READY   STATUS             RESTARTS   AGE
project-instance-introspect-domain-job-r2d6j   0/1     ImagePullBackOff   0          45s

This shows that the introspection pod status is not healthy. If the image can be pulled, it is possible that it took a long time to pull the image.

To resolve this issue, verify that the image name and the tag and that it is accessible from the repository by the pod.

You can also try the following:

• Increase the value of podStartupDeadlineSeconds in the instance specification.

  Start with a very high timeout value and then monitor the average time it takes, because it depends on the speed with which the images are downloaded and how busy your cluster is. Once you have a good idea of the average time, you can reduce the timeout values accordingly to a value that includes the average time and some buffer.

• Pull the container image manually on all Kubernetes nodes where the OSM cloud native pods can be started up.

Domain Introspection Errors Out

Some times, the domain introspector pod runs, but ends with an error.

kubectl logs introspector_pod -n project

The following are the possible causes for this issue:

• RCU Schema is pre-existing: If the logs shows the following, then RCU schema could be pre-existing:

  WLSDPLY-12409: createDomain failed to create the domain: Failed to write domain to /u01/oracle/user_projects/domains/domain: wlst.writeDomain(/u01/oracle/user_projects/domains/domain) failed : Error writing domain:
  64254: Error occurred in "OPSS Processing" phase execution
  64254: Encountered error: oracle.security.opss.tools.lifecycle.LifecycleException: Error during configuring DB security store. Exception oracle.security.opss.tools.lifecycle.LifecycleException: The schema FMW1_OPSS is already in use for security store(s).  Please create a new schema..
  64254: Check log for more detail.

  This could happen because the database was reused or cloned from an OSM cloud native instance. If this is so, and you wish to reuse the RCU schema as well, provide the required secrets. For details, see "Reusing the Database State".

  If you do not have the secrets required to reuse the RCU instance, you must use the OSM cloud native DB Installer to create a new RCU schema in the DB. Use this new schema in your rcudb secret. If you have login details for the old RCU users in your rcudb secret, you can use the OSM cloud native DB Installer to delete and re-create the RCU schema in place. Either of these options gives you a clean slate for your next attempt.

  Finally, it is possible that this was a clean RCU schema but the introspector ran into an issue after RCU data population but before it could generate the wallet secret (opssWF). If this is the case, debug the introspector failure and then use the OSM cloud native DB Installer to delete and re-create the RCU schema in place before the next attempt.

• Fusion MiddleWare cannot access the RCU: If the introspector logs show the following error, then it means that Fusion MiddleWare could not access the schema inside the RCU DB.

  WLSDPLY-12409: createDomain failed to create the domain: Failed to get FMW infrastructure database defaults from the service table: Failed to get the database defaults: Got exception when auto configuring the schema component(s) with data obtained from shadow table:
  Failed to build JDBC Connection object:

  Typically, this happens when wrong values are entered while creating secrets for this deployment. Less often, the cause is a corrupted RCU DB or an invalid one. Re-create your secrets, verifying the credentials and drop and re-create the RCU DB.

Recovery After Introspection Error

If the introspection fails during instance creation, once you have gathered the required information and have a solution, delete the instance and then re-run the instance creation script with the fixed specification, model extension, or other environmental failure cause.

If the introspection fails while upgrading a running instance, then do the following:

1. Make the change to fix the introspection failure. Trigger an instance upgrade. If this results in successful introspection, the recovery process stops here.
2. If the instance upgrade in step 1 fails to trigger a fresh introspection, then do the following:
   1. Rollback to the last good Helm release by first running the helm history -n project project-instance command to identify the version in the output that matches the running instance (that is, before the upgrade that led to introspection failure). The timestamp on each version helps you identify the version. Once you know the "good" version, rollback to that version by running: helm rollback -n project project-instance version. Monitor the pods in the instance to ensure only the Admin server and the appropriate number of Managed Server pods are running.
   2. Upgrade the instance with the fixed specification.

Instance Deletion Errors with Timeout

You use the delete-instance.sh script to delete an instance that is no longer required. The script attempts to do this in a graceful manner and is configured to wait up to 10 minutes for any running pods to shut down. If the pods remain after this time, the script times out and exits with an error after showing the details of the leftover pods.

The leftover pods can be OSM pods (Admin Server, Managed Server) or the DBInstaller pod.

For the leftover OSM pods, see the WKO logs to identify why cleanup has not run. Delete the pods manually if necessary, using the kubectl delete commands.

For the leftover DBInstaller pod, this happens only if install-osmdb.sh is interrupted or if it failed in its last run. This should have been identified and handled at that time itself. However, to complete the cleanup, run helm ls -n project to find the failed DBInstaller release, and then invoke helm uninstall -n project release. Monitor the pods in the project namespace until the DBInstaller pod disappears.

OSM Cloud Native Toolkit Instance Create and Update Scripts Timeout; Pods Show Readiness "0/1"

If your create-instance.sh or upgrade-instance.sh scripts timeout, and you see that the desired managed server pods are present, but one or more of them show "0/1" in the "READY" column, this could be because OSM hit a fatal problem while starting up. The following could be the causes for this issue:

• A mismatch in the OSM schema found and the expected version: If this is the case, the OSM managed server log shows the following issue:

  Error: The OSM application is not compatible with the schema code detected in the OSM database.
  Expected version[7.4.0.0.68], found version[7.4.0.0.70]
  This likely means that a recent installation or upgrade was not successful.
  Please check your install/upgrade error log and take steps to ensure the schema is at the correct version.

  To resolve this issue, check the container image used for the DB installer and the OSM domain instances. They should match.
• OSM internal users are missing: This can happen if there are issues with the configuration of the external authentication provider and the standard OSM users (for example, oms-internal) and the group association is not loaded. The managed server log shows something like the following:

  <Error> <Deployer> <BEA-149205> <Failed to initialize the application "oms" due to error 
  weblogic.management.DeploymentException: The ApplicationLifecycleListener "com.mslv.oms.j2ee.LifecycleListener" of application "oms" 
  has a run-as user configured with the principal name "oms-internal" but a principal of that name 
  could not be found. Ensure that a user with that name exists.

  To resolve this issue, review your external authentication system to validate users and groups. Review your configuration to ensure that the instance is configured for the correct external authenticator.

OSM Cloud Native Pods Do Not Distribute Evenly Across Worker Nodes

In some occasions, OSM cloud native pods do not distribute evenly across the worker nodes.

$ $OSM_CNTK/samples/image-primer.sh -p project image-name:image-tag

This script is offered as a sample and may need to be customized for your environment. If you are using an image from a repository that requires pull credentials, edit the image-primer.sh script to uncomment these lines and add your pull secret:

#imagePullSecrets:
  #- name: secret-name

If you are planning to target OSM cloud native to specific worker nodes, edit the sample to ensure only those nodes are selected (typically by using a specific label value) as per standard Kubernetes configuration. See the Kubernetes documentation for DaemonSet objects.

User Workgroup Association Lost

During cartridge deployment, if users are not present in LDAP or if LDAP is not accessible, the user workgroup associations could get deleted.

To resolve this issue, restore the connectivity to LDAP and the users. You may need to redo the workgroup associations.

Changing the WebLogic Kubernetes Operator Log Level

Some situations may require analysis of the WKO logs. These logs can be certain kinds of introspection failures or unexpected behavior from the operator. The default log level for the Operator is INFO.

For information about changing the log level for debugging, see the documentation at: https://oracle.github.io/weblogic-kubernetes-operator/managing-operators/troubleshooting/#operator-and-conversion-webhook-logging-level.

Deleting and Re-creating the WLS Operator

You may need to delete a WLS operator and re-create it. You do this when you want to use a new version of the operator where upgrade is not possible, or when the installation is corrupted.

When the controlling operator is removed, the existing OSM cloud native instances continue to function. However, they cannot process any updates (when you run upgrade-instance.sh) or respond to the Kubernetes events such as the termination of a pod.

To avoid common mistakes during the installation of WKO, refer to the WKO troubleshooting information at: https://oracle.github.io/weblogic-kubernetes-operator/managing-operators/common-mistakes/#forgetting-to-configure-the-operator-to-monitor-a-namespace.

To uninstall WKO, follow the steps in WKO documentation at: https://oracle.github.io/weblogic-kubernetes-operator/managing-operators/installation/#uninstall-the-operator.

Re-register your namespaces using the register-namespace.sh and unregister-namespace.sh scripts in the cloud native toolkit.

You can install the operator by following the instructions in WKO documentation at: https://oracle.github.io/weblogic-kubernetes-operator/managing-operators/installation/#install-the-operator. Then, register all the projects again, one by one. See "Registering the Namespace" for details.

Lost or Missing opssWF and opssWP Contents

For an OSM instance to successfully connect to a previously initialized set of DB schemas, it needs to have the opssWF (OPSS Wallet File) and opssWP (OPSS Wallet-file Password) secrets in place. The $OSM_CNTK/scripts/manage-instance-credentials.sh script can be used to set these up if they are not already present.

If these secrets or their contents are lost, you can delete and recreate the RCU schemas (using $OSM_CNTK/scripts/install-osmdb.sh with command code 5). This deletes data (such as some user preferences and so on) stored in the RCU schemas. On the other hand, if there is a WebLogic domain currently running against that DB (or its clone), the "exportEncryptionKey" offline WLST command can be run to dump out the "ewallet.p12" file. This command also takes a new encryption password. For details about WLST Command Reference for Infrastructure Security, see Oracle Fusion MiddleWare documentation. The contents of the resulting ewallet.p12 file can be used to recreate the opssWF secret, and the encryption password can be used to recreate the opssWP secret. This method is also suitable when a DB (or the clone of a DB) from a traditional OSM installation needs to be brought into OSM cloud native.

Clock Skew or Delay

When submitting JMS message over the Web Service queue, you might see the following in the SOAP response:

Security token failed to validate. weblogic.xml.crypto.wss.SecurityTokenValidateResult@5f1aec15[status: false][msg UNT Error:Message older than allowed MessageAge]

Oracle recommends synchronizing the time across all machines that are involved in communication. See "Synchronizing Time Across Servers" for more details. Implement Network Time Protocol (NTP) across the hosts involved, including the Kubernetes cluster hosts.

It is also possible to temporarily fix this through configuration by adding the following properties to java_options in the project specification for each managed server.managedServers: project:

#JAVA_OPTIONS for all managed servers at project level java_options:
-Dweblogic.wsee.security.clock.skew=72000000 
-Dweblogic.wsee.security.delay.max=72000000

This section describes known issues that you may come across, their causes, and the resolutions.

SituationalConfig NullPointerException

In the managed server logs, you might notice a stacktrace that indicates a NullPointerException in situational config.

This exception can be safely ignored.

Connectivity Issues During Cluster Re-size

When the cluster size changes, whether from the termination and re-creation of a pod, through an explicit upgrade to the cluster size, or due to a rolling restart, transient errors are logged as the system adjusts.

These transient errors can usually be ignored and stop after the cluster has stabilized with the correct number of Managed Servers in the Ready state.

If the error messages were to persist after a Ready state is achieved, then looking for secondary symptoms of a real problem would be appropriate. Such connectivity errors could result in orders that were inexplicably stuck or were otherwise processing abnormally.

Upgrade Instance failed with spec.persistentvolumesource: Forbidden: is immutable after creation.

Error: UPGRADE FAILED: cannot patch "<project>-<instance>-nfs-pv" with kind 
PersistentVolume: PersistentVolume "<project>-<instance>-nfs-pv" is invalid: spec.persistentvolumesource: 
Forbidden: is immutable after creation
Error in upgrading osm helm chart

JMS Servers for Managed Servers are Reassigned to Remaining Managed Servers

When scaling down, the JMS servers for managed servers that do not exist are getting reassigned to remaining managed servers.

Jun 15, 2020 11:01:32,821 AM UTC> <Info> <oracle.communications.ordermanagement.automation.plugin.JMSDestinationListener> <BEA-000000> <
All local JMS destinations: ms1
JNDI                                                                         JMS Server         WLS Server Migratable Target Local    Member Type                   Partition
---------------------------------------------------------------------------- ------------------ ---------- ----------------- -------- ----------------------------- ---------
osm_jms_server@ms1@mslv/oms/oms1/internal/jms/events                         osm_jms_server@ms1 ms1                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN  
osm_jms_server@ms1@oracle.communications.ordermanagement.SimpleResponseQueue osm_jms_server@ms1 ms1                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN  
>

<Jun 15, 2020 11:02:20,461 AM UTC> <Info> <oracle.communications.ordermanagement.automation.plugin.JMSDestinationListener> <BEA-000000> <
All local JMS destinations: ms1
JNDI                                                                         JMS Server         WLS Server Migratable Target Local    Member Type                   Partition
---------------------------------------------------------------------------- ------------------ ---------- ----------------- -------- ----------------------------- ---------
osm_jms_server@ms1@mslv/oms/oms1/internal/jms/events                         osm_jms_server@ms1 ms1                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN
osm_jms_server@ms1@oracle.communications.ordermanagement.SimpleResponseQueue osm_jms_server@ms1 ms1                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN
osm_jms_server@ms2@mslv/oms/oms1/internal/jms/events                         osm_jms_server@ms2 ms2                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN
osm_jms_server@ms2@oracle.communications.ordermanagement.SimpleResponseQueue osm_jms_server@ms2 ms2                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN
>

<Jun 15, 2020 11:02:20,461 AM UTC> <Info> <oracle.communications.ordermanagement.automation.plugin.JMSDestinationListener> <BEA-000000> <
All local JMS destinations: ms1
JNDI                                                                         JMS Server         WLS Server Migratable Target Local    Member Type                   Partition
---------------------------------------------------------------------------- ------------------ ---------- ----------------- -------- ----------------------------- ---------
osm_jms_server@ms1@mslv/oms/oms1/internal/jms/events                         osm_jms_server@ms1 ms1                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN
osm_jms_server@ms1@oracle.communications.ordermanagement.SimpleResponseQueue osm_jms_server@ms1 ms1                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN
osm_jms_server@ms2@mslv/oms/oms1/internal/jms/events                         osm_jms_server@ms2 ms1                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN
osm_jms_server@ms2@oracle.communications.ordermanagement.SimpleResponseQueue osm_jms_server@ms2 ms1                          true     MEMBER_TYPE_CLUSTERED_DYNAMIC DOMAIN
>

This is completely expected behavior. This is a WebLogic feature and not to be mistaken for any inconsistency in the functionality.