Troubleshooting

WebLogic Kubernetes Operator installation failure

If the WebLogic Kubernetes Operator installation failed with timing out:

Check the status of the operator Helm release using the command helm ls -n <operator-namespace>.
Check if the operator pod is successfully created in the operator namespace.
Describe the operator pod using kubectl describe pod <operator-pod-name> -n <operator-namespace> to identify any obvious errors.

RCU schema creation failure

When creating the RCU schema using create-rcu-schema.sh, the possible causes for RCU schema creation failure are:

Database is not up and running
Incorrect database connection URL used
Invalid database credentials used
Schema prefix already exists

Make sure that all the above causes are reviewed and corrected as needed.

Also drop the existing schema with the same prefix before rerunning the create-rcu-schema.sh with correct values.

Domain creation failure

If the Oracle SOA Suite domain creation fails when running create-domain.sh, perform the following steps to diagnose the issue:

Run the following command to diagnose the create domain job:

$ kubectl logs jobs/<domain_job> -n <domain_namespace>

For example:

$ kubectl logs jobs/soainfra-create-soa-infra-domain-job -n soans

Also run:

$ kubectl describe pod <domain_job> -n <domain_namespace>

Use the output to diagnose the problem and resolve the issue.

Clean up the failed domain creation:
1. Delete the failed domain creation job in the domain namespace using the command:
```
kubectl delete job <domain-creation-job-name> -n
    <domain-namespace>
```
  .
2. Delete the contents of the domain home directory.
3. Drop the existing RCU schema.
Recreate the domain:
1. Recreate the RCU schema
2. Make sure the Persistent Volume and Persistent Volume Claim used for the domain are created with correct permissions and bound together.
3. Rerun the create domain script

Common domain creation issues

A common domain creation issue is error

Failed to build JDBC Connection object

in the create domain job logs.

Stack Trace

Configuring the Service Table DataSource...
fmwDatabase  jdbc:oracle:thin:@orclcdb.soainfra-domain-ns-293-10202010:1521/orclpdb1
Getting Database Defaults...
Error: getDatabaseDefaults() failed. Do dumpStack() to see details.
Error: runCmd() failed. Do dumpStack() to see details.
Problem invoking WLST - Traceback (innermost last):
File "/u01/weblogic/..2021_10_20_20_29_37.256759996/createSOADomain.py", line 943, in ?
File "/u01/weblogic/..2021_10_20_20_29_37.256759996/createSOADomain.py", line 75, in createSOADomain
File "/u01/weblogic/..2021_10_20_20_29_37.256759996/createSOADomain.py", line 695, in extendSoaB2BDomain
File "/u01/weblogic/..2021_10_20_20_29_37.256759996/createSOADomain.py", line 588, in configureJDBCTemplates
File "/tmp/WLSTOfflineIni956349269221112379.py", line 267, in getDatabaseDefaults
File "/tmp/WLSTOfflineIni956349269221112379.py", line 19, in command
Failed to build JDBC Connection object:
   at com.oracle.cie.domain.script.jython.CommandExceptionHandler.handleException(CommandExceptionHandler.java:69)
   at com.oracle.cie.domain.script.jython.WLScriptContext.handleException(WLScriptContext.java:3085)
   at com.oracle.cie.domain.script.jython.WLScriptContext.runCmd(WLScriptContext.java:738)
   at sun.reflect.GeneratedMethodAccessor152.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
com.oracle.cie.domain.script.jython.WLSTException: com.oracle.cie.domain.script.jython.WLSTException: Got exception when auto configuring the schema component(s) with data obtained from shadow table:
Failed to build JDBC Connection object:
ERROR: /u01/weblogic/create-domain-script.sh failed.

This error is reported when there is an issue with database schema access during domain creation. The possible causes are:

Incorrect schema name specified in
```
create-domain-inputs.yaml
```
.
RCU schema credentials specified in the secret
```
soainfra-rcu-credentials
```
are different from the credentials specified while creating the RCU schema using
```
create-rcu-schema.sh
```
.

To resolve these possible causes, check that the schema name and credentials used during the domain creation are the same as when the RCU schema was created.

Server pods not started after applying domain configuration file

When a domain configuration file (YAML) is deployed and no introspector or server pods are initiated, as well as there is no mention of the domain in the operator log, ensure that the domain’s namespace has been configured to be managed by WebLogic Kubernetes Operator.

See Namespace management :: WebLogic Kubernetes Operator for details.

The default value is LabelSelector. The operator will manage namespaces with Kubernetes labels that match the label selector defined by your Helm chart configuration domainNamespaceLabelSelector attribute, which defaults to weblogic-operator=enabled.

Verify if the label weblogic-operator=enabled is specified for domain namespace that is to be managed by the operator, by running the following command:

$ kubectl get ns --selector="weblogic-operator=enabled"

For example, if your domain namespace is soans and the preceding command did not list the soans namespace, then execute the following command for operator to manage the domain namespace:

$ kubectl label namespace soans weblogic-operator=enabled

Ingress controller not serving the domain URLs

To diagnose this issue:

Verify that the Ingress controller is installed successfully. For example, to verify the Traefik Ingress controller status, run the following command:

$ helm list -n traefik
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART               APP VERSION
traefik                  traefik         2               2022-11-30 11:31:18.599876918 +0000 UTC deployed        traefik-20.5.3       v2.9.5
$

Verify that the Ingress controller is setup to monitor the domain namespace. For example, to verify the Traefik Ingress controller manages the soans domain namespace, run the following command and check the values under namespaces section.
```
$ helm get values traefik-operator -n traefik
USER-SUPPLIED VALUES:
kubernetes:
   namespaces:
   - traefik
   - soans
$
```

Verify that the Ingress chart is installed correctly in domain namespace. For example, run the following command:

$ helm list -n soans
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                       APP VERSION
soainfra-traefik        soans           1               2021-10-27 11:24:31.7572791 +0000 UTC   deployed        ingress-per-domain-0.1.0    1.0
$

Verify that the Ingress URL paths and hostnames are configured correctly by running the following commands:

See the following to see the sample commands and output:

$ kubectl get ingress soainfra-traefik -n soans
NAME               CLASS    HOSTS                                                     ADDRESS   PORTS   AGE
soainfra-traefik   <none>   <Hostname>             80      20h
$
$ kubectl describe ingress soainfra-traefik -n soans
Name:             soainfra-traefik
Namespace:        soans
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host                                                     Path  Backends
----                                                     ----  --------
<Hostname>
                                                         /console                   soainfra-adminserver:7001 (10.244.0.123:7001)
                                                         /em                        soainfra-adminserver:7001 (10.244.0.123:7001)
                                                         /weblogic/ready            soainfra-adminserver:7001 (10.244.0.123:7001)
                                                         /soa-infra                 soainfra-cluster-soa-cluster:8001 (10.244.0.126:8001,10.244.0.127:8001)
                                                         /soa/composer              soainfra-cluster-soa-cluster:8001 (10.244.0.126:8001,10.244.0.127:8001)
                                                         /integration/worklistapp   soainfra-cluster-soa-cluster:8001 (10.244.0.126:8001,10.244.0.127:8001)
                                                         /EssHealthCheck            soainfra-cluster-soa-cluster:8001 (10.244.0.126:8001,10.244.0.127:8001)
Annotations:                                               kubernetes.io/ingress.class: traefik
                                                         meta.helm.sh/release-name: soainfra-traefik
                                                         meta.helm.sh/release-namespace: soans
Events:                                                    <none>
$

Security warnings reported in WebLogic Remote console

WebLogic Server regularly validates your domain configuration settings against a set of security configuration guidelines to determine whether the domain meets key security guidelines recommended by Oracle. If your domain does not meet a recommendation for a security configuration setting, a warning is logged in the Security Warnings Report in the WebLogic Remote Console.

See Review Potential Security Issues in Securing a Production Environment for Oracle WebLogic Server for more information.

Disable Remote Anonymous RMI T3 and IIOP Requests

If you see security warning message that Remote Anonymous RMI T3 or IIOP requests are enabled, resolve this warning by setting RemoteAnonymousRMIT3Enabled and RemoteAnonymousRMIIIOPEnabled attributes to false in domain.yaml with JAVA_OPTIONS before starting the domain as shown below:

serverPod:
    # an (optional) list of environment variable to be set on the servers
    env:
    - name: JAVA_OPTIONS
      value: "-Dweblogic.StdoutDebugEnabled=false -Dweblogic.ssl.Enabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.remoteAnonymousRMIT3Enabled=false -Dweblogic.security.remoteAnonymousRMIIIOPEnabled=false"

See link for more details.