1. Upgrading Oracle Identity Manager
  2. Troubleshooting the Oracle Identity Manager Upgrade

A Troubleshooting the Oracle Identity Manager Upgrade

If you encounter errors during or after the upgrade of Oracle Identity Manager to 14c (14.1.2.1.0), review the following troubleshooting procedures.

Error CFGFWK-60953: Application or library was not relocated to the new MW home

An error in the reconfiguration templates can result if there are deployments remaining in the original Middleware home. You must delete the deployments from the Middleware home before running the Reconfiguration Wizard.

While executing the Reconfiguration Wizard on a domain that was created in 12.2.1.3.0 and then upgraded to 12.2.1.4.0, errors can occur if there are extraneous deployments remaining in the domain.

Correct the reconfiguration template as detailed in the error. In this case it is the ""jax-rs(2,2.22.4.0)" library and this is only seen when the environment being upgraded to 14c was upgraded from 12.2.1.3.0 to 12.2.1.4.0.
  1. Using the 12c WebLogic Admin Console, access and log into the WLS admin Console for the OAM WebLogic Domain
  2. Select "Deployments"
  3. Navigate the deployment called "jax-rs(2,2.22.4.0)" and select the checkbox.
  4. Click Delete.

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

Reconfig.sh OPSS Processing Phase Generates an ORA-00001

The OPSS schema uses sequences to generate next values for some of its tables. If one or some of the sequences next values are lower than the value maximum value in the tables, then they need to be changed.

In the OPSS schema run the following: 

SELECT sequence_name, last_number FROM all_sequences WHERE sequence_owner = '<Prefix>_OPSS';

The SELECT sequence_name...... query will show what the database sees as the next sequence number to be used for the various tables.

Run the following queries:

Select max(entryid) from jps_dn;
Select max(jps_attrs_id) from jps_attrs; 
Select max(logid) from jps_changelog;

If any of these return a higher number than that from the sequence_name query, increment that sequence to a higher value: 

ALTER SEQUENCE xxx INCREMENT BY N

Where the xxx is the sequence being >= last_number

Make N greater than the value returned in the jps-dn, jps_attrs, and jps_changelog queries

Run the Reconfiguration Wizard again. .

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

Oracle Identity Manager Server Throws OutOfMemoryError

When you start the servers post upgrade, OutOfMemoryError is thrown.

The following error is seen in the OIM server logs for this issue:

[oim_server1] [NOTIFICATION] [] 
[oracle.iam.oimdataproviders.impl] [tid: [ACTIVE].ExecuteThread: '9' for 
queue: 'weblogic.kernel.Default (self-tuning)'] [userId: xelsysadm] [ecid: 
5679ce10-f0df-457f-88f1-6bc04e10aa13-000013b1,0] [APP: oim-runtime] 
[partition-name: DOMAIN] [tenant-name: GLOBAL] [DSID: 
0000Lg0PPYTBd5I_Ipt1if1OpGGi00000U] RM_DEBUG_PERF - 2017-03-24 06:09:51.087 - 
search criteria = arg1 = (usr_key) EQUAL arg2 = (1)[[ 
 query = Select usr.usr_key, usr.usr_status  from usr where usr.usr_key = ? 
 time = 1 
]] 
[2017-03-24T06:09:52.286-07:00] [oim_server1] [NOTIFICATION] [] 
[oracle.iam.oimdataproviders.impl] [tid: [ACTIVE].ExecuteThread: '9' for 
queue: 'weblogic.kernel.Default (self-tuning)'] [userId: xelsysadm] [ecid: 
5679ce10-f0df-457f-88f1-6bc04e10aa13-000013b1,0] [APP: oim-runtime] 
[partition-name: DOMAIN] [tenant-name: GLOBAL] [DSID: 
0000Lg0PPYTBd5I_Ipt1if1OpGGi00000U] 
oracle.iam.oimdataproviders.impl.OIMUserDataProvider 
[2017-03-24T06:11:52.171-07:00] [oim_server1] [ERROR] [ADFC-50018] 
[oracle.adfinternal.controller.application.AdfcExceptionHandler] [tid: 
[ACTIVE].ExecuteThread: '27' for queue: 'weblogic.kernel.Default 
(self-tuning)'] [userId: xelsysadm] [ecid: 
5679ce10-f0df-457f-88f1-6bc04e10aa13-000013e0,0] [APP: 
oracle.iam.console.identity.self-service.ear] [partition-name: DOMAIN] 
[tenant-name: GLOBAL] [DSID: 0000Lg0RtM9Bd5I_Ipt1if1OpGGi00000V] ADFc: No 
exception handler was found for an application exception.[[ 
java.lang.OutOfMemoryError: GC overhead limit exceeded ]

To resolve this issue, do the following (on Linux):

  1. Ensure that you set the following parameters in the /etc/security/limits.conf file, to the specified values:
    FUSION_USER_ACCOUNT soft nofile 32767
FUSION_USER_ACCOUNT hard nofile 327679
  2. Ensure that you set UsePAM to Yes in the /etc/ssh/sshd_config file.
  3. Restart sshd.
  4. Log out (or reboot) and log in to the system again.
Before you start the Oracle Identity Manager 12c Server, run the following command to increase the limit of open files, so that you do not hit into memory issues:

limit maxproc 16384

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

Failure in UPDATE_WORKFLOW_POLICIES Post-Bootstrap Task

The UPDATE_WORKFLOW_POLICIES post-bootstrap task fails when you start the OIM Managed server after the upgrade.

The OIM Managed server displays the following error message:
Update WF policies started. Update SOA composite name from
default/DefaultRequestApproval!5.0 to default/DefaultRequestApproval!6.0>
<Apr 13, 2021 5:09:50,451 PM UTC> <Error> <OIM Authenticator> <BEA-000000>
<Authentication of user xelsysadm failed because of invalid password>

The OIM Managed server fails because the OIM administrator password is incorrect in the CSF keys.

Solution

Ensure that the OIM administrator (xelsysadm) password is same and correct in the following CSF keys:

Table A-1 OIM Managed Server CSF Keys

Sl. No CSF Map CSF Key

1.

oracle.wsm.security

OIMAdmin

2.

oim

sysadmin

To correct the password of the CSF keys:

  1. Log in to the Oracle Enterprise Manager Console with the WebLogic administrator credentials.
  2. From the WebLogic Domain drop-down, select Security, and then Credentials.
  3. On the Credentials page, expand the oim CSF map, select the sysadmin CSF key, and then click the Edit icon to change the XELSYSADM credentials from the pop-up window.
  4. Repeat Step 3 for the OIMAdmin CSF Key under oracle.wsm.security CSF Map.

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

MDS Customizations are Removed After You Restart the OIM Managed Server of an Upgraded Setup

If any MDS customizations are done after a successful upgrade to 14c (14.1.2.1.0) and if those customizations are lost after you restart the OIM Managed Server, you cannot recover the MDS changes. You have to do the MDS customizations again.

To avoid the repeated occurrence of this issue each time you restart the Managed Server, replace the existing 14c (14.1.2.1.0)_ORACLE_HOME>/idm/server/apps/oim.ear/metadata.tar file with the file that is present at the same location after you install the 14c (14.1.2.1.0) binaries, prior to the upgrade.

Note:

This issue is applicable only for MDS customizations that were made after the successful upgrade to 12c but lost after restarting the OIM Managed Server.

As part of the pre-upgrade tasks, after installing the 14c (14.1.2.1.0) binaries, you would have already taken a backup of the original 14c (14.1.2.1.0)_ORACLE_HOME>/idm/server/apps/oim.ear/metadata.tar file. See Backing Up the metadata.mar File Manually.

If the backup of the original file is not present after you install the binaries, you should install the 14c (14.1.2.1.0) binaries at any temporary location and extract the file.

For a HA setup, the original 14c (14.1.2.1.0)_ORACLE_HOME>/idm/server/apps/oim.ear/metadata.tar file is present on the secondary nodes where upgrade bootstrap was not executed.

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

OPatch Fails for not Finding the 'fuser' Command

OPatch fails when it is unable to locate the fuser command.

OPatch fails with the following error on the command line:

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Exception occured : fuser could not be located:
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
Log file location: <PATH>/fmw/cfgtoollogs/opatch/opatch20xx-0x-20_11-40-12AM_1.log

Following options are available to resolve this issue:

Pass argument for OPatch to ignore fuser and continue with patching:

  1. Set the environment variable OPATCH_NO_FUSER=true. Setting this variable to "true" informs OPatch to skip the check for active executables.
  2. Shut down the WebLogic instances.
  3. Run the OPatch utility.
Set a temporary fuser:
  1. Set /tmp in your PATH.
  2. Create an empty file named "fuser".
  3. Shut down the WebLogic instances.
  4. Run the OPatch utility.
Install the 'fuser' utility:
  1. Install the 'fuser' utility on the machine (contact your OS Admin).
  2. Ensure that 'fuser' is located under /sbin/fuser or /bin/fuser.
  3. Shut down the WebLogic instances.
  4. Run the OPatch utility.

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

Administration Server Has a Slow Start After the Upgrade

The Administration Server experiences a slow start after the upgrade.

The thread dump displays the following information:
[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default
(self-tuning)'" #76 daemon prio=5 os_prio=0 tid=0x00007f4fcc008000 nid=0x20c6
runnable [0x00007f4fbc2d6000]
  java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:424)
at
sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:441
)
- locked <0x0000000640b92be8> (a java.lang.Object)
at sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:331)
at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:226)
at java.security.SecureRandom.generateSeed(SecureRandom.java:546)
at
com.bea.security.utils.random.AbstractRandomData.ensureInittedAndSeeded(Abstra
ctRandomData.java:92)
- locked <0x000000075b7af6b8> (a
com.bea.security.utils.random.SecureRandomData)
at
com.bea.security.utils.random.AbstractRandomData.getRandomLong(AbstractRandomD
ata.java:117)
- locked <0x000000075b7af6b8> (a
com.bea.security.utils.random.SecureRandomData)

To resolve this issue, set the -Djava.security.egd=file:/dev/./urandom parameter in the JAVA_OPTIONS section of the setDomainEnv.sh/cmd file and restart the server.

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

NPE Encountered on Starting OIM Server After Running the Upgrade Assistant

A Null Pointer Exception (NPE) is encountered when starting the OIM server after running the Upgrade Assistant for upgrading the domain configuration.

The OIM server fails to start and displays the following error message:
Exception[[
java.lang.NullPointerException
   at
oracle.iam.rcu.LoadTemplateDataLogger.writeLog(LoadTemplateDataLogger.java:31)

   at
oracle.iam.rcu.LoadTemplates.loadAllTempalteImplementation(LoadTemplates.java:
113)
   at oracle.iam.rcu.LoadTemplates.loadAllTemplates(LoadTemplates.java:168)
   at
oracle.iam.OIMPostConfigManager.config.OIMConfigManager.seedNotificationTempla
te(OIMConfigManager.java:2866)
   at
oracle.iam.OIMPostConfigManager.config.OIMConfigManager.executeAndRegisterTask
(OIMConfigManager.java:1754)
   at
oracle.iam.OIMPostConfigManager.config.OIMConfigManager.configureOIM(OIMConfig
Manager.java:1558)
   at
oracle.iam.OIMPostConfigManager.config.OIMConfigManager.doExecute(OIMConfigMan
ager.java:1179)
   at
oracle.iam.OIMPostConfigManager.appListener.BootStrapListener.preStart(BootStr
apListener.java:134)

To resolve this error, you should include /idm in the value of ORACLE_HOME in the setDomainEnv.sh file.

For example: /u01/oracle/product/ORACLE_HOME/idm

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

OIM Bootstrap Fails Due to the Presence of Custom Application JARs

If there are any custom developed libraries or JARs placed inside the OIM_HOME, the OIM bootstrap fails during the upgrade to Oracle Identity Manager 14c (14.1.2.1.0).

The failure results in an error message similar to the following:
<Server state changed to FORCE_SHUTTING_DOWN.>
<Nov 19, 2020 4:04:50,356 PM EST> <Notice> <Log Management> <BEA-170037> <The
log monitoring service timer has been stopped.>
<Nov 19, 2020 4:06:16,377 PM EST> <Warning> <JMX> <BEA-149513> <JMX Connector
Server stopped at
service:jmx:iiop://idmoimtl3.chop.edu:14000/jndi/weblogic.management.mbeanserv
ers.runtime.>
<Nov 19, 2020 4:15:43,045 PM EST> <Error> <netuix> <BEA-423142> <The control
com.bea.netuix.servlets.controls.layout.Layout could not be rendered properly
due to the following error:>
<Nov 19, 2020 4:15:44,356 PM EST> <Warning> <Socket> <BEA-000449> <Closing
the socket, as no data read from it on 10.250.116.181:54,532 during the
configured idle timeout of 5 seconds.>
<Nov 19, 2020 4:17:57,525 PM EST> <Warning> <J2EE> <BEA-160188> <Unresolved
application library references, for application
oracle.iam.console.identity.self-service.ear, defined in
weblogic-application.xml: [Extension-Name: oracle.iam.ui.model, exact-match:
false].>
<Nov 19, 2020 4:17:57,810 PM EST> <Warning> <J2EE> <BEA-160188> <Unresolved
WebApp library references defined in weblogic.xml, of module
'oracle.iam.console.identity.self-service.war' [Extension-Name:
oracle.iam.ui.view, exact-match: false], [Extension-Name:
oracle.iam.ui.oia-view, exact-match: false], [Extension-Name:
oracle.iam.ui.custom, exact-match: false], [Extension-Name:
oracle.idm.msm.ui.library, exact-match: false].>
java.lang.ClassNotFoundException:
oracle.iam.ui.platform.view.backing.SkinBean at
weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoad
er.java:1029) at
weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.ja
va:990)  at
weblogic.utils.classloaders.GenericClassLoader.doFindClass(GenericClassLoader.
java:611) at
weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.ja
va:543) at
weblogic.servlet.internal.AnnotationProcessingManager.processAnnotations(Annot
ationProcessingManager.java:105) at
weblogic.servlet.tools.WARModule.processAnnotations(WARModule.java:513) at
weblogic.servlet.tools.WARModule.processAnnotations(WARModule.java:605) at
weblogic.servlet.tools.WARModule.merge(WARModule.java:553) at
weblogic.application.compiler.ToolsModuleWrapper.merge(ToolsModuleWrapper.java
:96) at
weblogic.application.utils.CustomModuleManager.merge(CustomModuleManager.java:
78) at
weblogic.application.compiler.flow.MergeModuleFlow.compile(MergeModuleFlow.jav
a:38) at
weblogic.application.compiler.FlowDriver$FlowStateChange.next(FlowDriver.java:
70) at
weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.jav
a:45) at
weblogic.application.compiler.FlowDriver.nextState(FlowDriver.java:37)  
weblogic.application.compiler.flow.AppMergerFlow.mergeInput(AppMergerFlow.java
:75)at
weblogic.application.compiler.flow.AppMergerFlow.compile(AppMergerFlow.java:40
) at
weblogic.application.compiler.FlowDriver$FlowStateChange.next(FlowDriver.java:
70) at
weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.jav
a:45) at
weblogic.application.compiler.FlowDriver.nextState(FlowDriver.java:37) at
weblogic.application.compiler.AppMerge.runBody(AppMerge.java:168) at
weblogic.utils.compiler.Tool.run(Tool.java:159) at
weblogic.utils.compiler.Tool.run(Tool.java:116) at
weblogic.application.compiler.AppMerge.merge(AppMerge.java:198) at
weblogic.deploy.api.internal.utils.AppMerger.merge(AppMerger.java:94)at
weblogic.deploy.api.internal.utils.AppMerger.getMergedApp(AppMerger.java:58)
at
weblogic.deploy.api.model.internal.WebLogicDeployableObjectFactoryImpl.createD
eployableObject(WebLogicDeployableObjectFactoryImpl.java:186) at
weblogic.deploy.api.model.internal.WebLogicDeployableObjectFactoryImpl.createD
eployableObject(WebLogicDeployableObjectFactoryImpl.java:167)at
com.bea.console.utils.DeploymentConfigurationHelper$1.execute(DeploymentConfig
urationHelper.java:860) at
com.bea.console.utils.DeploymentUtils.runDeploymentAction(DeploymentUtils.java
:5690) at
com.bea.console.utils.DeploymentConfigurationHelper.initDeploymentConfiguratio
n(DeploymentConfigurationHelper.java:848) at
com.bea.console.utils.DeploymentConfigurationHelper.completeInitialization(Dep
loymentConfigurationHelper.java:444) at
com.bea.console.utils.DeploymentConfigurationManager.getDeploymentConfiguratio
n(DeploymentConfigurationManager.java:151) at
com.bea.console.utils.DeploymentConfigurationManager.getDeploymentConfiguratio
n(DeploymentConfigurationManager.java:104) at

To resolve this issue, Oracle recommends not to keep the custom-developed JARs or libraries inside OIM_HOME to avoid file system dependencies. The file system dependencies add an overhead of maintaining such custom libraries during the out-of-place Oracle Home upgrades because such custom JARs remain in the old Oracle Home (Oracle Home before the upgrade process).

To avoid such issues, you should upload the custom libraries to the database. If the custom library is in the OIM plug-in compressed (.zip) format, register them using the plug-in utility. If the custom library is a JAR, upload the same to the database using the Upload JAR Utility.

If for some reason, you do not want to follow the above recommendations, you can manually copy the custom-developed JARs from the old to the new Oracle home, in the appropriate location.

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade

Incorrect Links in Password Reset Emails

The OIG system generated password reset email has links in the applewebdata://<ANY_RANDOM_GUID>/null format, which is incorrect.

To resolve this issue, update the OIMExternalFrontEndURL parameter with the correct value in the Discovery MBean of OIM by completing the following steps:
  1. Log in to the Enterprise Manager Console.
  2. Navigate to System MBean Browser.
  3. Under Application Defined MBeans, navigate to oracle.iam, select Server <server>, click Application:oim, click XMLConfig, select Config, select XMLConfig.DiscoveryConfig, and then click Discovery.
  4. Update the OIMExternalFrontEndURL parameter with the appropriate value. This parameter should not be empty.

Parent topic: Troubleshooting the Oracle Identity Manager Upgrade