OEM silent upgrade from 13.4 to 13.5 failed with WLS_ADMIN_SERVER_PASSWORD not set error

I was working on upgrading my OEM 13.4 installation to 13.5. I followed the Oracle Doc ID 2815000.1 and this Oracle documentation for this activity.

Oracle recommends following this path when upgrading OEM 13.4 to 13.5
1. Upgrade Software-Only with Plug-Ins (Basically install OEM 13.5 Binaries)
2. Configure OEM 13.5 (Upgrade Repository Database to 13.5 and configure OMS on 13.5)

Because of a lack of GUI access to my servers, I decided to follow the silent install approach.

Step 1, to install OEM 13.5 binaries went smooth

1.1 Create response files template
oracle:test@oms /stage: ./em13500_linux64.bin -getResponseFileTemplates -outputLoc /stage/
Launcher log file is /tmp/OraInstall2022-05-04_02-33-20PM/launcher2022-05-04_02-33-20PM.log.
Extracting the installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Done
Copying response file template(s)
to /stage/
  Copying response file template softwareOnlyWithPlugins_install.rsp
  Copying response file template emprereqkit_upgrade.rsp
  Copying response file template upgrade.rsp
  Copying response file template new_install.rsp
  Copying response file template emprereqkit_install.rsp
  Copying response file template softwareOnlyWithPlugins_upgrade.rsp
Finished copying response file template(s)

1.2 Setup my response file.

oracle:test@oms /stage: cat softwareOnlyWithPlugins_upgrade.rsp
RESPONSEFILE_VERSION=2.2.1.0.0
UNIX_GROUP_NAME=dba
INVENTORY_LOCATION=/u01/app/oraInventory
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false
DECLINE_SECURITY_UPDATES=true
MYORACLESUPPORT_USERNAME=<string>
MYORACLESUPPORT_PASSWORD=<string>
INSTALL_UPDATES_SELECTION=skip
STAGE_LOCATION=<string>
MYORACLESUPPORT_USERNAME_FOR_SOFTWAREUPDATES=<string>
MYORACLESUPPORT_PASSWORD_FOR_SOFTWAREUPDATES=<string>
ORACLE_MIDDLEWARE_HOME_LOCATION=/u01/app/oracle/oem
OLD_BASE_DIR=/u01/app/oracle/middleware
ORACLE_HOSTNAME=test.test.com
AGENT_BASE_DIR=/u01/app/oracle/product/agent
SYS_PASSWORD=xxxxx
SYSMAN_PASSWORD=xxxxx
REPOSITORY_BACKUP_DONE=TRUE
DISABLE_DDMP_JOBS=FALSE
EMPREREQ_AUTO_CORRECTION=false
b_upgrade=true
EM_INSTALL_TYPE=NOSEED
CONFIGURATION_TYPE=LATER

1.3 Performed the Silent install

oracle:test@oms /stage: ./em13500_linux64.bin -silent -responseFile /stage/softwareOnlyWithPlugins_upgrade.rsp
Launcher log file is /tmp/OraInstall2022-05-04_02-42-10PM/launcher2022-05-04_02-42-10PM.log.
Extracting the installer . . . . . . . . . . . . . Done
Checking swap space: must be greater than 512 MB.   Actual 16383 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2022-05-04_02-42-10PM
ScratchPathValue :/tmp/OraInstall2022-05-04_02-42-10PM
..
..
Session log file is /tmp/OraInstall2022-05-04_02-42-10PM/install2022-05-04_02-42-10PM.log
Installation in progress

Install successful
Linking in progress
..
..
Warning: You must run the following configuration scripts as the "root" user.
  /u01/app/oracle/oem/allroot.sh
To execute the configuration scripts:
 1. Open a new  terminal window.
 2. Login in as "root".
 3. Run the scripts.
[root@test ~]# /u01/app/oracle/oem/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/app/oracle/oem/root.sh ......
Check /u01/app/oracle/oem/install/root_test_2022-05-04_14-50-56.log for the output of root script
Finished execution of  /u01/app/oracle/oem/root.sh ......

1.4 Apply RU 6 on my OEM 13.5 installation before upgrade.

oracle:test@oemweb0 /u01/app/oracle/oem: export ORACLE_HOME=/u01/app/oracle/oem
oracle:test@oemweb0 /u01/app/oracle/oem: cd /tmp/
oracle:test@oemweb0 /tmp/: $ORACLE_HOME/OMSPatcher/omspatcher apply -analyze -bitonly
OMSPatcher Automation Tool

Copyright (c) 2017, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.9.5.4.0
OUI version        : 13.9.4.0.0
Running from       : /u01/app/oracle/oem
Log file location  : /u01/app/oracle/oem/cfgtoollogs/omspatcher/opatch2022-05-04_15-08-58PM_1.log
..
..
OMSPatcher has saved inventory details for the above component at below location.
"/u01/app/oracle/oem"

For more details on installed patch(es), Please do "/u01/app/oracle/oem/OPatch/opatch lsinventory -details"

OMSPatcher succeeded.

Step 2, Upgrade to OEM 13.5 failed.

2.1 Setup the response file. Referenced this document to setup the response file.

oracle:test@oemweb0 /u01/app/oracle/oem: cat /stage/upgrade.rsp
RESPONSEFILE_VERSION=2.2.1.0.0
UNIX_GROUP_NAME=dba
INVENTORY_LOCATION=/u01/app/oraInventory
INSTALL_UPDATES_SELECTION=skip
STAGE_LOCATION=<string>
MYORACLESUPPORT_USERNAME_FOR_SOFTWAREUPDATES=<string>
MYORACLESUPPORT_PASSWORD_FOR_SOFTWAREUPDATES=<string>
#ORACLE_MIDDLEWARE_HOME_LOCATION=/u01/app/oracle/oem
ORACLE_INSTANCE_HOME_LOCATION=/u01/app/oracle/gc_inst
OLD_BASE_DIR=/u01/app/oracle/middleware
ORACLE_HOSTNAME=test.test.com
ONE_SYSTEM=true
AGENT_BASE_DIR=/u01/app/oracle/product/agent
SYS_PASSWORD=xxxxx
SYSMAN_PASSWORD=xxxxx
EMPREREQ_AUTO_CORRECTION=false
REPOSITORY_BACKUP_DONE=TRUE
PLUGIN_SELECTION=<>
b_upgrade=true
EM_INSTALL_TYPE=NOSEED

2.2 Upgrade seemed to run okay for a while but failed at the final step. I could see messages related to WLS_ADMIN_SERVER_PASSWORD throughout the install.

oracle:test@oms /u01/app/oracle: /u01/app/oracle/oem/sysman/install/ConfigureGC.sh -silent -responseFile /stage/upgrade.rsp
ScratchPathValue :/u01/app/oracle/oem
*************************************
Error Messages:
----------------------------------------------------
* Invalid option(s): sourceLoc
May 05, 2022 8:12:42 AM org.apache.sshd.common.io.DefaultIoServiceFactoryFactory getIoServiceProvider
INFO: No detected/configured IoServiceFactoryFactory using Nio2ServiceFactoryFactory

13NGCHEKAGGREGATE  : oracle.sysman.top.oms
13NGCHEKAGGREGATE  : encap_oms
13NGCHEKAGGREGATE  : OuiConfigVariables
2022-05-05_08-12-59AM: Configuration Assistant "Upgrade Prerequisite Check" is in progress.

oracle.sysman.emCfg.common.CfwExceptionUninitializedParameter: {Parameter:WLS_ADMIN_SERVER_PASSWORD in {Aggregate:ora
        at oracle.sysman.emCfg.core.CfmParameterState.expressData(CfmParameterState.java:720)
..
..
        at oracle.sysman.emCfg.core.ActionPerformer.performMicroStep(ActionPerformer.java:917)
        at oracle.sysman.emCfg.core.ActionPerformer$Performer.run(ActionPerformer.java:1038)
Stopping the Admin Server ...: /u01/app/oracle/middleware/bin/emctl stop oms -all

Check whether the Admin Server port is free.
Admin Server Port : 7102
..
..

2022-05-05_08-14-09AM: Configuration Assistant "Upgrade Prerequisite Check" has Succeeded.

2022-05-05_08-14-09AM: Configuration Assistant "Plugins Prerequisites Check" is in progress.

2022-05-05_08-56-29AM: Configuration Assistant "OMS Configuration" is in progress.

oracle.sysman.emCfg.common.CfwExceptionUninitializedParameter: {Parameter:WLS_ADMIN_SERVER_PASSWORD in {Aggregate:oracle.sysman.top.oms:13.1.0.0.0:common}}: parameter value has never been set and no default is present.
        at oracle.sysman.emCfg.core.CfmParameterState.expressData(CfmParameterState.java:720)
        at oracle.sysman.emCfg.core.CfmParameterState.getValueAsSimplestString(CfmParameterState.java:341)
        at oracle.sysman.emCfg.core.CfmParameterPair.getValueAsSimplestString(CfmParameterPair.java:113)
        at oracle.sysman.emCfg.core.CfmSession.processParameterCommands(CfmSession.java:659)
        at oracle.sysman.emCfg.core.CfmSession.command(CfmSession.java:146)
        at oracle.sysman.emCfg.core.CfmSession.cmd(CfmSession.java:106)
        at oracle.sysman.emCfg.client.CfwSession.cmd(CfwSession.java:194)
        at oracle.sysman.emCfg.client.CfwParameter.getValueAsSimplestString(CfwParameter.java:225)
        at oracle.sysman.oms.util.ParamFactory.getParamString(ParamFactory.java:423)
        at oracle.sysman.oms.util.ParamFactory.getParamString(ParamFactory.java:415)
        at oracle.sysman.oms.util.ParamFactory.fetchConfigParameters(ParamFactory.java:249)
        at oracle.sysman.oms.util.ParamFactory.<init>(ParamFactory.java:82)
        at oracle.sysman.oms.BaseConfiguration.init(BaseConfiguration.java:156)
        at oracle.sysman.oms.OMSConfiguration.perform(OMSConfiguration.java:113)
        at oracle.sysman.oms.BaseConfiguration.invoke(BaseConfiguration.java:211)
        at oracle.sysman.emCfg.core.PerformMicroStep.runJavaClass(PerformMicroStep.java:540)
        at oracle.sysman.emCfg.core.PerformMicroStep.executeMicroStep(PerformMicroStep.java:120)
        at oracle.sysman.emCfg.core.ActionPerformer.performMicroStep(ActionPerformer.java:917)
        at oracle.sysman.emCfg.core.ActionPerformer$Performer.run(ActionPerformer.java:1038)
*** The installation was Successful, but some configuration assistants were failed or cancelled or skipped. ***
Prompt for the allroot.sh
oracle:test@oms /u01/app/oracle:

Tried to run runConfig.sh from new OEM home but that failed as well

oracle:test@oms /u01/app/oracle/middleware: /u01/app/oracle/oem/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/oem MODE=perform ACTION=configure COMPONENT_XML={encap_oms.1_0_0_0_0.xml}
Setting the invPtrLoc to /u01/app/oracle/oem/oraInst.loc

perform - mode is starting for action: configure

oracle.sysman.emCfg.common.CfwExceptionUninitializedParameter: {Parameter:WLS_ADMIN_SERVER_PASSWORD in {Aggregate:oracle.sysman.top.oms:13.1.0.0.0:common}}: parameter value has never been set and no default is present.
        at oracle.sysman.emCfg.core.CfmParameterState.expressData(CfmParameterState.java:720)
        at oracle.sysman.emCfg.core.CfmParameterState.getValueAsSimplestString(CfmParameterState.java:341)
        at oracle.sysman.emCfg.core.CfmParameterPair.getValueAsSimplestString(CfmParameterPair.java:113)
        at oracle.sysman.emCfg.core.CfmSession.processParameterCommands(CfmSession.java:659)
        at oracle.sysman.emCfg.core.CfmSession.command(CfmSession.java:146)
        at oracle.sysman.emCfg.core.CfmSession.cmd(CfmSession.java:106)
        at oracle.sysman.emCfg.client.CfwSession.cmd(CfwSession.java:194)
        at oracle.sysman.emCfg.client.CfwParameter.getValueAsSimplestString(CfwParameter.java:225)
        at oracle.sysman.oms.util.ParamFactory.getParamString(ParamFactory.java:423)
        at oracle.sysman.oms.util.ParamFactory.getParamString(ParamFactory.java:415)
        at oracle.sysman.oms.util.ParamFactory.fetchConfigParameters(ParamFactory.java:249)
        at oracle.sysman.oms.util.ParamFactory.<init>(ParamFactory.java:82)
        at oracle.sysman.oms.BaseConfiguration.init(BaseConfiguration.java:156)
        at oracle.sysman.oms.OMSConfiguration.perform(OMSConfiguration.java:113)
        at oracle.sysman.oms.BaseConfiguration.invoke(BaseConfiguration.java:211)
        at oracle.sysman.emCfg.core.PerformMicroStep.runJavaClass(PerformMicroStep.java:540)
        at oracle.sysman.emCfg.core.PerformMicroStep.executeMicroStep(PerformMicroStep.java:120)
        at oracle.sysman.emCfg.core.ActionPerformer.performMicroStep(ActionPerformer.java:917)
        at oracle.sysman.emCfg.core.ActionPerformer$Performer.run(ActionPerformer.java:1038)

perform - mode finished for action: configure

You can see the log file: /u01/app/oracle/oem/cfgtoollogs/oui/configActions2022-05-05_09-53-20AM.log

Tried to start OEM from 13.4 home but that was not successful as well because my repository schema was already upgraded.

oracle:test@oms /u01/app/oracle/middleware/sysman/config: emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Oracle Management Server version is not compatible with repository version. Oms needs to be Patched.
JVMD Engine is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up
oracle:test@oms /u01/app/oracle/middleware/sysman/config:

At this point I had the option to restore a backup of the repository database to fix the state of the OEM install. My OEM installation was barely used and had very few targets discovered. Hence, I decided to clean up my OEM installation entirely and start installating 13.5 from scratch.

So why did I get the errors with WLS_ADMIN_SERVER_PASSWORD in the first place? I suspect the response file for my upgrade needed this parameter to be set for the upgrade to complete successfully. However, since the Oracle upgrade documentation did not ask for the parameter to be set, I did not do so.

If you are planning a silent upgrade of OEM, I would recommend you set the WLS_ADMIN_SERVER_PASSWORD parameter in response file before starting the upgrade step.

Leave a comment