Environment: Current Database: Oracle RAC 19.26 Operating System : HP-UX 11.31 Activity: Rollback the patch 39111604 from RDBMS HOME. Issue Description: The following error messages were observed in the patching log file after attempting to roll back patch 39111604 from the RDBMS home. Despite encountering errors during the rollback process, the user proceeded with the rollback. As a result, the patch was successfully rolled back from the database home. However, the linking failed, which is a non-ignorable issue. The main concern now is how to resolve the failed linking step and restore the RDBMS home to a consistent and functional state. Let's troubleshoot the issue.... Error: oracle@testdb1:/u01/app/oracle/product/19.3/OPatch#./opatch rollback -local -id 39111604 -oh /u01/app/oracle/product/19.3 Oracle Interim Patch Installer version 12.2.0.1.46 Copyright (c) 2026, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/19.3 Central Inventory : /oracle12c_RAC/app/oraInventory from : /u01/app/oracle/product/19.3/oraInst.loc OPatch version : 12.2.0.1.46 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-02_21-57-08PM_1.log Patches will be rolled back in the following order: 39111604 The following patch(es) will be rolled back: 39111604 Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/19.3') Is the local system ready for patching? [y|n] y User Responded with: Y Rolling back patch 39111604... RollbackSession rolling back interim patch '39111604' from OH '/u01/app/oracle/product/19.3' Patching component oracle.rdbms, 19.0.0.0.0... Patching component oracle.rdbms.rman, 19.0.0.0.0... Patching component oracle.rdbms.rsf.ic, 19.0.0.0.0... Patching component oracle.rdbms.rsf, 19.0.0.0.0... Patching component oracle.rdbms.util, 19.0.0.0.0... RollbackSession removing interim patch '39111604' from inventory OPatch found the word "warning" in the stderr of the make command. Please look at this stderr. You can re-run this make command. Stderr output: ld: (Warning) Unsatisfied symbol "kollasg" in file /u01/app/oracle/product/19.3/lib//libttsh19.so 1 warning. OPatch found the word "warning" in the stderr of the make command. Please look at this stderr. You can re-run this make command. Stderr output: ld: (Warning) Unsatisfied symbol "kollasg" in file /u01/app/oracle/product/19.3/lib//libttsh19.so 1 warning. Make failed to invoke "/usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback"....'genclntsh: genclntsh: Could not locate /u01/app/oracle/produt Stop. ' The following make actions have failed : Re-link fails on target "client_sharedlib". Do you want to proceed? [y|n] y User Responded with: Y Inactive sub-set patch [37975568] has become active due to the rolling back of a super-set patch [39111604]. Please refer to Doc ID 2161861.1 for any possible further required actions. OPatch Session completed with warnings. Log file location: /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-02_21-57-08PM_1.log OPatch completed with warnings. Troubleshooting: To investigate the above error message, the patching log file was reviewed to gather more details and identify the root cause of the issue. Let’s review the relevant entries from the patching log file. oracle@testdb1:/home/oracle# view /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-02_21-57-08PM_1.log [Apr 2, 2026 9:57:09 PM] [INFO] CAS Dynamic Loading : [Apr 2, 2026 9:57:09 PM] [INFO] CUP_LOG: Trying to load HomeOperations object [Apr 2, 2026 9:57:09 PM] [INFO] CUP_LOG: HomeOperations object created. CUP1.0 is enabled [Apr 2, 2026 9:57:09 PM] [INFO] OPatch invoked as follows: 'rollback -local -id 39111604 -oh /u01/app/oracle/product/19.3 -invPtrLoc /u01/app/oracle/product/19.3/oraInst.loc ' [Apr 2, 2026 9:57:09 PM] [INFO] Runtime args: [-Xmx1536m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/u01/app/oracle/product/19.3/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=rollback, -DOPatch.ORACLE_HOME=/u01/app/oracle/product/19.3, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=., -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/oracle_19C _Night/app/oracle/product/19.3/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=] [Apr 2, 2026 9:57:09 PM] [INFO] Heap in use : 22 MB Total memory: 1472 MB Free memory : 1449 MB Max memory : 1472 MB [Apr 2, 2026 9:57:09 PM] [INFO] Oracle Home : /u01/app/oracle/product/19.3 Central Inventory : /oracle12c_RAC/app/oraInventory from : /u01/app/oracle/product/19.3/oraInst.loc OPatch version : 12.2.0.1.46 OUI version : 12.2.0.7.0 OUI location : /u01/app/oracle/product/19.3/oui Log file location : /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-02_21-57-08PM_1.log [Apr 2, 2026 9:57:09 PM] [INFO] Patch history file: /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch_history.txt ...... [Apr 2, 2026 9:58:36 PM] [INFO] Common File: /u01/app/oracle/product/19.3/lib/libserver19.a:kcvfdb.o, in patch (32545013) and (39111604) "/u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-02_21-57-08PM_1.log" 4188 lines, 626643 characters [Apr 2, 2026 10:02:22 PM] [INFO] The following re-links will eventually be skipped because they are duplicates: iorion, [Apr 2, 2026 10:02:22 PM] [INFO] OUI-67050:Running make for target ioracle [Apr 2, 2026 10:02:22 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:02:22 IST 2026Thu Apr 02 22:02:22 IST 2026 [Apr 2, 2026 10:02:22 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:03:59 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:03:59 IST 2026 [Apr 2, 2026 10:03:59 PM] [INFO] OUI-67050:Running make for target irenamedg [Apr 2, 2026 10:03:59 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:03:59 IST 2026Thu Apr 02 22:03:59 IST 2026 [Apr 2, 2026 10:03:59 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk irenamedg ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:00 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:00 IST 2026 [Apr 2, 2026 10:04:00 PM] [WARNING] OUI-67215: OPatch found the word "warning" in the stderr of the make command. Please look at this stderr. You can re-run this make command. Stderr output: ld: (Warning) Unsatisfied symbol "kollasg" in file /u01/app/oracle/product/19.3/lib//libttsh19.so 1 warning. [Apr 2, 2026 10:04:00 PM] [INFO] OUI-67050:Running make for target ikfod [Apr 2, 2026 10:04:00 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:00 IST 2026Thu Apr 02 22:04:00 IST 2026 [Apr 2, 2026 10:04:00 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk ikfod ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:02 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:02 IST 2026 [Apr 2, 2026 10:04:02 PM] [WARNING] OUI-67215: OPatch found the word "warning" in the stderr of the make command. Please look at this stderr. You can re-run this make command. Stderr output: ld: (Warning) Unsatisfied symbol "kollasg" in file /u01/app/oracle/product/19.3/lib//libttsh19.so 1 warning. [Apr 2, 2026 10:04:02 PM] [INFO] OUI-67050:Running make for target iorion [Apr 2, 2026 10:04:02 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:02 IST 2026Thu Apr 02 22:04:02 IST 2026 [Apr 2, 2026 10:04:02 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk iorion ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:03 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:03 IST 2026 [Apr 2, 2026 10:04:03 PM] [INFO] OUI-67050:Running make for target irman [Apr 2, 2026 10:04:03 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:03 IST 2026Thu Apr 02 22:04:03 IST 2026 [Apr 2, 2026 10:04:03 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk irman ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:05 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:05 IST 2026 [Apr 2, 2026 10:04:05 PM] [INFO] OUI-67050:Running make for target iwrc [Apr 2, 2026 10:04:05 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:05 IST 2026Thu Apr 02 22:04:05 IST 2026 [Apr 2, 2026 10:04:05 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk iwrc ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:07 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:07 IST 2026 [Apr 2, 2026 10:04:07 PM] [INFO] OUI-67050:Running make for target client_sharedlib [Apr 2, 2026 10:04:07 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:07 IST 2026Thu Apr 02 22:04:07 IST 2026 [Apr 2, 2026 10:04:03 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:03 IST 2026 [Apr 2, 2026 10:04:03 PM] [INFO] OUI-67050:Running make for target irman [Apr 2, 2026 10:04:03 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:03 IST 2026Thu Apr 02 22:04:03 IST 2026 [Apr 2, 2026 10:04:03 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk irman ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:05 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:05 IST 2026 [Apr 2, 2026 10:04:05 PM] [INFO] OUI-67050:Running make for target iwrc [Apr 2, 2026 10:04:05 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:05 IST 2026Thu Apr 02 22:04:05 IST 2026 [Apr 2, 2026 10:04:05 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk iwrc ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:07 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:07 IST 2026 [Apr 2, 2026 10:04:07 PM] [INFO] OUI-67050:Running make for target client_sharedlib [Apr 2, 2026 10:04:07 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:07 IST 2026Thu Apr 02 22:04:07 IST 2026 [Apr 2, 2026 10:04:07 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:07 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:07 IST 2026 [Apr 2, 2026 10:04:07 PM] [WARNING] OUI-67200:Make failed to invoke "/usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback"....'genclntsh: genclntsh: Could not locate /u01/app/oracle/product/19.3/network/admin/shrept.lst Stop. ' [Apr 2, 2026 10:04:07 PM] [INFO] Stack Description: java.lang.RuntimeException: genclntsh: genclntsh: Could not locate /u01/app/oracle/product/19.3/network/admin/shrept.lst Stop. ....... ...... at oracle.opatch.MakeAction.apply(MakeAction.java:543) at oracle.opatch.OPatchSession.runMake(OPatchSession.java:757) at oracle.opatch.OPatchSession.runMake(OPatchSession.java:699) at oracle.opatch.MergedPatchObject.invokeConsolidatedMake(MergedPatchObject.java:1903) at oracle.opatch.opatchutil.NRollback.legacy_process(NRollback.java:1112) at oracle.opatch.opatchutil.NRollback.process(NRollback.java:217) at oracle.opatch.opatchutil.OUSession.nrollback(OUSession.java:1154) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at oracle.opatch.UtilSession.process(UtilSession.java:355) at oracle.opatch.OPatchSession.process(OPatchSession.java:2640) at oracle.opatch.OPatch.process(OPatch.java:888) at oracle.opatch.OPatch.main(OPatch.java:945) [Apr 2, 2026 10:04:07 PM] [INFO] OUI-67050:Running make for target isqlldr [Apr 2, 2026 10:04:07 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:07 IST 2026Thu Apr 02 22:04:07 IST 2026 [Apr 2, 2026 10:04:07 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk isqlldr ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:08 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:08 IST 2026 [Apr 2, 2026 10:04:08 PM] [INFO] The following make actions have failed : [Apr 2, 2026 10:04:08 PM] [WARNING] OUI-67124:Re-link fails on target "client_sharedlib". [Apr 2, 2026 10:04:08 PM] [INFO] Do you want to proceed? [y|n] [Apr 2, 2026 10:04:08 PM] [INFO] Start to wait for user-input at Thu Apr 02 22:04:08 IST 2026 [Apr 2, 2026 10:04:33 PM] [INFO] Finish waiting for user-input at Thu Apr 02 22:04:33 IST 2026 [Apr 2, 2026 10:04:33 PM] [INFO] User Responded with: Y [Apr 2, 2026 10:04:33 PM] [INFO] Got online patch property as NULL for the node testdb1 [Apr 2, 2026 10:04:33 PM] [INFO] opatch-external.jar is in /u01/app/oracle/product/19.3/OPatch/jlib/opatch-external.jar [Apr 2, 2026 10:04:33 PM] [INFO] opatch-external.jar is in /u01/app/oracle/product/19.3/OPatch/jlib/opatch-external.jar [Apr 2, 2026 10:04:33 PM] [INFO] OUI-67179 [Apr 2, 2026 10:04:33 PM] [INFO] Loading classpatch: /u01/app/oracle/product/19.3/oui/jlib/OraInstaller.jar:/u01/app/oracle/product/19.3/oui/jlib/OraInstallerNet.jar:/u01/app/oracle/product/19.3/ oui/jlib/OraPrereq.jar:/u01/app/oracle/product/19.3/oui/jlib/share.jar:/u01/app/oracle/product/19.3/oui/jlib/orai18n-mapping.jar:/u01/app/oracle/product/19.3/oui/jlib/xmlparserv2.jar:/oracle_19C_Nig ht/app/oracle/product/19.3/oui/jlib/emCfg.jar:/u01/app/oracle/product/19.3/oui/jlib/ojmisc.jar:./ocm/lib/emocmclnt.jar:./jlib/opatch.jar:./jlib/opatchsdk.jar:./oplan/jlib/automation.jar:./oplan/jlib/apache-commons/commons-cl i-1.0.jar:./jlib/oracle.opatch.classpath.jar:./oplan/jlib/jaxb/activation.jar:./oplan/jlib/jaxb/jaxb-api.jar:./oplan/jlib/jaxb/jaxb-impl.jar:./oplan/jlib/jaxb/jsr173_1.0_api.jar:./oplan/jlib/OsysModel.jar:./oplan/jlib/osysmodel-utils.jar :./oplan/jlib/CRSProductDriver.jar:./oplan/jlib/oracle.oplan.classpath.jar:/u01/app/oracle/product/19.3/jlib/srvm.jar:/u01/app/oracle/product/19.3/jlib/srvmhas.jar [Apr 2, 2026 10:04:33 PM] [INFO] opatch-external.jar is in /u01/app/oracle/product/19.3/OPatch/jlib/opatch-external.jar [Apr 2, 2026 10:04:33 PM] [INFO] Inactive sub-set patch [37975568] has become active due to the rolling back of a super-set patch [39111604]. Please refer to Doc ID 2161861.1 for any possible further required actions. [Apr 2, 2026 10:04:33 PM] [INFO] [OPSR-TIME] Cleaning up backup [Apr 2, 2026 10:04:33 PM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories. [Apr 2, 2026 10:04:07 PM] [WARNING] OUI-67200:Make failed to invoke "/usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback"....'genclntsh: genclntsh: Could not locate /u01/app/oracle/product/19.3/network/admin/shrept.lst Stop. ' [Apr 2, 2026 10:04:07 PM] [INFO] Stack Description: java.lang.RuntimeException: genclntsh: genclntsh: Could not locate /u01/app/oracle/product/19.3/network/admin/shrept.lst Stop. at oracle.opatch.MakeAction.apply(MakeAction.java:543) at oracle.opatch.OPatchSession.runMake(OPatchSession.java:757) at oracle.opatch.OPatchSession.runMake(OPatchSession.java:699) at oracle.opatch.MergedPatchObject.invokeConsolidatedMake(MergedPatchObject.java:1903) at oracle.opatch.opatchutil.NRollback.legacy_process(NRollback.java:1112) at oracle.opatch.opatchutil.NRollback.process(NRollback.java:217) at oracle.opatch.opatchutil.OUSession.nrollback(OUSession.java:1154) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at oracle.opatch.UtilSession.process(UtilSession.java:355) at oracle.opatch.OPatchSession.process(OPatchSession.java:2640) at oracle.opatch.OPatch.process(OPatch.java:888) at oracle.opatch.OPatch.main(OPatch.java:945) [Apr 2, 2026 10:04:07 PM] [INFO] OUI-67050:Running make for target isqlldr [Apr 2, 2026 10:04:07 PM] [INFO] Start invoking 'make' at Thu Apr 02 22:04:07 IST 2026Thu Apr 02 22:04:07 IST 2026 [Apr 2, 2026 10:04:07 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk isqlldr ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback [Apr 2, 2026 10:04:08 PM] [INFO] Finish invoking 'make' at Thu Apr 02 22:04:08 IST 2026 [Apr 2, 2026 10:04:08 PM] [INFO] The following make actions have failed : [Apr 2, 2026 10:04:08 PM] [WARNING] OUI-67124:Re-link fails on target "client_sharedlib". [Apr 2, 2026 10:04:08 PM] [INFO] Do you want to proceed? [y|n] [Apr 2, 2026 10:04:08 PM] [INFO] Start to wait for user-input at Thu Apr 02 22:04:08 IST 2026 [Apr 2, 2026 10:04:33 PM] [INFO] Finish waiting for user-input at Thu Apr 02 22:04:33 IST 2026 [Apr 2, 2026 10:04:33 PM] [INFO] User Responded with: Y [Apr 2, 2026 10:04:33 PM] [INFO] Got online patch property as NULL for the node testdb1 [Apr 2, 2026 10:04:33 PM] [INFO] opatch-external.jar is in /u01/app/oracle/product/19.3/OPatch/jlib/opatch-external.jar [Apr 2, 2026 10:04:33 PM] [INFO] opatch-external.jar is in /u01/app/oracle/product/19.3/OPatch/jlib/opatch-external.jar [Apr 2, 2026 10:04:33 PM] [INFO] OUI-67179 [Apr 2, 2026 10:04:33 PM] [INFO] Loading classpatch: /u01/app/oracle/product/19.3/oui/jlib/OraInstaller.jar:/u01/app/oracle/product/19.3/oui/jlib/OraInstallerNet.jar:/u01/app/oracle/product/19.3/ oui/jlib/OraPrereq.jar:/u01/app/oracle/product/19.3/oui/jlib/share.jar:/u01/app/oracle/product/19.3/oui/jlib/orai18n-mapping.jar:/u01/app/oracle/product/19.3/oui/jlib/xmlparserv2.jar:/oracle_19C_Nig ht/app/oracle/product/19.3/oui/jlib/emCfg.jar:/u01/app/oracle/product/19.3/oui/jlib/ojmisc.jar:./ocm/lib/emocmclnt.jar:./jlib/opatch.jar:./jlib/opatchsdk.jar:./oplan/jlib/automation.jar:./oplan/jlib/apache-commons/commons-cl i-1.0.jar:./jlib/oracle.opatch.classpath.jar:./oplan/jlib/jaxb/activation.jar:./oplan/jlib/jaxb/jaxb-api.jar:./oplan/jlib/jaxb/jaxb-impl.jar:./oplan/jlib/jaxb/jsr173_1.0_api.jar:./oplan/jlib/OsysModel.jar:./oplan/jlib/osysmodel-utils.jar :./oplan/jlib/CRSProductDriver.jar:./oplan/jlib/oracle.oplan.classpath.jar:/u01/app/oracle/product/19.3/jlib/srvm.jar:/u01/app/oracle/product/19.3/jlib/srvmhas.jar [Apr 2, 2026 10:04:33 PM] [INFO] opatch-external.jar is in /u01/app/oracle/product/19.3/OPatch/jlib/opatch-external.jar [Apr 2, 2026 10:04:33 PM] [INFO] Inactive sub-set patch [37975568] has become active due to the rolling back of a super-set patch [39111604]. Please refer to Doc ID 2161861.1 for any possible further required actions. [Apr 2, 2026 10:04:33 PM] [INFO] [OPSR-TIME] Cleaning up backup [Apr 2, 2026 10:04:33 PM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories. You will be still able to rollback patches after this cleanup. Do you want to proceed? [y|n] [Apr 2, 2026 10:04:33 PM] [INFO] Y (auto-answered by -silent) [Apr 2, 2026 10:04:33 PM] [INFO] User Responded with: Y [Apr 2, 2026 10:04:33 PM] [INFO] Deleted the file "/u01/app/oracle/product/19.3/.patch_storage/NRollback/2026-04-02_21-57-08PM/backup/inventory/Clone/clone.xml" [Apr 2, 2026 10:04:33 PM] [INFO] Deleted the file "/u01/app/oracle/product/19.3/.patch_storage/NRollback/2026-04-02_21-57-08PM/backup/inventory/oneoffs/29517242/etc/config/inventory.xml" [Apr 2, 2026 10:04:33 PM] [INFO] Deleted the file "/u01/app/oracle/product/19.3/.patch_storage/NRollback/2026-04-02_21-57-08PM/backup/inventory/oneoffs/29517242/etc/config/actions.xml" [Apr 2, 2026 10:04:33 PM] [INFO] Deleted the file "/u01/app/oracle/product/19.3/.patch_storage/NRollback/2026-04-02_21-57-08PM/backup/inventory/oneoffs/29585399/etc/config/inventory.xml" ..... ..... [Apr 2, 2026 10:04:34 PM] [INFO] Just wrote: /oracle_19c/app/oracle/product/19.3/jdk/jre/man/ja_JP.UTF-8 to file: /u01/app/oracle/product/19.3/.patch_storage/newdirs.txt_tmp [Apr 2, 2026 10:04:34 PM] [INFO] Exiting OPatchUtilHelper::updateNewdirsTxtPS() [Apr 2, 2026 10:04:34 PM] [INFO] UtilSession: N-Rollback done. [Apr 2, 2026 10:04:34 PM] [INFO] -------------------------------------------------------------------------------- [Apr 2, 2026 10:04:34 PM] [INFO] The following warnings have occurred during OPatch execution: [Apr 2, 2026 10:04:34 PM] [INFO] 1) OUI-67215: OPatch found the word "warning" in the stderr of the make command. Please look at this stderr. You can re-run this make command. Stderr output: ld: (Warning) Unsatisfied symbol "kollasg" in file /u01/app/oracle/product/19.3/lib//libttsh19.so 1 warning. [Apr 2, 2026 10:04:34 PM] [INFO] 2) OUI-67215: OPatch found the word "warning" in the stderr of the make command. Please look at this stderr. You can re-run this make command. Stderr output: ld: (Warning) Unsatisfied symbol "kollasg" in file /u01/app/oracle/product/19.3/lib//libttsh19.so 1 warning. [Apr 2, 2026 10:04:34 PM] [INFO] 3) OUI-67200:Make failed to invoke "/usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback"....'genclntsh: genclntsh: Coul d not locate /u01/app/oracle/product/19.3/network/admin/shrept.lst Stop. ' [Apr 2, 2026 10:04:34 PM] [INFO] 4) OUI-67124:Re-link fails on target "client_sharedlib". [Apr 2, 2026 10:04:34 PM] [INFO] -------------------------------------------------------------------------------- [Apr 2, 2026 10:04:34 PM] [SEVERE] OUI-67008:OPatch Session completed with warnings. [Apr 2, 2026 10:04:34 PM] [INFO] Finishing UtilSession at Thu Apr 02 22:04:34 IST 2026 [Apr 2, 2026 10:04:34 PM] [INFO] Log file location: /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-02_21-57-08PM_1.log From the log analysis, it is evident that the user responded with 'Y' during the patching activity, even though the process had already encountered a failure during the make file creation step. As a result, the patch rollback was forcefully completed. However, the library file linking step failed, leaving the RDBMS home in an inconsistent state. Error Message: [Apr 2, 2026 10:04:07 PM] [WARNING] OUI-67200:Make failed to invoke "/usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/19.3 OPATCH_SESSION=rollback"....'genclntsh: genclntsh: Could not locate /u01/app/oracle/product/19.3/network/admin/shrept.lst There is a clear indicative error message in the logs. The error indicates that the genclntsh utility could not locate the shrept.lst file under the $ORACLE_HOME/network/admin directory. This suggests a potential issue with the availability, path, or integrity of the shrept.lst file. Let's verify the existence and accessibility of the shrept.lst file in the specified location. Cause: From the logs, it is observed that the rollback operation failed due to the unavailability of the shrept.lst file located under the $ORACLE_HOME/network/admin directory. Let's verify the availability of the shrept.lst file in the specified location. oracle@testdb1:/u01/app/oracle/product/19.3/network/admin#ls -ltr shrept.lst shrept.lst not found After verification, it was observed that the shrept.lst file was not available at the specified location. Further checks were performed on other nodes/sites, where the file was found to be present. The root cause of the issue has now been identified. The missing shrept.lst file was the reason for the rollback failure. What is shrept.lst ? What is it's significance ? The shrept.lst file located under $ORACLE_HOME/network/admin is an internal Oracle file used primarily during the installation, patching, or linking phase of Oracle Net Services. It is used by the genclntsh script to define which symbols should be explicitly marked as undefined when creating shared client libraries like libclntsh.so. This file is a configuration file and that should not be modified, moved, or deleted. Proceed with implementing a permanent fix by restoring the file from a valid source and ensuring consistency across all nodes to prevent recurrence of the issue. The required shrept.lst file was found to be available on another site with a similar environment. Therefore, the same file will be copied to the affected node to resolve the issue. Solution: Once the file copy activity is completed, review the contents of the shrept.lst file oracle@testdb1:/home/oracle#cd /u01/app/oracle/product/19.3/network/admin oracle@testdb1:/u01/app/oracle/product/19.3/network/admin#ls -tlr drwxr-xr-x 3 oracle oinstall 2048 Oct 29 16:31 bkp drwxr-xr-x 2 oracle oinstall 1024 Mar 14 22:10 oldest -rw-r--r-- 1 oracle oinstall 4092 Mar 14 22:23 listener.ora -rwxr-xr-x 1 oracle oinstall 4736 Mar 16 13:19 16032026_sqlnet.ora -rwxr-xr-x 1 oracle oinstall 5121 Mar 16 13:20 sqlnet.ora -rw-r--r-- 1 oracle oinstall 26105 Mar 16 23:24 16032026_tnsnames.ora -rw-r--r-- 1 oracle oinstall 26582 Mar 18 02:47 tnsnames.ora -rwxr-xr-x 1 oracle oinstall 1536 Apr 2 23:12 shrept.lst oracle@testdb1:/u01/app/oracle/product/19.3/network/admin#cat shrept.lst # function entry points for genclntsh.sh network : snaumihi_inithostinfo network : snaumbg_gmt network : naedpwd_encrypt network : naumbsb_bld_singlebyte network : nlgh network : nsdhctx_establish network : nsdhctx_respond network : nsdhctx_close network : nsdhctx_inuse network : snlsmDscFromString network : snlsmCreate network : snlsmDestroy network : snlsmDscToString network : ngsmutl_init_ul network : ngsmutl_init_com network : ngsmutl_destroy network : ngsmutl_add_value network : ngsmutl_add_type network : ngsmutl_convert_key network : ngsmutl_make_hash_type network : ngsmutl_key_hash network : ngsmutl_key_init network : ngsmutl_key_save network : ngsmutl_key_copy network : ngsmutl_key_free network : ngsmutl_key_cmpns network : ngsmutl_key_cmp_optimistic network : ngsmutl_sort network : ngsmuit_new network : ngsmuit_root network : ngsmuit_find_node network : ngsmuit_find_nodes network : ngsmuit_find_insts network : ngsmuit_insert network : ngsmuit_delete network : ngsmuit_remove_inst_opt network : ngsmuit_add_inst network : ngsmuit_destroy network : ngsmsl_init network : ngsmsl_terminate network : ngsmsl_sdbctx_init network : ngsmsl_sdbctx_terminate network : ngsmsl_key_init network : ngsmsl_shardkey_add_column network : ngsmsl_key2base64 network : ngsmsl_shardkey_free network : ngsmsl_build_topology network : ngsmsl_lookup_insts network : ngsmsl_chunk_down network : ngsmsl_chunk_up network : ngsmsl_remove_inst network : snlsinf network : snlsinfs network : snlsinfwf network : snlsinfi network : snlsinft Follow the below steps to resolve the issue permanently. Step 1: Copy the file shrept.lst located under $ORACLE_HOME/network/admin from good node or another site to problematic node in the same location. Step 2: Relink the ORACLE_HOME binaries. DB must be down to perform below action. cd $ORACLE_HOME/bin ./relink all Step 3: Apply the failed patch 39111604 again. Check for any errors. Step 4: Rollback the patch 39111604 again and check the errors. Let's review the rollback patch log file for any error messages. oracle@testdb1:/home/oracle# view /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-03_00-57-58AM_1.log [Apr 3, 2026 12:57:59 AM] [INFO] CAS Dynamic Loading : [Apr 3, 2026 12:57:59 AM] [INFO] CUP_LOG: Trying to load HomeOperations object [Apr 3, 2026 12:57:59 AM] [INFO] CUP_LOG: HomeOperations object created. CUP1.0 is enabled [Apr 3, 2026 12:57:59 AM] [INFO] OPatch invoked as follows: 'rollback -local -id 39111604 -oh /u01/app/oracle/product/19.3 -invPtrLoc /u01/app/oracle/product/19.3/oraInst.loc ' [Apr 3, 2026 12:57:59 AM] [INFO] Runtime args: [-Xmx1536m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/u01/app/oracle/product/19.3/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=rollback, -DOPatch.ORACLE_HOME=/u01/app/oracle/product/19.3, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=., -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/oracle_19C _Night/app/oracle/product/19.3/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=] [Apr 3, 2026 12:57:59 AM] [INFO] Heap in use : 22 MB Total memory: 1472 MB Free memory : 1449 MB Max memory : 1472 MB [Apr 3, 2026 12:57:59 AM] [INFO] Oracle Home : /u01/app/oracle/product/19.3 Central Inventory : /oracle12c_RAC/app/oraInventory from : /u01/app/oracle/product/19.3/oraInst.loc OPatch version : 12.2.0.1.46 OUI version : 12.2.0.7.0 OUI location : /u01/app/oracle/product/19.3/oui Log file location : /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-03_00-57-58AM_1.log [Apr 3, 2026 12:57:59 AM] [INFO] Patch history file: /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch_history.txt [Apr 3, 2026 12:58:02 AM] [INFO] [OPSR-TIME] Loading raw inventory .... .... [Apr 3, 2026 1:00:22 AM] [INFO] Patches will be rolled back in the following order: 39111604 [Apr 3, 2026 1:00:22 AM] [INFO] Running prerequisite checks... [Apr 3, 2026 1:00:22 AM] [INFO] Running Prereq : checkOPatchCompatibleAgainstOH() ..... ..... [Apr 3, 2026 1:05:33 AM] [INFO] Loading classpatch: /u01/app/oracle/product/19.3/oui/jlib/OraInstaller.jar:/u01/app/oracle/product/19.3/oui/jlib/OraInstallerNet.jar:/u01/app/oracle/product/19.3/ oui/jlib/OraPrereq.jar:/u01/app/oracle/product/19.3/oui/jlib/share.jar:/u01/app/oracle/product/19.3/oui/jlib/orai18n-mapping.jar:/u01/app/oracle/product/19.3/oui/jlib/xmlparserv2.jar:/oracle_19C_Nig ht/app/oracle/product/19.3/oui/jlib/emCfg.jar:/u01/app/oracle/product/19.3/oui/jlib/ojmisc.jar:./ocm/lib/emocmclnt.jar:./jlib/opatch.jar:./jlib/opatchsdk.jar:./oplan/jlib/automation.jar:./oplan/jlib/apache-commons/commons-cl i-1.0.jar:./jlib/oracle.opatch.classpath.jar:./oplan/jlib/jaxb/activation.jar:./oplan/jlib/jaxb/jaxb-api.jar:./oplan/jlib/jaxb/jaxb-impl.jar:./oplan/jlib/jaxb/jsr173_1.0_api.jar:./oplan/jlib/OsysModel.jar:./oplan/jlib/osysmodel-utils.jar :./oplan/jlib/CRSProductDriver.jar:./oplan/jlib/oracle.oplan.classpath.jar:/u01/app/oracle/product/19.3/jlib/srvm.jar:/u01/app/oracle/product/19.3/jlib/srvmhas.jar [Apr 3, 2026 1:05:33 AM] [INFO] opatch-external.jar is in /u01/app/oracle/product/19.3/OPatch/jlib/opatch-external.jar [Apr 3, 2026 1:05:33 AM] [INFO] Inactive sub-set patch [37975568] has become active due to the rolling back of a super-set patch [39111604]. Please refer to Doc ID 2161861.1 for any possible further required actions. [Apr 3, 2026 1:05:33 AM] [INFO] [OPSR-TIME] Cleaning up backup [Apr 3, 2026 1:05:33 AM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories. You will be still able to rollback patches after this cleanup. Do you want to proceed? [y|n] [Apr 3, 2026 1:05:33 AM] [INFO] Y (auto-answered by -silent) [Apr 3, 2026 1:05:33 AM] [INFO] User Responded with: Y [Apr 3, 2026 1:05:33 AM] [INFO] Deleted the file "/u01/app/oracle/product/19.3/.patch_storage/NRollback/2026-04-03_00-57-58AM/backup/inventory/Clone/clone.xml" [Apr 3, 2026 1:05:33 AM] [INFO] Deleted the file "/u01/app/oracle/product/19.3/.patch_storage/NRollback/2026-04-03_00-57-58AM/backup/inventory/oneoffs/29517242/etc/config/inventory.xml" [Apr 3, 2026 1:05:33 AM] [INFO] Deleted the file "/u01/app/oracle/product/19.3/.patch_storage/NRollback/2026-04-03_00-57-58AM/backup/inventory/oneoffs/29517242/etc/config/actions.xml" [Apr 3, 2026 1:05:34 AM] [INFO] -------------------------------------------------------------------------------- [Apr 3, 2026 1:05:34 AM] [SEVERE] OUI-67008:OPatch Session completed with warnings. [Apr 3, 2026 1:05:34 AM] [INFO] Finishing UtilSession at Fri Apr 03 01:05:34 IST 2026 [Apr 3, 2026 1:05:34 AM] [INFO] Log file location: /u01/app/oracle/product/19.3/cfgtoollogs/opatch/opatch2026-04-03_00-57-58AM_1.log Only the relevant portions of the log file have been captured here, not entire log. The logs clearly indicate that the patch rollback has been successfully completed without any issues. ✅ Issue Resolution Confirmation: After applying the above workaround, the patch was successfully rolled back across all the nodes one by one, and no reoccurrence of the issue was observed. 📌 Final Conclusion: This confirms that the issue was successfully resolved by restoring the missing shrept.lst file on the affected node. This action stabilized the environment and prevented further library build failures during critical operations. 🎉 Enjoy the troubleshooting journey!!! 📝 Stay tuned for a detailed blog post on this case !!! |
Thanks for reading this post ! Please comment if you like this post ! Click FOLLOW to get future blog updates !
Thank you for visiting my blog ! Thanks for your comment !