Note: This document varies as per Patch Number. Open the patch Readme File and follow the instructions. I have not used OCM file method here. This document is valid only for Apr-2017 patch. Environment configuration: Operating System: Oracle Enterprise Linux 6.3 64-bit Database Name: DC GRID_HOME: /u01/app/11.2.0/gridORACLE_HOME: /u01/app/oracle/product/11.2.0/dbhome_1RDBMS Version: 11.2.0.4 Node1: rac1 Node2: rac2 Grid & Oracle Homes are not shared. ACFS is not used. |
There are total three methods by which we can apply patch to RAC Cluster Environment on Unix System:
In this mode, all instances must be down during the whole patching process. OPatch applies the patch to the local node first, then propagates the patch to all the other nodes, and finally updates the inventory.
In this mode, OPatch patches the local node, asks users for a sub-set of nodes, which will be the first subset of nodes to be patched. After the initial subset of nodes are patched, Opatch propagates the patch to the other nodes and finally updates the inventory. The downtime would happen between the shutdown of the second subset of nodes and the startup of the initial subset of nodes patched.
With this method, there is no downtime. Each node would be patched and brought up while all the other nodes are up and running, resulting in no disruption of the system. Rolling patching strategy incur no downtime, however, some rolling patches may incur downtime due to post-installation steps, i.e. running sql scripts to patch the actual database. Please refer to patch readme to find out whether post-installation steps require downtime or not.
The algorithm used to decide which method is going to be used is the following: If (users specify minimize_downtime) patching mechanism = Min. Downtime else if (patch is a rolling patch) patching mechanism = Rolling else patching mechanism = All-Node When patches are released, they have a tag as "rolling" or "not rolling" patch. While most patches can be applied in a rolling fashion, some patches cannot be applied in this fashion. Patches that could potentially be installed on rolling fashion include:
Note: If you plan to apply a patch that is marked as "not rolling" and want to check if is possible to take advantage of the rolling patch strategy, please contact Oracle Support. How to determine if a patch is a "rolling patch" or not?As database user execute the following: - 9i or 10gR1: opatch query -is_rolling - 10gR2: opatch query -all [unzipped patch location] | grep rolling - 10gR2 on Windows: opatch query -all [unzipped patch location] | findstr rolling - Later 10gR2 or 11g: opatch query -is_rolling_patch [unzipped patch location] Also, the type of OPatch can be determined by checking the <Patch Number>/etc/config/inventory file. If the variable online_rac_installable is set to true then the patch is a rolling patch. |
Here we are using 3rd method of applying patch i.e. “Rolling Patch”. ====================================================== Start Here Patch 25476126 - Oracle Grid Infrastructure Patch Set Update 11.2.0.4.170418 Released: Apr 18, 2017 Some Important Notes:
|
Check whether below components are configured in your environment. Q: Is Audit Vault enabled in your database? Ans: NO Q: Is Dataguard configured in your environment? Ans: NO Note: If Dataguard is configured in your environment then apply the patch DR first. Then you can start to apply patch to production DC. |
- To get total number of servers to be patched.
- To download a patch.
- To upload the patch to target servers.
- To perform prerequisites.
- To confirm client if you find any conflict in prerequisites.
- To inform client that we are ready for patching if no conflict found.
Prerequisites Check: 1. Apply patch in UAT and Development: First apply patch in your UAT and development servers and then Production servers.2. Minimum Opatch Version: You must use the OPatch utility version 11.2.0.3.6 or later to apply this patch. Oracle recommends that you use the latest released OPatch version for 11.2 releases, which is available for download from My Oracle Support patch 6880880 by selecting ARU link for the 11.2.0.0.0 release. Replace the same with existing opatch version by taking backup of the current one. $opatch version => It should be 11.2.0.3.6 or later. 3. Download Patch: Download latest PSU patch for the existing Operating System and correct Oracle Version. For Example: We have Oracle Linux 6.3 64-bit with Oracle 11.2.0.4 Enterprise Edition Create new directory on target server and give read permission to the ORA_INST group and this directory must be empty because the PSU patch may include multiple patches and you may notice that there are multiple directories created with distinct bug numbers. unzip p25476126_112040_<platform>.zip4. Perform prerequisites: You will get any conflict if any after executing below commands. If you will not get any conflict then you are OK to go ahead. As a grid user: $opatch lsinventory –detail –oh $GRID_HOME $opatch prereq CheckConflictAgainstOHWithDetail –ph ./ As an oracle user: $opatchlsinventory –detail –oh $ORACLE_HOME $opatch prereq CheckConflictgainstOhWithDetail –ph ./ Also execute below commands and get the inventory details. As a grid user: $opatch lsinventory –detail –oh $GRID_HOME As an oracle user: $opatch lsinventory –detail –oh $ORACLE_HOME 5. Client Confirmation: Initiate mail to client at least a week prior the Patching Activity with below detail stating that “We have completed patching prerequisites and we did not find any conflict”. |
Start Here (Plan of Action) We are applying latest PSU patch using Rolling Patch Method. Note: We are following 3rd method of applying patching i.e. Rolling Patch and hence do not touch 2nd node while applying patch to 1st node. Once done with 1st node, then you can proceed with 2nd node. Also, this document is for Non-Shared Cluster Environment. For Cluster shared environment, there is a different method to apply patch. Note: Initiate mail to client prior the activity that “We are starting patching activity in next 10-15 minutes”. Step 1: Backup Take tar backup of Both Grid and Oracle Home By grid User: $tar -cvzf /backup_location/TAR_BKP_Oracle_Binaries.tar.gz $GRID_HOME By Oracle User: $tar -cvzf /backup_location/TAR_BKP_Oracle_Binaries.tar.gz $ORACLE_HOME Step 2: Data Capture Capture below details prior the patching activity. By Grid User: $opatch lsinventory –detail –oh $GRID_HOME $crsctl stat res –t $crsctl check crs $id By Oracle user: $opatch lsinventory –detail –oh $ORACLE_HOME SQL>set lines 300 pages 3000 SQL>col action_time for a34 SQL>col version for a14 SQL>col comments for a28 SQL>col bundle_series for a28 SQL> col action_name for a23 SQL> col action for a10 SQL> col namespace for a9 SQL> select action,bundle_series,comments,action_time,version from registry$history; $srvctl status database –d DC -v -f $srvctl config database –d DC -v -f $ps –ef | grep pmon $ps –ef | grep tns $id Step 3: Stop Agent Process You must stop the EM agent processes running from the database home, prior to patching the Oracle RAC database or GI Home $ps –ef | grep agent $cd $AGENT_HOME/bin $./emctl status agent $./emctl stop agent Step 4: Stop Oracle services
The below command will stop all Oracle services corresponds to mentioned Oracle Home. Please do not delete status.log file which will be required later to start the service.By oracle user:$srvctl stop home -o $ORACLE_HOME -s status.log -n rac1 Step 5: Unmount ACFS if available Unmount the Oracle ACFS file systems. See My Oracle Support Document 1494652.1 for unmounting ACFS file systems. In our environment, we are not using ACFS (ASM Cluster File System). Step 6: Run pre-root script Go to the GI Home location and execute below command as the root user: #/u01/app/11.2.0/grid/crs/install/rootcrs.pl -unlock Step 7: Apply Grid Patch Execute below command as a grid user: $/u01/app/11.2.0/grid/OPatch/opatch napply -oh /u01/app/11.2.0/grid -local /home/oracle/25476126Step 8: Run pre-script for Database Home $/home/oracle/25476126/23054319/custom/server/23054319/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_1 /home/oracle/25476126/23054319/custom/server/23054319/custom/scripts/prepatch.sh completed successfully. Step 9: Apply DB Patch Execute below commands by oracle user. $/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply -oh /u01/app/oracle/product/11.2.0/dbhome_1 –local /home/oracle/25476126/23054319/custom/server/23054319 Step 10: Run Post Script for Database Home Run below command by oracle user. $/home/oracle/25476126/23054319/custom/server/23054319/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_1Note: The above command will read the file permissions of below utilities from params.ora file, parse it, verify it and then finally reapply it. Reading /u01/app/oracle/product/11.2.0/dbhome_1/install/params.ora.. Step 11: Post Script for Grid By Root User:#/u01/app/11.2.0/grid/rdbms/install/rootadd_rdbms.shStep 12: Post Root Script #/u01/app/11.2.0/grid/crs/install/rootcrs.pl -patch Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Installing Trace File Analyzer CRS-4123: Oracle High Availability Services has been started. Note: If you are using ACFS and the message “A system reboot is recommended before using ACFS” is shown then a reboot must be issued before continuing. Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver. Step 13: Start Oracle services The below command will start all Oracle services using status.log file corresponds to mentioned Oracle Home. By oracle user:$srvctl start home -o $ORACLE_HOME -s status.log -n rac1 Step 14: Post Patch Verification Check everything is running fine on 1st node before applying patch to 2nd node. By Grid User: $crscrt check crs $crsctl stat res –t $opatch lsinventory –detail –oh $GRID_HOME By Oracle User: $srvctl status database -d DC -v -f $opatch lsinventory –detail –oh $ORACLE_HOME Check latest patch has been applied successfully to both GRID and Oracle Homes. |
Apply patch to 2nd Node Perform all steps from Step 1 to Step 15. Step 1: Backup Step 2: Data Capture Step 3: Stop Agent Process Step 4: Stop Oracle services $srvctl stop home -o $ORACLE_HOME -s status.log -n rac2Step 5: Unmount ACFS if available. Step 6: Run pre-root script Step 7: Apply Grid Patch Step 8: Run pre-script for Database Home Step 9: Apply DB Patch Step 10: Run Post Script for Database Home Step 11: Post Script for Grid Step 12: Post Root Script Step 13: Start Oracle services $srvctl start home -o $ORACLE_HOME -s status.log -n rac2Step 14: Post Patch Verification Step 15: Registry Update The following steps load modified SQL files into the database. For an Oracle RAC environment, perform these steps on only one node. The catbundle.sql execution is reflected in the dba_registry_history view by a row associated with bundle series PSU.cd $ORACLE_HOME/rdbms/admin Verify below logfiles for any errors.catbundle_PSU_<database SID>_APPLY_<TIMESTAMP>.log catbundle_PSU_<database SID>_GENERATE_<TIMESTAMP>.log Step 16: Upgrade Catalog If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. $ rman catalog username/password@alias You have successfully applied latest PSU patch to both Grid and Oracle Homes. |
Rollback Plan Execute the following manual steps on each node of the cluster in non-shared CRS and DB home environment to roll back the patches. Step 1: Stop Oracle services
The below command will stop all Oracle services corresponds to mentioned Oracle Home. Please do not delete status.log file which will be required later to start the service.By oracle user:$srvctl stop home -o $ORACLE_HOME -s status.log -n rac1 Step 2: Unmount ACFS if available Unmount the Oracle ACFS file systems. See My Oracle Support Document 1494652.1 for unmounting ACFS file systems. Step 3: Run pre-root Script Go to the GI Home location and execute below command as the root user: #/u01/app/11.2.0/grid/crs/install/rootcrs.pl -unlockStep 4: Rollback Grid Infrastructure Patch By Grid User: $ /u01/app/11.2.0/grid/OPatch/opatch rollback -local -id 23054319 -oh /u01/app/11.2.0/grid Step 5: Pre-script for Database Home By Oracle User: $/home/grid/25476126/23054319/custom/server/23054319/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_1 Step 6: Rollback Oracle Database Patch By Oracle User: $opatch rollback -local -id 23054319 -oh /u01/app/oracle/product/11.2.0/dbhome_1 Step 7: Post Script for Database Component By Oracle User: $/home/grid/25476126/23054319/custom/server/23054319/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_1 Step 8: Run post root script #/u01/app/11.2.0/grid/rdbms/install/rootadd_rdbms.sh Note: If you are using ACFS and the message “A system reboot is recommended before using ACFS” is shown then a reboot must be issued before continuing. Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver. Step 9: Start Oracle services The below command will start all Oracle services using status.log file corresponds to mentioned Oracle Home. By oracle user: Note: Verify everything is running fine on Node1 before rolling back patch on 2nd node. Follow the steps of "Rollback Patch" from Step1 to Step9 on Node 2. Step 1: Stop Oracle services $srvctl stop home -o $ORACLE_HOME -s status.log -n rac1Step 2: Unmount ACFS if available Step 3: Run pre-root Script Step 4: Rollback Grid Infrastructure Patch Step 5: Pre-script for Database Home Step 6: Rollback Oracle Database Patch Step 7: Post Script for Database Component Step 8: Run post root script Step 9: Start Oracle services Step 10: Start all database instances running from the Oracle home. Step 11: Run catbundle Script For each database instance running out of the ORACLE_HOME, connect to the database using SQL*Plus as SYSDBA and run the rollback script as follows:cd $ORACLE_HOME/rdbms/admin $ opatch lsinventoryAll other instances can be started and accessed as usual while you are executing the deinstallation steps. You have successfully rolled back the Patch. |
