🖥️ Environment Details : 🗄️Database Version: Oracle Database 26ai (2-Node RAC) 🐧Operating System: RHEL 9 Error: CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00065:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbnode1' CRS-5172: Start of Highly Available IP (HAIP) aborted. CRS-2676: Start of 'ora.cssdmonitor' on 'dbnode1' succeeded CRS-2674: Start of 'ora.cluster_interconnect.haip' on 'dbnode1' failed CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'dbnode1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'dbnode1' succeeded CRS-2883: Resource 'ora.cssd' failed during Clusterware stack start. CRS-4406: Oracle High Availability Services synchronous start failed. CRS-41053: checking Oracle Grid Infrastructure for file permission issues PRVE-10077 : NOZEROCONF parameter was not specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "dbnode1.localdomain" CRS-4000: Command Start failed, or completed with errors. Issue Description: One of the production database servers was hosted on a RHEL 9 server in a VirtualBox environment. Following a scheduled downtime activity performed by the VMware team, Oracle Clusterware (CRS) was unable to start. Cause: During CRS startup, the CRS was unable to start and failed with the following error messages: [root@dbnode1 ~]# /grid_26ai/app/26.0.0/grid/bin/crsctl start crs -wait CRS-4123: Starting Oracle High Availability Services-managed resources CRS-2672: Attempting to start 'ora.mdnsd' on 'dbnode1' CRS-2672: Attempting to start 'ora.evmd' on 'dbnode1' CRS-2676: Start of 'ora.mdnsd' on 'dbnode1' succeeded CRS-2676: Start of 'ora.evmd' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'dbnode1' CRS-2676: Start of 'ora.gpnpd' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'dbnode1' CRS-2676: Start of 'ora.gipcd' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.crf' on 'dbnode1' CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbnode1' CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dbnode1' CRS-2676: Start of 'ora.crf' on 'dbnode1' succeeded CRS-2676: Start of 'ora.cssdmonitor' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'dbnode1' CRS-2672: Attempting to start 'ora.diskmon' on 'dbnode1' CRS-2676: Start of 'ora.diskmon' on 'dbnode1' succeeded CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00065:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbnode1' CRS-5172: Start of Highly Available IP (HAIP) aborted. CRS-2676: Start of 'ora.cssdmonitor' on 'dbnode1' succeeded CRS-2674: Start of 'ora.cluster_interconnect.haip' on 'dbnode1' failed CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'dbnode1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'dbnode1' succeeded CRS-2883: Resource 'ora.cssd' failed during Clusterware stack start. CRS-4406: Oracle High Availability Services synchronous start failed. CRS-41053: checking Oracle Grid Infrastructure for file permission issues PRVE-10077 : NOZEROCONF parameter was not specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "dbnode1.localdomain" CRS-4000: Command Start failed, or completed with errors. Troubleshooting Steps: Step1: Based on the error message, there appears to be an issue with the voting devices. We will verify the availability and permissions of the voting files. [root@dbnode1 ~]# oracleasm listdisks ARCH1 DATA1 DATA2 DATA3 DATA4 DATA5 DATA6 DATA7 DATA8 DATA9 OCR1 OCR2 OCR3 OCR4 OCR5 RECO1 #Check ASM disk permissions. [root@dbnode1 ~]# ls -ltr /dev/oracleasm/disks/* The permissions of the voting disks were correct, and all disks were present on the server. What should we check next? What else could be causing the issue? Step2: Check the CRS alert log file for more details. CRS Alert Log File: 2026-08-21 13:33:47.765 [OHASD(6909)] CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 6909 2026-08-21 13:33:47.808 [OHASD(6909)] CRS-0714: Oracle Clusterware Release 23.26.1.0.0. 2026-08-21 13:33:48.235 [ORAROOTAGENT(7587)] CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 7587 2026-08-21 13:33:48.229 [ORAAGENT(7563)] CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 7563 2026-08-21 13:33:48.259 [CSSDMONITOR(7605)] CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 7605 2026-08-21 13:33:48.260 [CSSDAGENT(7597)] CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 7597 2026-08-21 13:33:48.537 [ORAAGENT(7858)] CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 7858 2026-08-21 13:33:48.617 [EVMD(7945)] CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 7945 2026-08-21 13:33:48.615 [MDNSD(7941)] CRS-8500: Oracle Clusterware MDNSD process is starting with operating system process ID 7941 2026-08-21 13:33:48.729 [GPNPD(8088)] CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 8088 2026-08-21 13:33:48.842 [GPNPD(8088)] CRS-2328: Grid Plug and Play Daemon(GPNPD) started on node dbnode1. 2026-08-21 13:33:48.862 [GIPCD(8171)] CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 8171 2026-08-21 13:33:48.945 [GIPCD(8171)] CRS-7517: The Oracle Grid Interprocess Communication (GIPC) is failing to identify the Instant Failure Detection (IFD). 2026-08-21 13:33:49.078 [GIPCD(8171)] CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred 2026-08-21 13:33:50.978 [OSYSMOND(8676)] CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 8676 2026-08-21 13:33:50.965 [CSSDMONITOR(8670)] CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 8670 2026-08-21 13:33:51.190 [CSSDAGENT(8823)] CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 8823 2026-08-21 13:33:51.421 [OCSSD(8858)] CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 8858 2026-08-21 13:33:51.421 [ONMD(8856)] CRS-8500: Oracle Clusterware ONMD process is starting with operating system process ID 8856 2026-08-21 13:33:51.492 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:34:06.499 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:34:21.502 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:34:36.506 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:34:51.509 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc .......... 2026-08-21 13:38:21.561 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:38:36.564 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:38:46.434 [OHASD(6909)] CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbnode1' 2026-08-21 13:38:48.217 [MDNSD(7941)] CRS-5602: mDNS service stopping by request. 2026-08-21 13:38:51.568 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc ......... 2026-08-21 13:41:06.600 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:41:21.604 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:41:36.608 [ONMD(8856)] CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:41:51.455 [ONMD(8856)] CRS-1656: The CSS daemon is terminating due to a fatal error from thread: Alarm; Details at (:CSSSC00011:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc 2026-08-21 13:41:51.456 [ONMD(8856)] CRS-8506: internal error code, arguments: [(:CSSNM00198:)], [ONMD], [clsscssd_ReadyBussTimeout_CB: G], [], [], [], [], [], [], [], [], [] 2026-08-21T13:41:51.483639+05:30 Errors in file /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/onmd.trc (incident=49): CRS-8506 [(:CSSNM00198:)] [ONMD] [clsscssd_ReadyBussTimeout_CB: GMCD ready for business timedout. Exiting.] [] [] [] [] [] [] [] [] [] Incident details in: /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/incident/incdir_49/onmd_i49.trc 2026-08-21 13:41:51.543 [ONMD(8856)] CRS-8504: Oracle Clusterware ONMD process with operating system process ID 8856 is exiting 2026-08-21 13:43:50.933 [ORAROOTAGENT(7587)] CRS-5818: Aborted command 'start' for resource 'ora.cluster_interconnect.haip' due to a timeout of 600 seconds. Details at (:CRSAGF00113:) {0:5:4} in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ohasd_orarootagent_root.trc. 2026-08-21 13:43:51.375 [CSSDAGENT(8823)] CRS-5818: Aborted command 'start' for resource 'ora.cssd' due to a timeout of 600 seconds. Details at (:CRSAGF00113:) {0:5:4} in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ohasd_cssdagent_root.trc. 2026-08-21 13:43:52.376 [OCSSD(8858)] CRS-1745: Cluster Synchronization Service will not undergo recovery due to it is still not open for business; details at '(:CSSSC00040:)' in '/grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ocssd.trc'. 2026-08-21 13:43:52.387 [OCSSD(8858)] CRS-1656: The CSS daemon is terminating due to a fatal error from thread: AgLsnr; Details at (:CSSSC00036:) in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ocssd.trc 2026-08-21 13:43:52.388 [OCSSD(8858)] CRS-8506: internal error code, arguments: [(:CSSSC00129:)], [CSSD], [clssscagProcAgReq: shutdown abo], [], [], [], [], [], [], [], [], [] 2026-08-21T13:43:52.401698+05:30 Errors in file /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ocssd.trc (incident=57): CRS-8506 [(:CSSSC00129:)] [CSSD] [clssscagProcAgReq: shutdown abort requested by the agent] [] [] [] [] [] [] [] [] [] Incident details in: /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/incident/incdir_57/ocssd_i57.trc 2026-08-21 13:43:52.467 [OCSSD(8858)] CRS-8504: Oracle Clusterware OCSSD process with operating system process ID 8858 is exiting 2026-08-21 13:43:52.622 [ORAROOTAGENT(7587)] CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error: 2026-08-21 13:43:52.622+Start action for HAIP aborted. For details refer to "(:CLSN00107:)" in "/grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ohasd_orarootagent_root.trc". 2026-08-21 13:43:53.936 [OHASD(6909)] CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00221:) {0:5:4} in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ohasd.trc. 2026-08-21 13:43:59.359 [CRSCTL(14909)] CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/crsctl_14909.trc. 2026-08-21 13:44:02.478 [OHASD(6909)] CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00221:) {0:5:4} in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ohasd.trc. 2026-08-21 13:44:03.037 [CSSDAGENT(20634)] CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 20634 2026-08-21 13:44:03.230 [GPNPD(8088)] CRS-2329: Grid Plug and Play Daemon(GPNPD) on node dbnode1 shut down. 2026-08-21 13:44:03.839 [OHASD(6909)] CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbnode1' has completed 2026-08-21 13:44:03.852 [ORAROOTAGENT(7587)] CRS-5822: Agent '/grid_26ai/app/26.0.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:1:10} in /grid_26ai/app/26.0.0/grid_base/diag/crs/dbnode1/crs/trace/ohasd_orarootagent_root.trc. As you can see from the CRS alert log, the message “CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds” indicates that something went wrong with the OCR/voting disks. What could have caused Oracle Clusterware to fail to discover the voting disks even though all the disks were present on the server? Let's check the issue. Step3: Let's check the OS logs for any relevant messages, especially since there was a planned activity at the VMware end during that time. view /var/log/messages Aug 21 14:15:54 dbnode1 systemd[1]: /etc/systemd/system/SplunkForwarder.service:29: Unit uses MemoryLimit=; please use MemoryMax= instead. Support for MemoryLimit= will be removed soon. Aug 21 14:16:05 dbnode1 oracleasm-discover[89440]: ASMLib: CRIT(version): Failed initializing io_uring, exiting: 1 (EPERM) Aug 21 14:17:45 dbnode1 PackageKit[82279]: daemon quit Aug 21 14:17:45 dbnode1 systemd[1]: packagekit.service: Deactivated successfully. Aug 21 14:17:50 dbnode1 sssd_kcm[82628]: Shutting down (status = 0) Aug 21 14:17:50 dbnode1 systemd[1]: sssd-kcm.service: Deactivated successfully. Aug 21 14:20:47 dbnode1 NetworkManager[1836]: <info> [1787302247.5922] device (ens256): Activation: starting connection 'Ethernet connection 1' (a62e214e-8e8e-4ec8-8139-b311816a0469) Aug 21 14:20:47 dbnode1 NetworkManager[1836]: <info> [1787302247.5923] device (ens256): state change: disconnected -> prepare (reason 'none', managed-type: 'full') Aug 21 14:20:47 dbnode1 NetworkManager[1836]: <info> [1787302247.5927] device (ens256): state change: prepare -> config (reason 'none', managed-type: 'full') Aug 21 14:20:47 dbnode1 NetworkManager[1836]: <info> [1787302247.5933] device (ens256): state change: config -> ip-config (reason 'none', managed-type: 'full') Aug 21 14:20:47 dbnode1 NetworkManager[1836]: <info> [1787302247.5946] dhcp4 (ens256): activation: beginning transaction (timeout in 45 seconds) Aug 21 14:21:19 dbnode1 oracleasm-discover[91738]: ASMLib: CRIT(version): Failed initializing io_uring, exiting: 1 (EPERM) Aug 21 14:21:30 dbnode1 systemd[1]: var-lib-containers-storage-overlay.mount: Deactivated successfully. Aug 21 14:21:30 dbnode1 systemd[1]: var-lib-containers-storage-overlay.mount: Deactivated successfully. Aug 21 14:21:32 dbnode1 NetworkManager[1836]: <info> [1787302292.5914] device (ens256): state change: ip-config -> failed (reason 'ip-config-unavailable', managed-type: 'full') Aug 21 14:21:32 dbnode1 NetworkManager[1836]: <warn> [1787302292.5919] device (ens256): Activation: failed for connection 'Ethernet connection 1' Aug 21 14:21:32 dbnode1 NetworkManager[1836]: <info> [1787302292.5921] device (ens256): state change: failed -> disconnected (reason 'none', managed-type: 'full') Aug 21 14:21:32 dbnode1 NetworkManager[1836]: <info> [1787302292.6079] dhcp4 (ens256): canceled DHCP transaction You can see the message "ASMLib: CRIT(version): Failed initializing io_uring, exiting: 1 (EPERM)" This indicates that ASMLib was unable to initialize io_uring, and the operation was denied with EPERM (Operation not permitted). Since ASMLib is involved in ASM disk discovery, this could potentially explain why Oracle was unable to discover the OCR/voting disks. Let's check how io_uring is configured and being used on the server, and determine whether its failure could have impacted ASM disk discovery. What is io_uring ? io_uring is a Linux kernel interface that performs asynchronous I/O introduced in Linux kernel 5.1 that provides a faster and more efficient way for applications to perform disk reads and writes operations. Traditionally, Linux applications have relied on synchronous system calls such as read() and write() to perform I/O operations. Under heavy I/O workloads, frequent transitions between user space and kernel space can add considerable processing overhead and affect performance. Linux also offered an earlier asynchronous I/O framework called Linux AIO, but its capabilities were limited. io_uring was introduced to overcome these limitations and provide a more efficient and flexible asynchronous I/O mechanism in Linux. ASMLib V3 supports for the io_uring API implemented in the Linux kernel. Let's check what is there in ASMLIB V3 support. Starting with ASMLib V3, Oracle added support for the Linux io_uring interface as an alternative to the traditional oracleasm kernel driver. ASMLib automatically detects the available I/O mechanism and uses io_uring when it is supported and enabled by the running kernel. Below changes are added in ASMLib V3: - oracleasm status now performs additional checks to confirm that ASMLib is functioning correctly. - io_uring validation verifies whether io_uring is accessible to the configured ORACLEASM_UID and ORACLEASM_GID. - Prevents accidental relabeling, renaming, or deletion of ASM disks already in use. - oracleasm querydisk -i and oracleasm discover -l provide detailed information about ASM disks. Why does io_uring matter for ASMLib V3 ? ASMLib V3 is significantly different from older ASMLib implementations. On supported Linux environments, it uses io_uring for communication (I/O) rather than depending on the old oracleasm kernel driver model. Therefore, ASMLib needs to confirm that the OS permits the Oracle/ASM operating-system user to create and use io_uring. ASMLib configuration contains UID and GID and it needs io_uring to be available to that UID/GID. Hence, "oracleasm status" command performs an additional validation related to io_uring. ORACLEASM_UID=grid ORACLEASM_GID=asmadmin ┌─────────────────────────┐ │ RHEL 9 Kernel │ └────────────┬────────────┘ ↓ ┌─────────────────────────┐ │ Is io_uring available? │ └────────────┬────────────┘ ↓ ┌─────────────────────────┐ │ Is io_uring permitted? │ └────────────┬────────────┘ ↓ ┌─────────────────────────────────────┐ │ Can ORACLEASM_UID / ORACLEASM_GID │ │ use io_uring? │ └────────────┬────────────────────────┘ ↓ ┌─────────────────────────┐ │ ASMLib V3 │ └────────────┬────────────┘ ↓ ┌─────────────────────────┐ │ ASM Disks │ └─────────────────────────┘ Even when the RHEL kernel supports io_uring, security controls can restrict its use. #To Check kernel settings io_uring: #sysctl kernel.io_uring_disabled OR cat /proc/sys/kernel/io_uring_disabled #sysctl kernel.io_uring_group #For ASMLib: #oracleasm status io_uring parameters ? By default, this feature is disabled, but you can enable it by setting parameter "kernel.io_uring_disabled". 1) kernel.io_uring_disabled=0 - This means all processes can create io_uring. 2) kernel.io_uring_disabled=1 - This means only privileged processes with the CAP_SYS_ADMIN permission can create new io_uring instances. Other processes will get an EPERM error. However, any existing io_uring instances can continue to be used. 3) kernel.io_uring_disabled=2 - This means io_uring creation is disabled for all processes. Any attempt will fail with an EPERM error. However, existing io_uring instances can still be used. This is the default setting. Step4: Let's check the existing io_uring entry located in "/etc/sysctl.conf" or /etc/sysctl.d/io_uring.conf file. [root@dbnode1 sysctl.d]# cat /etc/sysctl.conf | grep -i io_uring_disabled kernel.io_uring_disabled = 2 [root@dbnode1 sysctl.d]# oracleasm status Checking if the oracleasm kernel module is loaded: no (not required with RHEL9) Checking if /dev/oracleasm is mounted: no (not required with RHEL9) Checking which I/O Interface is in use: io_uring (KABI_V3) Checking if ASMLIB can be loaded: yes Checking if io_uring is enabled: yes Checking if io_uring is accessible to the configured DB user: no (Unable to call io_uring_setup() as oracle/asmadmin: Operation not permitted) Checking if io_uring supports integrity passthrough: no Checking if ASM disks have the correct ownership and permissions: yes Checking if ASM I/O filter is set up: yes Here is the issue. kernel.io_uring_disabled is currently set to 2, which means io_uring is completely disabled. This setting may have changed following the VMware activity and should have been enabled before the planned activity. Since io_uring is disabled, the Oracle user is unable to initialize the required io_uring interface and receives an “Operation not permitted” error. As a result, Oracle ASMLib cannot access the required I/O interface, preventing Oracle from discovering the OCR disks. The oracleasm status output also confirms that io_uring is not accessible to the configured Oracle DB user. This appears to be the primary reason for the OCR disk discovery issue. The io_uring kernel setting needs to be corrected to the expected value and verified before proceeding with CRS startup. Let's change the value in configuration file to 0 so that io_uring feature will be enabled. [root@dbnode1 ]# vi /etc/sysctl.conf [root@dbnode1 ]# cat /etc/sysctl.conf | grep -i io_uring_disabled kernel.io_uring_disabled = 0 [root@dbnode1 ]# sysctl -p .... .... kernel.io_uring_disabled = 0 Step5: Let's start the CRS now after setting the io_uring value. #restart the Oracle ASMLIB service on the server [root@dbnode1 ]# systemctl restart oracleasm.service [root@dbnode1 ]# systemctl status oracleasm.service ● oracleasm.service - Oracle ASM Service Loaded: loaded (/usr/lib/systemd/system/oracleasm.service; enabled; preset: disabled) Active: active (exited) since Fri 2026-08-21 14:31:39 IST; 6s ago Process: 99696 ExecStartPre=/usr/bin/udevadm settle -t 120 (code=exited, status=0/SUCCESS) Process: 99698 ExecStart=/usr/localdomainn/oracleasm.init start (code=exited, status=0/SUCCESS) Main PID: 99698 (code=exited, status=0/SUCCESS) CPU: 1.523s Aug 21 14:31:38 dbnode1.localdomain systemd[1]: Starting Oracle ASM Service... Aug 21 14:31:38 dbnode1.localdomain oracleasm.init[99714]: Mounting oracleasm driver filesystem: Not applicable with RHEL9 Aug 21 14:31:39 dbnode1.localdomain oracleasm.init[99954]: Reloading disk partitions: done Aug 21 14:31:39 dbnode1.localdomain oracleasm.init[99954]: Cleaning any stale ASM disks... Aug 21 14:31:39 dbnode1.localdomain oracleasm.init[99954]: Setting up iofilter map for ASM disks: done Aug 21 14:31:39 dbnode1.localdomain oracleasm.init[99971]: Scanning system for ASM disks... Aug 21 14:31:39 dbnode1.localdomain oracleasm.init[100640]: Disk scan successful Aug 21 14:31:39 dbnode1.localdomain systemd[1]: Finished Oracle ASM Service. [root@dbnode1 ]# oracleasm status Checking if the oracleasm kernel module is loaded: no (not required with RHEL9) Checking if /dev/oracleasm is mounted: no (not required with RHEL9) Checking which I/O Interface is in use: io_uring (KABI_V3) Checking if ASMLIB can be loaded: yes Checking if io_uring is enabled: yes Checking if io_uring is accessible to the configured DB user: yes Checking if io_uring supports integrity passthrough: yes Checking if ASM disks have the correct ownership and permissions: yes Checking if ASM I/O filter is set up: yes #Start the CRS on local node by "crsctl start crs -wait" command. [root@dbnode1 ]# /grid_26ai/app/26.0.0/grid/bin/crsctl start crs -wait CRS-4123: Starting Oracle High Availability Services-managed resources CRS-2672: Attempting to start 'ora.evmd' on 'dbnode1' CRS-2672: Attempting to start 'ora.mdnsd' on 'dbnode1' CRS-2676: Start of 'ora.mdnsd' on 'dbnode1' succeeded CRS-2676: Start of 'ora.evmd' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'dbnode1' CRS-2676: Start of 'ora.gpnpd' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'dbnode1' CRS-2676: Start of 'ora.gipcd' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.crf' on 'dbnode1' CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dbnode1' CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbnode1' CRS-2676: Start of 'ora.crf' on 'dbnode1' succeeded CRS-2676: Start of 'ora.cssdmonitor' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'dbnode1' CRS-2672: Attempting to start 'ora.diskmon' on 'dbnode1' CRS-2676: Start of 'ora.diskmon' on 'dbnode1' succeeded CRS-2676: Start of 'ora.cssd' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.diskmon' on 'dbnode1' CRS-2676: Start of 'ora.diskmon' on 'dbnode1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'dbnode1' CRS-2676: Start of 'ora.asm' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.storage' on 'dbnode1' CRS-2676: Start of 'ora.storage' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'dbnode1' CRS-2676: Start of 'ora.crsd' on 'dbnode1' succeeded CRS-6023: Starting Oracle Cluster Ready Services-managed resources CRS-2672: Attempting to start 'ora.dbnode1.vip' on 'dbnode1' CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'dbnode1' CRS-6017: Processing resource auto-start for servers: dbnode1 CRS-2672: Attempting to start 'ora.scan3.vip' on 'dbnode1' CRS-2672: Attempting to start 'ora.scan2.vip' on 'dbnode1' CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbnode1' CRS-2672: Attempting to start 'ora.dbnode2.vip' on 'dbnode1' CRS-2672: Attempting to start 'ora.chad' on 'dbnode1' CRS-2672: Attempting to start 'ora.cvu' on 'dbnode1' CRS-2672: Attempting to start 'ora.ons' on 'dbnode1' CRS-2676: Start of 'ora.cvu' on 'dbnode1' succeeded CRS-2676: Start of 'ora.chad' on 'dbnode1' succeeded CRS-2676: Start of 'ora.dbnode1.vip' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'dbnode1' CRS-2676: Start of 'ora.dbnode2.vip' on 'dbnode1' succeeded CRS-2676: Start of 'ora.ons' on 'dbnode1' succeeded CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'dbnode1' succeeded CRS-2676: Start of 'ora.scan3.vip' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'dbnode1' CRS-2676: Start of 'ora.scan2.vip' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'dbnode1' CRS-2676: Start of 'ora.scan1.vip' on 'dbnode1' succeeded CRS-2676: Start of 'ora.LISTENER.lsnr' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'dbnode1' CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'dbnode1' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'dbnode1' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'dbnode1' succeeded CRS-2679: Attempting to clean 'ora.testdb.db' on 'dbnode1' CRS-2681: Clean of 'ora.testdb.db' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.testdb.db' on 'dbnode1' CRS-2676: Start of 'ora.testdb.db' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.testdb.testdbpdb1.pdb' on 'dbnode1' CRS-2676: Start of 'ora.testdb.testdbpdb1.pdb' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.testdb.testdb_testdbpdb1.svc' on 'dbnode1' CRS-6016: Resource auto-start has completed for server dbnode1 CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources CRS-4123: Oracle High Availability Services has been started. [root@dbnode1 ]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online After correcting the kernel.io_uring_disabled parameter and enabling io_uring, Oracle Clusterware was started successfully. Previously, io_uring was disabled (kernel.io_uring_disabled = 2), which prevented the Oracle user from accessing the required I/O interface and resulted in OCR/ASM disk discovery issues. After setting the parameter to the required value and making io_uring accessible, Oracle was able to discover the required storage and proceed with the Clusterware startup. Step6: Let's apply the similar workaround on DB Node2 as well to start the CRS. [root@dbnode2 ]# vi /etc/sysctl.conf [root@dbnode2 ]# cat /etc/sysctl.conf | grep -i io_uring_disabled kernel.io_uring_disabled = 0 [root@dbnode2 ]# sysctl -p .... .... kernel.io_uring_disabled = 0 #restart the Oracle ASMLIB service on the server [root@dbnode2 ]# systemctl restart oracleasm.service [root@dbnode2 ]# systemctl status oracleasm.service ● oracleasm.service - Oracle ASM Service Loaded: loaded (/usr/lib/systemd/system/oracleasm.service; enabled; preset: disabled) Active: active (exited) since Fri 2026-08-21 14:39:03 IST; 4s ago Process: 120113 ExecStartPre=/usr/bin/udevadm settle -t 120 (code=exited, status=0/SUCCESS) Process: 120118 ExecStart=/usr/localdomainn/oracleasm.init start (code=exited, status=0/SUCCESS) Main PID: 120118 (code=exited, status=0/SUCCESS) CPU: 1.623s Aug 21 14:39:01 dbnode2.localdomain systemd[1]: Starting Oracle ASM Service... Aug 21 14:39:01 dbnode2.localdomain oracleasm.init[120134]: Mounting oracleasm driver filesystem: Not applicable with RHEL9 Aug 21 14:39:02 dbnode2.localdomain oracleasm.init[120363]: Reloading disk partitions: done Aug 21 14:39:02 dbnode2.localdomain oracleasm.init[120363]: Cleaning any stale ASM disks... Aug 21 14:39:02 dbnode2.localdomain oracleasm.init[120363]: Setting up iofilter map for ASM disks: done Aug 21 14:39:02 dbnode2.localdomain oracleasm.init[120380]: Scanning system for ASM disks... Aug 21 14:39:03 dbnode2.localdomain oracleasm.init[121049]: Disk scan successful Aug 21 14:39:03 dbnode2.localdomain systemd[1]: Finished Oracle ASM Service. [root@dbnode2 ]# oracleasm status Checking if the oracleasm kernel module is loaded: no (not required with RHEL9) Checking if /dev/oracleasm is mounted: no (not required with RHEL9) Checking which I/O Interface is in use: io_uring (KABI_V3) Checking if ASMLIB can be loaded: yes Checking if io_uring is enabled: yes Checking if io_uring is accessible to the configured DB user: yes Checking if io_uring supports integrity passthrough: yes Checking if ASM disks have the correct ownership and permissions: yes Checking if ASM I/O filter is set up: yes [root@dbnode2 ]# /grid_26ai/app/26.0.0/grid/bin/crsctl start crs -wait CRS-4123: Starting Oracle High Availability Services-managed resources CRS-2672: Attempting to start 'ora.mdnsd' on 'dbnode2' CRS-2672: Attempting to start 'ora.evmd' on 'dbnode2' CRS-2676: Start of 'ora.mdnsd' on 'dbnode2' succeeded CRS-2676: Start of 'ora.evmd' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'dbnode2' CRS-2676: Start of 'ora.gpnpd' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'dbnode2' CRS-2676: Start of 'ora.gipcd' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbnode2' CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dbnode2' CRS-2672: Attempting to start 'ora.crf' on 'dbnode2' CRS-2676: Start of 'ora.crf' on 'dbnode2' succeeded CRS-2676: Start of 'ora.cssdmonitor' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'dbnode2' CRS-2672: Attempting to start 'ora.diskmon' on 'dbnode2' CRS-2676: Start of 'ora.diskmon' on 'dbnode2' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dbnode2' succeeded CRS-2676: Start of 'ora.cssd' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.asm' on 'dbnode2' CRS-2672: Attempting to start 'ora.diskmon' on 'dbnode2' CRS-2676: Start of 'ora.diskmon' on 'dbnode2' succeeded CRS-2676: Start of 'ora.asm' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.storage' on 'dbnode2' CRS-2676: Start of 'ora.storage' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'dbnode2' CRS-2676: Start of 'ora.crsd' on 'dbnode2' succeeded CRS-6017: Processing resource auto-start for servers: dbnode2 CRS-2672: Attempting to start 'ora.ons' on 'dbnode2' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dbnode1' CRS-2673: Attempting to stop 'ora.dbnode2.vip' on 'dbnode1' CRS-2672: Attempting to start 'ora.chad' on 'dbnode2' CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dbnode1' succeeded CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dbnode1' CRS-2677: Stop of 'ora.dbnode2.vip' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.dbnode2.vip' on 'dbnode2' CRS-2677: Stop of 'ora.scan3.vip' on 'dbnode1' succeeded CRS-2672: Attempting to start 'ora.scan3.vip' on 'dbnode2' CRS-2676: Start of 'ora.chad' on 'dbnode2' succeeded CRS-2676: Start of 'ora.dbnode2.vip' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'dbnode2' CRS-2676: Start of 'ora.ons' on 'dbnode2' succeeded CRS-2676: Start of 'ora.scan3.vip' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'dbnode2' CRS-2676: Start of 'ora.LISTENER.lsnr' on 'dbnode2' succeeded CRS-33672: Attempting to start resource group 'ora.asmgroup' on server 'dbnode2' CRS-2672: Attempting to start 'ora.asmnet1.asmnetwork' on 'dbnode2' CRS-2676: Start of 'ora.asmnet1.asmnetwork' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'dbnode2' CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'dbnode2' succeeded CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'dbnode2' succeeded CRS-2679: Attempting to clean 'ora.asm' on 'dbnode2' CRS-2681: Clean of 'ora.asm' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.asm' on 'dbnode2' CRS-2676: Start of 'ora.asm' on 'dbnode2' succeeded CRS-33676: Start of resource group 'ora.asmgroup' on server 'dbnode2' succeeded. CRS-2672: Attempting to start 'ora.RECO.dg' on 'dbnode2' CRS-2672: Attempting to start 'ora.OCR.dg' on 'dbnode2' CRS-2672: Attempting to start 'ora.ARCH.dg' on 'dbnode2' CRS-2672: Attempting to start 'ora.DATA.dg' on 'dbnode2' CRS-2676: Start of 'ora.OCR.dg' on 'dbnode2' succeeded CRS-2676: Start of 'ora.DATA.dg' on 'dbnode2' succeeded CRS-2676: Start of 'ora.ARCH.dg' on 'dbnode2' succeeded CRS-2676: Start of 'ora.RECO.dg' on 'dbnode2' succeeded CRS-2679: Attempting to clean 'ora.testdb.db' on 'dbnode2' CRS-2681: Clean of 'ora.testdb.db' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.testdb.db' on 'dbnode2' CRS-2676: Start of 'ora.testdb.db' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.testdb.testdbpdb1.pdb' on 'dbnode2' CRS-2676: Start of 'ora.testdb.testdbpdb1.pdb' on 'dbnode2' succeeded CRS-2672: Attempting to start 'ora.testdb.testdb_testdbpdb1.svc' on 'dbnode2' CRS-2676: Start of 'ora.testdb.testdb_testdbpdb1.svc' on 'dbnode2' succeeded CRS-6016: Resource auto-start has completed for server dbnode2 CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources CRS-4123: Oracle High Availability Services has been started. [root@dbnode2 ~]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ✅ Final Conclusion : The issue was caused by kernel.io_uring_disabled = 2, which completely disabled io_uring. As a result, the Oracle user could not access the required io_uring interface, impacting ASM/OCR disk discovery and preventing Oracle Clusterware from starting normally. As a resolution, the io_uring kernel parameter was corrected to the required value, the configuration was applied, and Oracle ASMLib was able to access the io_uring interface. After correcting the parameter, CRS started successfully. When troubleshooting Oracle Clusterware/ASM disk discovery issues on RHEL 9, verify the io_uring configuration and confirm that it is accessible to the Oracle user. A disabled io_uring setting can prevent ASMLib from accessing the required storage and lead to Clusterware startup failures. 🎉 Enjoy the troubleshooting journey!!! 📝 Stay tuned for a detailed blog post on this case !!! |
Thanks for reading this post ! Please comment if you like this post ! Click on FOLLOW to get next blog updates !
