CRS (Cluster Ready Service) is not starting and getting ‘FATAL: Module oracleasm not found’ error on log files.
[root@ocmnode1 ~]# cat /var/log/messages
Aug 12 01:32:08 ocmnode1 modprobe: FATAL: Module oracleasm not found.
Aug 12 01:32:33 ocmnode1 modprobe: FATAL: Module oracleasm not found.
Aug 12 01:34:32 ocmnode1 modprobe: FATAL: Module oracleasm not found.
After checked in details found that oracleasm package / RPM was not working.
[root@ocmnode1 ~]# oracleasm init Creating /dev/oracleasm mount point: /dev/oracleasm Loading module "oracleasm": failed Unable to load module "oracleasm"
[root@ocmnode1 ~]# oracleasm status Checking if ASM is loaded: no Checking if /dev/oracleasm is mounted: no
[root@ocmnode1 ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: [ OK ]
Loading module "oracleasm": Unable to load module "oracleasm"
[FAILED]Installed again oracleasm RPM:
[root@ocmnode1 ~]# yum install oracleasm
Details – yum install oracleasm
Loaded plugins: refresh-packagekit, security public_ol6_UEKR3_latest | 2.5 kB 00:00 public_ol6_latest | 2.7 kB 00:00 Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package kernel-uek.x86_64 0:3.8.13-118.48.1.el6uek will be installed --> Processing Dependency: kernel-firmware = 3.8.13-118.48.1.el6uek for package: kernel-uek-3.8.13-118.48.1.el6uek.x86_64 --> Processing Dependency: ql8300-firmware >= 8.01.02-1.0.1 for package: kernel-uek-3.8.13-118.48.1.el6uek.x86_64 --> Processing Dependency: ql2500-firmware >= 8.02.00-1.0.1 for package: kernel-uek-3.8.13-118.48.1.el6uek.x86_64 --> Processing Dependency: ql2400-firmware >= 8.02.00-1.0.1 for package: kernel-uek-3.8.13-118.48.1.el6uek.x86_64 --> Processing Dependency: ql2600-firmware >= 8.01.02-1.0.1 for package: kernel-uek-3.8.13-118.48.1.el6uek.x86_64 --> Processing Dependency: ql2700-firmware >= 8.01.02-1.0.1 for package: kernel-uek-3.8.13-118.48.1.el6uek.x86_64 --> Running transaction check ---> Package kernel-uek-firmware.noarch 0:3.8.13-118.48.1.el6uek will be installed ---> Package ql2400-firmware.noarch 0:7.01.00-1.0.1.el6 will be updated ---> Package ql2400-firmware.noarch 0:8.02.00-1.0.1.el6 will be an update ---> Package ql2500-firmware.noarch 0:7.01.00-1.0.1.el6 will be updated ---> Package ql2500-firmware.noarch 0:8.02.00-1.0.1.el6 will be an update ---> Package ql2600-firmware.noarch 0:8.01.02-1.0.1.el6 will be installed ---> Package ql2700-firmware.noarch 0:8.01.02-1.0.1.el6 will be installed ---> Package ql8300-firmware.noarch 0:8.01.02-1.0.1.el6 will be installed --> Processing Conflict: kernel-uek-3.8.13-118.48.1.el6uek.x86_64 conflicts bfa-firmware < 3.2.23.0-2 --> Restarting Dependency Resolution with new changes. --> Running transaction check ---> Package bfa-firmware.noarch 0:3.2.21.1-2.el6 will be updated ---> Package bfa-firmware.noarch 0:3.2.23.0-2.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================= Installing: kernel-uek x86_64 3.8.13-118.48.1.el6uek public_ol6_UEKR3_latest 43 M Updating: bfa-firmware noarch 3.2.23.0-2.el6 public_ol6_latest 3.2 M Installing for dependencies: kernel-uek-firmware noarch 3.8.13-118.48.1.el6uek public_ol6_UEKR3_latest 2.2 M ql2600-firmware noarch 8.01.02-1.0.1.el6 public_ol6_UEKR3_latest 142 k ql2700-firmware noarch 8.01.02-1.0.1.el6 public_ol6_UEKR3_latest 145 k ql8300-firmware noarch 8.01.02-1.0.1.el6 public_ol6_UEKR3_latest 121 k Updating for dependencies: ql2400-firmware noarch 8.02.00-1.0.1.el6 public_ol6_latest 95 k ql2500-firmware noarch 8.02.00-1.0.1.el6 public_ol6_latest 121 k Transaction Summary ============================================================================================================================================================= Install 5 Package(s) Upgrade 3 Package(s) Total download size: 49 M Is this ok [y/N]: y Downloading Packages: (1/8): bfa-firmware-3.2.23.0-2.el6.noarch.rpm | 3.2 MB 00:01 (2/8): kernel-uek-3.8.13-118.48.1.el6uek.x86_64.rpm | 43 MB 00:15 (3/8): kernel-uek-firmware-3.8.13-118.48.1.el6uek.noarch.rpm | 2.2 MB 00:00 (4/8): ql2400-firmware-8.02.00-1.0.1.el6.noarch.rpm | 95 kB 00:00 (5/8): ql2500-firmware-8.02.00-1.0.1.el6.noarch.rpm | 121 kB 00:00 (6/8): ql2600-firmware-8.01.02-1.0.1.el6.noarch.rpm | 142 kB 00:00 (7/8): ql2700-firmware-8.01.02-1.0.1.el6.noarch.rpm | 145 kB 00:00 (8/8): ql8300-firmware-8.01.02-1.0.1.el6.noarch.rpm | 121 kB 00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 2.2 MB/s | 49 MB 00:22 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Warning: RPMDB altered outside of yum. Installing : kernel-uek-firmware-3.8.13-118.48.1.el6uek.noarch 1/11 Updating : ql2400-firmware-8.02.00-1.0.1.el6.noarch 2/11 Installing : ql2600-firmware-8.01.02-1.0.1.el6.noarch 3/11 Updating : ql2500-firmware-8.02.00-1.0.1.el6.noarch 4/11 Installing : ql2700-firmware-8.01.02-1.0.1.el6.noarch 5/11 Installing : ql8300-firmware-8.01.02-1.0.1.el6.noarch 6/11 Installing : kernel-uek-3.8.13-118.48.1.el6uek.x86_64 7/11 Updating : bfa-firmware-3.2.23.0-2.el6.noarch 8/11 Cleanup : bfa-firmware-3.2.21.1-2.el6.noarch 9/11 Cleanup : ql2400-firmware-7.01.00-1.0.1.el6.noarch 10/11 Cleanup : ql2500-firmware-7.01.00-1.0.1.el6.noarch 11/11 Verifying : ql8300-firmware-8.01.02-1.0.1.el6.noarch 1/11 Verifying : ql2700-firmware-8.01.02-1.0.1.el6.noarch 2/11 Verifying : ql2500-firmware-8.02.00-1.0.1.el6.noarch 3/11 Verifying : ql2600-firmware-8.01.02-1.0.1.el6.noarch 4/11 Verifying : ql2400-firmware-8.02.00-1.0.1.el6.noarch 5/11 Verifying : bfa-firmware-3.2.23.0-2.el6.noarch 6/11 Verifying : kernel-uek-firmware-3.8.13-118.48.1.el6uek.noarch 7/11 Verifying : kernel-uek-3.8.13-118.48.1.el6uek.x86_64 8/11 Verifying : ql2400-firmware-7.01.00-1.0.1.el6.noarch 9/11 Verifying : bfa-firmware-3.2.21.1-2.el6.noarch 10/11 Verifying : ql2500-firmware-7.01.00-1.0.1.el6.noarch 11/11 Installed: kernel-uek.x86_64 0:3.8.13-118.48.1.el6uek Dependency Installed: kernel-uek-firmware.noarch 0:3.8.13-118.48.1.el6uek ql2600-firmware.noarch 0:8.01.02-1.0.1.el6 ql2700-firmware.noarch 0:8.01.02-1.0.1.el6 ql8300-firmware.noarch 0:8.01.02-1.0.1.el6 Updated: bfa-firmware.noarch 0:3.2.23.0-2.el6 Dependency Updated: ql2400-firmware.noarch 0:8.02.00-1.0.1.el6 ql2500-firmware.noarch 0:8.02.00-1.0.1.el6 Complete!
[root@ocmnode1 ~]# oracleasm status Checking if ASM is loaded: no Checking if /dev/oracleasm is mounted: no
Reboot Server and Check:
oracleasm rpm looks good now.
[root@ocmnode1 ~]# oracleasm status Checking if ASM is loaded: yes Checking if /dev/oracleasm is mounted: yes
[root@ocmnode1 ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: doneShowing all ASM Disk.
[root@ocmnode1 ~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... [root@ocmnode1 ~]# oracleasm listdisks DATADISK01 DATADISK02 DATADISK03 DATADISK04 OCRDISK1 OCRDISK2 OCRDISK3 VOTINGDISK01 VOTINGDISK02 VOTINGDISK03
Good to start CRS process on the problematic node.
[root@ocmnode1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
[root@ocmnode1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[grid@ocmnode1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.FRA.dg
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.OCR.dg
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.OCR_VOTE.dg
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.asm
ONLINE ONLINE ocmnode1 Started,STABLE
ONLINE ONLINE ocmnode2 Started,STABLE
ora.net1.network
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.ons
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ocmnode2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE ocmnode1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE ocmnode1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE ocmnode1 169.254.84.152 192.1
68.10.10,STABLE
ora.cvu
1 ONLINE ONLINE ocmnode1 STABLE
ora.mgmtdb
1 ONLINE ONLINE ocmnode1 Open,STABLE
ora.oc4j
1 ONLINE ONLINE ocmnode1 STABLE
ora.ocmnode1.vip
1 ONLINE ONLINE ocmnode1 STABLE
ora.ocmnode2.vip
1 ONLINE ONLINE ocmnode2 STABLE
ora.orcl.db
1 ONLINE ONLINE ocmnode1 Open,STABLE
2 ONLINE ONLINE ocmnode2 Open,STABLE
ora.scan1.vip
1 ONLINE ONLINE ocmnode2 STABLE
ora.scan2.vip
1 ONLINE ONLINE ocmnode1 STABLE
ora.scan3.vip
1 ONLINE ONLINE ocmnode1 STABLE
--------------------------------------------------------------------------------
For more details, you may check this MOS Doc – Oracle ASM RPM issue
