Complete below list of tasks:
DBA may need to add a new node or existing node (which got deleted from cluster) into Clusterware.
List of tasks for New Node:
- Install same version of OS
- Setup environment variable
- Install all required package including Oracleasm
- Configure Networking (IP and others)
- Check Time Synchronization
- Change Recommended Kernel Parameters for Oracle
- Create Group and User (make sure these are same with existing Nodes)
- Setup SSH configuration for grid user or (user which was used for GRID installation) between new node and other nodes.
If it is existing node, make sure cleanup properly. You may go through: how to delete a node from cluster.
Compare setting between new nodes with existing any node by running below command and fix these if any discrimination is there.
[grid@ocmnode2 bin]$ pwd /u01/app/12.1.0/grid/bin [grid@ocmnode2 bin]$ ./cluvfy comp peer -n ocmnode1 -refnode ocmnode2 -r 12.1
Click on below line to view details:
./cluvfy comp peer -n ocmnode1 -refnode ocmnode2 -r 12.1
[grid@ocmnode2 bin]$ ./cluvfy comp peer -n ocmnode1 -refnode ocmnode2 -r 12.1
Verifying peer compatibility
Checking peer compatibility...
Compatibility check: Physical memory [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 3.7411GB (3922860.0KB) 3.8613GB (4048820.0KB) mismatched
Physical memory <null>
Compatibility check: Available memory [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 3.4843GB (3653536.0KB) 2.316GB (2428544.0KB) matched
Available memory <null>
Compatibility check: Swap space [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 3.9375GB (4128760.0KB) 3.9375GB (4128764.0KB) mismatched
Swap space <null>
Compatibility check: Free disk space for "/usr" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 23.6152GB (2.4762368E7KB) 19.5928GB (2.0544512E7KB) matched
Free disk space <null>
Compatibility check: Free disk space for "/var" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 23.6152GB (2.4762368E7KB) 19.5928GB (2.0544512E7KB) matched
Free disk space <null>
Compatibility check: Free disk space for "/etc" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 23.6152GB (2.4762368E7KB) 19.5928GB (2.0544512E7KB) matched
Free disk space <null>
Compatibility check: Free disk space for "/u01/app/12.1.0/grid" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 23.6152GB (2.4762368E7KB) 19.5928GB (2.0544512E7KB) matched
Free disk space <null>
Compatibility check: Free disk space for "/sbin" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 23.6152GB (2.4762368E7KB) 19.5928GB (2.0544512E7KB) matched
Free disk space <null>
Compatibility check: Free disk space for "/tmp" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 23.6152GB (2.4762368E7KB) 19.5928GB (2.0544512E7KB) matched
Free disk space <null>
Compatibility check: User existence for "grid" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 grid(54321) grid(54321) matched
User existence for "grid" check passed
Compatibility check: Group existence for "oinstall" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 oinstall(54321) oinstall(54321) matched
Group existence for "oinstall" check passed
Compatibility check: Group existence for "dba" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 dba(54322) dba(54322) matched
Group existence for "dba" check passed
Compatibility check: Group membership for "grid" in "oinstall (Primary)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 yes yes matched
Group membership for "grid" in "oinstall (Primary)" check passed
Compatibility check: Group membership for "grid" in "dba" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 no no matched
Group membership for "grid" in "dba" check passed
Compatibility check: Run level [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 5 5 matched
Run level check passed
Compatibility check: System architecture [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 x86_64 x86_64 matched
System architecture check passed
Compatibility check: Kernel version [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 2.6.32-431.el6.x86_64 3.8.13-16.2.1.el6uek.x86_64 mismatched
Kernel version check failed
Compatibility check: Kernel param "semmsl" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 250 250 matched
Kernel param "semmsl" check passed
Compatibility check: Kernel param "semmns" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 32000 32000 matched
Kernel param "semmns" check passed
Compatibility check: Kernel param "semopm" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 100 100 matched
Kernel param "semopm" check passed
Compatibility check: Kernel param "semmni" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 128 128 matched
Kernel param "semmni" check passed
Compatibility check: Kernel param "shmmax" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 68719476736 68719476736 matched
Kernel param "shmmax" check passed
Compatibility check: Kernel param "shmmni" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 4096 4096 matched
Kernel param "shmmni" check passed
Compatibility check: Kernel param "shmall" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 0 0 matched
Kernel param "shmall" check passed
Compatibility check: Kernel param "file-max" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 6815744 6815744 matched
Kernel param "file-max" check passed
Compatibility check: Kernel param "ip_local_port_range" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 9000 65500 9000 65500 matched
Kernel param "ip_local_port_range" check passed
Compatibility check: Kernel param "rmem_default" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 262144 262144 matched
Kernel param "rmem_default" check passed
Compatibility check: Kernel param "rmem_max" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 4194304 4194304 matched
Kernel param "rmem_max" check passed
Compatibility check: Kernel param "wmem_default" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 262144 262144 matched
Kernel param "wmem_default" check passed
Compatibility check: Kernel param "wmem_max" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 1048576 1048576 matched
Kernel param "wmem_max" check passed
Compatibility check: Kernel param "aio-max-nr" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 1048576 1048576 matched
Kernel param "aio-max-nr" check passed
Compatibility check: Kernel param "panic_on_oops" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 1 1 matched
Kernel param "panic_on_oops" check passed
Compatibility check: Package existence for "binutils" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2-5.36.el6 matched
Package existence for "binutils" check passed
Compatibility check: Package existence for "compat-libcap1" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 compat-libcap1-1.10-1 compat-libcap1-1.10-1 matched
Package existence for "compat-libcap1" check passed
Compatibility check: Package existence for "compat-libstdc++-33 (x86_64)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 compat-libstdc++-33-3.2.3-69.el6 (x86_64) compat-libstdc++-33-3.2.3-69.el6 (x86_64) matched
Package existence for "compat-libstdc++-33 (x86_64)" check passed
Compatibility check: Package existence for "libgcc (x86_64)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 libgcc-4.4.7-23.0.1.el6 (x86_64) libgcc-4.4.7-23.0.1.el6 (x86_64) matched
Package existence for "libgcc (x86_64)" check passed
Compatibility check: Package existence for "libstdc++ (x86_64)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 libstdc++-4.4.7-23.0.1.el6 (x86_64) libstdc++-4.4.7-23.0.1.el6 (x86_64) matched
Package existence for "libstdc++ (x86_64)" check passed
Compatibility check: Package existence for "libstdc++-devel (x86_64)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 libstdc++-devel-4.4.7-23.0.1.el6 (x86_64) libstdc++-devel-4.4.7-23.0.1.el6 (x86_64) matched
Package existence for "libstdc++-devel (x86_64)" check passed
Compatibility check: Package existence for "sysstat" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 sysstat-9.0.4-22.el6 sysstat-9.0.4-22.el6 matched
Package existence for "sysstat" check passed
Compatibility check: Package existence for "gcc" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 gcc-4.4.7-23.0.1.el6 gcc-4.4.7-23.0.1.el6 matched
Package existence for "gcc" check passed
Compatibility check: Package existence for "gcc-c++" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 gcc-c++-4.4.7-23.0.1.el6 gcc-c++-4.4.7-23.0.1.el6 matched
Package existence for "gcc-c++" check passed
Compatibility check: Package existence for "ksh" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 missing missing matched
Package existence for "ksh" check passed
Compatibility check: Package existence for "make" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 make-3.81-20.el6 make-3.81-20.el6 matched
Package existence for "make" check passed
Compatibility check: Package existence for "glibc (x86_64)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 glibc-2.12-1.132.el6 (x86_64) glibc-2.12-1.132.el6 (x86_64) matched
Package existence for "glibc (x86_64)" check passed
Compatibility check: Package existence for "glibc-devel (x86_64)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 glibc-devel-2.12-1.132.el6 (x86_64) glibc-devel-2.12-1.132.el6 (x86_64) matched
Package existence for "glibc-devel (x86_64)" check passed
Compatibility check: Package existence for "libaio (x86_64)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 libaio-0.3.107-10.el6 (x86_64) libaio-0.3.107-10.el6 (x86_64) matched
Package existence for "libaio (x86_64)" check passed
Compatibility check: Package existence for "libaio-devel (x86_64)" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 libaio-devel-0.3.107-10.el6 (x86_64) libaio-devel-0.3.107-10.el6 (x86_64) matched
Package existence for "libaio-devel (x86_64)" check passed
Compatibility check: Package existence for "nfs-utils" [reference node: ocmnode2]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
ocmnode1 nfs-utils-1.2.3-39.el6 nfs-utils-1.2.3-39.el6 matched
Package existence for "nfs-utils" check passed
Verification of peer compatibility was successful.
Checks passed for the following node(s):
ocmnode1
Run below command to find the prerequisites status on New node or existing problematic node.
[grid@ocmnode2 bin]$ pwd /u01/app/12.1.0/grid/bin [grid@ocmnode2 bin]$ ./cluvfy stage -pre crsinst -n ocmnode1
Click on below line to view details:
[grid@ocmnode2 bin]$ ./cluvfy stage -pre crsinst -n ocmnode1
[grid@ocmnode2 bin]$ ./cluvfy stage -pre crsinst -n ocmnode1
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "ocmnode2"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Check: Node connectivity using interfaces on subnet "192.168.56.0"
Node connectivity passed for subnet "192.168.56.0" with node(s) ocmnode1
TCP connectivity check passed for subnet "192.168.56.0"
Check: Node connectivity using interfaces on subnet "192.168.10.0"
Node connectivity passed for subnet "192.168.10.0" with node(s) ocmnode1
TCP connectivity check passed for subnet "192.168.10.0"
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check failed
Check failed on nodes:
ocmnode1
Available memory check passed
Swap space check passed
Free disk space check passed for "ocmnode1:/usr,ocmnode1:/var,ocmnode1:/etc,ocmnode1:/u01/app/12.1.0/grid,ocmnode1:/sbin,ocmnode1:/tmp"
Check for multiple users with UID value 54321 passed
User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" failed
Check failed on nodes:
ocmnode1
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check failed
Check failed on nodes:
ocmnode1
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Kernel parameter check passed for "panic_on_oops"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check failed for "ksh"
Check failed on nodes:
ocmnode1
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "nfs-utils"
Check for multiple users with UID value 0 passed
Current group ID check passed
Starting check for consistency of primary group of root user
Check for consistency of root user's primary group passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP configuration file "/etc/ntp.conf" existence check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
ocmnode1
Clock synchronization check using Network Time Protocol(NTP) failed
Core file name pattern consistency check passed.
User "grid" is not part of "root" group. Check passed
Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes
WARNING:
PRVF-5640 : Both 'search' and 'domain' entries are present in file "/etc/resolv.conf" on the following nodes: ocmnode1
Check for integrity of file "/etc/resolv.conf" passed
Time zone consistency check passed
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
Checking daemon "avahi-daemon" is not configured and running
Daemon not configured check passed for process "avahi-daemon"
Daemon not running check passed for process "avahi-daemon"
Starting check for Reverse path filter setting ...
Check for Reverse path filter setting passed
Starting check for Network interface bonding status of private interconnect network interfaces ...
Check for Network interface bonding status of private interconnect network interfaces passed
Starting check for /dev/shm mounted as temporary file system ...
Check for /dev/shm mounted as temporary file system passed
Starting check for /boot mount ...
Check for /boot mount passed
Starting check for zeroconf check ...
Check for zeroconf check passed
Pre-check for cluster services setup was unsuccessful on all the nodes.
Below command will generate fix-up script for prerequisites.
[grid@ocmnode2 bin]$ pwd /u01/app/12.1.0/grid/bin [grid@ocmnode2 bin]$ ./cluvfy stage -pre crsinst -n ocmnode1 -fixup
[grid@ocmnode2 bin]$ ./cluvfy stage -pre crsinst -n ocmnode1 -fixup
[grid@ocmnode2 .ssh]$ cd /u01/app/12.1.0/grid/bin
[grid@ocmnode2 bin]$ ./cluvfy stage -pre crsinst -n ocmnode1 -fixup
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "ocmnode2"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Check: Node connectivity using interfaces on subnet "192.168.10.0"
Node connectivity passed for subnet "192.168.10.0" with node(s) ocmnode1
TCP connectivity check passed for subnet "192.168.10.0"
Check: Node connectivity using interfaces on subnet "192.168.56.0"
Node connectivity passed for subnet "192.168.56.0" with node(s) ocmnode1
TCP connectivity check passed for subnet "192.168.56.0"
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.10.0" for multicast communication with multicast group "224.0. 0.251"...
Check of subnet "192.168.10.0" for multicast communication with multicast group "224.0. 0.251" passed.
Check of multicast communication passed.
Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check failed
Check failed on nodes:
ocmnode1
Available memory check passed
Swap space check passed
Free disk space check passed for "ocmnode1:/usr,ocmnode1:/var,ocmnode1:/etc,ocmnode1:/u01/app/12.1.0/grid,ocmnode1:/sbin,ocmnode1:/tmp"
Check for multiple users with UID value 54321 passed
User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" failed
Check failed on nodes:
ocmnode1
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
PRVG-1206 : Check cannot be performed for configured value of kernel parameter "aio-max-nr" on node "ocmnode1"
Kernel parameter check passed for "aio-max-nr"
PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "ocmnode1"
Kernel parameter check passed for "panic_on_oops"
Package existence check passed for "binutils"
Package existence check failed for "compat-libcap1"
Check failed on nodes:
ocmnode1
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check failed for "ksh"
Check failed on nodes:
ocmnode1
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check failed for "libaio-devel(x86_64)"
Check failed on nodes:
ocmnode1
Package existence check passed for "nfs-utils"
Check for multiple users with UID value 0 passed
Current group ID check passed
Starting check for consistency of primary group of root user
Check for consistency of root user's primary group passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP configuration file "/etc/ntp.conf" existence check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
ocmnode1
Clock synchronization check using Network Time Protocol(NTP) failed
Core file name pattern consistency check passed.
User "grid" is not part of "root" group. Check passed
Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes
WARNING:
PRVF-5640 : Both 'search' and 'domain' entries are present in file "/etc/resolv.conf" on the following nodes: ocmnode1
Check for integrity of file "/etc/resolv.conf" passed
Time zone consistency check passed
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
Checking daemon "avahi-daemon" is not configured and running
Daemon not configured check passed for process "avahi-daemon"
Daemon not running check passed for process "avahi-daemon"
Starting check for Reverse path filter setting ...
Check for Reverse path filter setting passed
Starting check for Network interface bonding status of private interconnect network interfaces ...
Check for Network interface bonding status of private interconnect network interfaces passed
Starting check for /dev/shm mounted as temporary file system ...
ERROR:
PRVE-0426 : The size of in-memory file system mounted as /dev/shm is "1002" megabytes which is less than the required size of "2048" megabytes on node ""
Check for /dev/shm mounted as temporary file system failed
Starting check for /boot mount ...
Check for /boot mount passed
Starting check for zeroconf check ...
ERROR:
PRVE-10077 : NOZEROCONF parameter was not specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "ocmnode1.localdomain"
Check for zeroconf check failed
Pre-check for cluster services setup was unsuccessful on all the nodes.
******************************************************************************************
Following is the list of fixable prerequisites selected to fix in this session
******************************************************************************************
-------------- --------------- ----------------
Check failed. Failed on nodes Reboot required?
-------------- --------------- ----------------
zeroconf check ocmnode1 no
Group Membership: dba ocmnode1 no
OS Kernel Parameter: ocmnode1 no
aio-max-nr
OS Kernel Parameter: ocmnode1 no
panic_on_oops
Execute "/tmp/CVU_12.1.0.2.0_grid/runfixup.sh" as root user on nodes "ocmnode1" to perform the fix up operations manually
Press ENTER key to continue after execution of "/tmp/CVU_12.1.0.2.0_grid/runfixup.sh" has completed on nodes "ocmnode1"
Fix: zeroconf check
"zeroconf check" was successfully fixed on all the applicable nodes
Fix: Group Membership: dba
"Group Membership: dba" was successfully fixed on all the applicable nodes
Fix: OS Kernel Parameter: aio-max-nr
"OS Kernel Parameter: aio-max-nr" was successfully fixed on all the applicable nodes
Fix: OS Kernel Parameter: panic_on_oops
"OS Kernel Parameter: panic_on_oops" was successfully fixed on all the applicable nodes
Pre-check for node addition:
[grid@ocmnode2 bin]$ pwd /u01/app/12.1.0/grid/bin [grid@ocmnode2 bin]$ ./cluvfy stage -pre nodeadd -n ocmnode1
[grid@ocmnode2 bin]$ ./cluvfy stage -pre nodeadd -n ocmnode1
[grid@ocmnode2 bin]$ ./cluvfy stage -pre nodeadd -n ocmnode1
Performing pre-checks for node addition
Checking node reachability...
Node reachability check passed from node "ocmnode2"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking CRS integrity...
CRS integrity check passed
Clusterware version consistency passed.
Checking shared resources...
Checking CRS home location...
Path "/u01/app/12.1.0/grid" either already exists or can be successfully created on nodes: "ocmnode1"
Shared resources check for node addition passed
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Check: Node connectivity using interfaces on subnet "192.168.10.0"
Node connectivity passed for subnet "192.168.10.0" with node(s) ocmnode2,ocmnode1
TCP connectivity check passed for subnet "192.168.10.0"
Check: Node connectivity using interfaces on subnet "192.168.56.0"
Node connectivity passed for subnet "192.168.56.0" with node(s) ocmnode1,ocmnode2
TCP connectivity check passed for subnet "192.168.56.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed.
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Total memory check failed
Check failed on nodes:
ocmnode2,ocmnode1
Available memory check passed
Swap space check passed
Free disk space check passed for "ocmnode2:/usr,ocmnode2:/var,ocmnode2:/etc,ocmnode2:/u01/app/12.1.0/grid,ocmnode2:/sbin,ocmnode2:/tmp"
Free disk space check passed for "ocmnode1:/usr,ocmnode1:/var,ocmnode1:/etc,ocmnode1:/u01/app/12.1.0/grid,ocmnode1:/sbin,ocmnode1:/tmp"
Check for multiple users with UID value 54321 passed
User existence check passed for "grid"
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Kernel parameter check passed for "panic_on_oops"
Package existence check passed for "binutils"
Package existence check failed for "compat-libcap1"
Check failed on nodes:
ocmnode1
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check failed for "ksh"
Check failed on nodes:
ocmnode2,ocmnode1
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check failed for "libaio-devel(x86_64)"
Check failed on nodes:
ocmnode1
Package existence check passed for "nfs-utils"
Check for multiple users with UID value 0 passed
Current group ID check passed
Starting check for consistency of primary group of root user
Check for consistency of root user's primary group passed
Group existence check passed for "asmadmin"
Membership check for user "grid" in group "asmadmin" passed
Group existence check passed for "asmoper"
Membership check for user "grid" in group "asmoper" passed
Group existence check passed for "asmdba"
Membership check for user "grid" in group "asmdba" passed
Group existence check passed for "oinstall"
Membership check for user "grid" in group "oinstall" passed
Check for multiple users with UID value 0 passed
User existence check passed for "root"
Check for multiple users with UID value 54321 passed
User existence check passed for "grid"
Checking ASMLib configuration.
Check for ASMLib configuration passed.
Checking OCR integrity...
Disks "+OCR/ocmnode-cluster/OCRFILE/registry.255.1047853917" are managed by ASM.
OCR integrity check passed
Checking Oracle Cluster Voting Disk configuration...
Disks "ORCL:OCRDISK1,ORCL:OCRDISK2,ORCL:OCRDISK3" are managed by ASM.
Oracle Cluster Voting Disk configuration check passed
Time zone consistency check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP configuration file "/etc/ntp.conf" existence check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
ocmnode2,ocmnode1
Clock synchronization check using Network Time Protocol(NTP) failed
User "grid" is not part of "root" group. Check passed
Checking integrity of file "/etc/resolv.conf" across nodes
WARNING:
PRVF-5640 : Both 'search' and 'domain' entries are present in file "/etc/resolv.conf" on the following nodes: ocmnode2,ocmnode1
Check for integrity of file "/etc/resolv.conf" passed
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
Pre-check for node addition was unsuccessful on all the nodes.
Run below command from workable node.
[grid@ocmnode2 addnode]$ ./addnode.sh -silent CLUSTER_NEW_NODES={ocmnode1} CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip} -ignoreSysPrereqs -ignorePrereq
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 15883 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4029 MB Passed
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 8% Done.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2020-08-08_10-48-12PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 14% Done.
Copying files to node in progress.
Copying files to node successful.
.................................................. 73% Done.
Saving cluster inventory in progress.
.................................................. 80% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/12.1.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 88% Done.
As a root user, execute the following script(s):
1. /u01/app/oraInventory/orainstRoot.sh
2. /u01/app/12.1.0/grid/root.sh
Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[ocmnode1]
Execute /u01/app/12.1.0/grid/root.sh on the following nodes:
[ocmnode1]
The scripts can be executed in parallel on all the nodes.
..........
Update Inventory in progress.
.................................................. 100% Done.
Update Inventory successful.
Successfully Setup Software.
[root@ocmnode1 ~]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete.
[root@ocmnode1 ~]# /u01/app/12.1.0/grid/root.sh Check /u01/app/12.1.0/grid/install/root_ocmnode1.localdomain_2020-08-10_01-35-07.log for t he output of root script
[grid@ocmnode2 ~]$ cat root_ocmnode1.localdomain_2020-08-10_01-35-07.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2020/08/10 01:35:08 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2020/08/10 01:35:08 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/08/10 01:35:09 CLSRSC-363: User ignored prerequisites during installation
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.crsd' on 'ocmnode1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'ocmnode1'
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'ocmnode1'
CRS-2677: Stop of 'ora.OCR.dg' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'ocmnode1'
CRS-2677: Stop of 'ora.DATA.dg' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'ocmnode1'
CRS-2677: Stop of 'ora.asm' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'ocmnode2'
CRS-2676: Start of 'ora.scan1.vip' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'ocmnode1'
CRS-2677: Stop of 'ora.ons' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'ocmnode1'
CRS-2677: Stop of 'ora.net1.network' on 'ocmnode1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'ocmnode1' has completed
CRS-2677: Stop of 'ora.crsd' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'ocmnode1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.storage' on 'ocmnode1'
CRS-2677: Stop of 'ora.storage' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'ocmnode1'
CRS-2677: Stop of 'ora.ctssd' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.asm' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'ocmnode1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ocmnode1'
CRS-2677: Stop of 'ora.cssd' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'ocmnode1'
CRS-2677: Stop of 'ora.crf' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'ocmnode1'
CRS-2677: Stop of 'ora.gipcd' on 'ocmnode1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ocmnode1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.evmd' on 'ocmnode1'
CRS-2676: Start of 'ora.evmd' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ocmnode1'
CRS-2676: Start of 'ora.gpnpd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'ocmnode1'
CRS-2676: Start of 'ora.gipcd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ocmnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'ocmnode1'
CRS-2676: Start of 'ora.diskmon' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.ctssd' on 'ocmnode1'
CRS-2676: Start of 'ora.ctssd' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ocmnode1'
CRS-2676: Start of 'ora.asm' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'ocmnode1'
CRS-2676: Start of 'ora.storage' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'ocmnode1'
CRS-2676: Start of 'ora.crf' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ocmnode1'
CRS-2676: Start of 'ora.crsd' on 'ocmnode1' succeeded
CRS-6017: Processing resource auto-start for servers: ocmnode1
CRS-2672: Attempting to start 'ora.net1.network' on 'ocmnode1'
CRS-2676: Start of 'ora.net1.network' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'ocmnode1'
CRS-2676: Start of 'ora.ons' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'ocmnode2'
CRS-2677: Stop of 'ora.scan1.vip' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'ocmnode1'
CRS-2676: Start of 'ora.scan1.vip' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode1' succeeded
CRS-6016: Resource auto-start has completed for server ocmnode1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2020/08/10 01:37:21 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1
2020/08/10 01:37:38 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Node added successfully with Clusterware.
Validation:
[grid@ocmnode2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.FRA.dg
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.OCR.dg
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.OCR_VOTE.dg
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.asm
ONLINE ONLINE ocmnode1 Started,STABLE
ONLINE ONLINE ocmnode2 Started,STABLE
ora.net1.network
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
ora.ons
ONLINE ONLINE ocmnode1 STABLE
ONLINE ONLINE ocmnode2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ocmnode1 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE ocmnode2 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE ocmnode2 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE ocmnode2 169.254.43.148 192.1
68.10.11,STABLE
ora.cvu
1 ONLINE ONLINE ocmnode2 STABLE
ora.mgmtdb
1 ONLINE ONLINE ocmnode2 Open,STABLE
ora.oc4j
1 ONLINE ONLINE ocmnode2 STABLE
ora.ocmnode1.vip
1 ONLINE ONLINE ocmnode1 STABLE
ora.ocmnode2.vip
1 ONLINE ONLINE ocmnode2 STABLE
ora.scan1.vip
1 ONLINE ONLINE ocmnode1 STABLE
ora.scan2.vip
1 ONLINE ONLINE ocmnode2 STABLE
ora.scan3.vip
1 ONLINE ONLINE ocmnode2 STABLE
--------------------------------------------------------------------------------
Adding Node to Oracle Home:
[oracle@ocmnode2 addnode]$ pwd
/u01/app/oracle/product/12.1.0/dbhome_1/addnode
[oracle@ocmnode2 addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={ocmnode1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip}"
[oracle@ocmnode2 addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={ocmnode1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 12822 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4031 MB Passed
[FATAL] [INS-30160] Installer has detected that the nodes [ocmnode1] specified for addnode operation have uncleaned inventory.
ACTION: Ensure that the inventory location /u01/app/oraInventory is cleaned before performing addnode procedure.
[oracle@ocmnode2 addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={ocmnode1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 12822 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4031 MB Passed
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2020-08-10_03-17-13PM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2020-08-10_03-17-13PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 8% Done.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2020-08-10_03-17-13PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 14% Done.
Copying files to node in progress.
Copying files to node successful.
.................................................. 73% Done.
Saving cluster inventory in progress.
SEVERE:Remote 'AttachHome' failed on nodes: 'ocmnode1'. Refer to '/u01/app/oraInventory/logs/addNodeActions2020-08-10_03-17-13PM.log' for details.
It is recommended that the following command needs to be manually run on the failed nodes:
/u01/app/oracle/product/12.1.0/dbhome_1/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 ORACLE_HOME_NAME=OraDB12Home1 CLUSTER_NODES=ocmnode2,ocmnode1 "INVENTORY_LOCATION=/u01/app/oraInventory" -invPtrLoc "/etc/oraInst.loc" LOCAL_NODE=<node on which command is to be run>.
Please refer 'AttachHome' logs under central inventory of remote nodes where failure occurred for more details.
SEVERE:Remote 'UpdateNodeList' failed on nodes: 'ocmnode1'. Refer to '/u01/app/oraInventory/logs/addNodeActions2020-08-10_03-17-13PM.log' for details.
It is recommended that the following command needs to be manually run on the failed nodes:
/u01/app/oracle/product/12.1.0/dbhome_1/oui/bin/runInstaller -updateNodeList -noClusterEnabled ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 CLUSTER_NODES=ocmnode2,ocmnode1 CRS=false "INVENTORY_LOCATION=/u01/app/oraInventory" -invPtrLoc "/etc/oraInst.loc" LOCAL_NODE=<node on which command is to be run>.
Please refer 'UpdateNodeList' logs under central inventory of remote nodes where failure occurred for more details.
.................................................. 80% Done.
Saving cluster inventory successful.
WARNING:OUI-10234:Failed to copy the root script, /u01/app/oraInventory/orainstRoot.sh to the cluster nodes ocmnode1.[Error in copying file '/u01/app/oraInventory/orainstRoot.sh' present inside directory '/' on nodes 'ocmnode1'. [PRKC-1080 : Failed to transfer file "/u01/app/oraInventory/orainstRoot.sh" to any of the given nodes "ocmnode1 ".
Error on node ocmnode1:/bin/tar: ./u01/app/oraInventory/orainstRoot.sh: Cannot open: No such file or directory
/bin/tar: Exiting with failure status due to previous errors]]
Please copy them manually to these nodes and execute the script.
The Cluster Node Addition of /u01/app/oracle/product/12.1.0/dbhome_1 was unsuccessful.
Please check '/tmp/silentInstall.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 88% Done.
As a root user, execute the following script(s):
1. /u01/app/oraInventory/orainstRoot.sh
2. /u01/app/oracle/product/12.1.0/dbhome_1/root.sh
Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[ocmnode1]
Execute /u01/app/oracle/product/12.1.0/dbhome_1/root.sh on the following nodes:
[ocmnode1]
..........
Update Inventory in progress.
SEVERE:Remote 'UpdateNodeList' failed on nodes: 'ocmnode1'. Refer to '/u01/app/oraInventory/logs/UpdateNodeList2020-08-10_03-17-13PM.log' for details.
It is recommended that the following command needs to be manually run on the failed nodes:
/u01/app/oracle/product/12.1.0/dbhome_1/oui/bin/runInstaller -updateNodeList -setCustomNodelist -noClusterEnabled ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 CLUSTER_NODES=ocmnode2,ocmnode1 "NODES_TO_SET={ocmnode2,ocmnode1}" CRS=false "INVENTORY_LOCATION=/u01/app/oraInventory" -invPtrLoc "/etc/oraInst.loc" LOCAL_NODE=<node on which command is to be run>.
Please refer 'UpdateNodeList' logs under central inventory of remote nodes where failure occurred for more details.
[WARNING] [INS-10016] Installer failed to update the cluster related details, for this Oracle home, in the inventory on all/some of the nodes
ACTION: You may chose to retry the operation, without continuing further. Alternatively you can refer to information given below and manually execute the mentioned commands on the failed nodes now or later to update the inventory.
*MORE DETAILS*
Execute the following command on node(s) [ocmnode1]:
/u01/app/oracle/product/12.1.0/dbhome_1/oui/bin/runInstaller -jreLoc /u01/app/oracle/product/12.1.0/dbhome_1/jdk/jre -paramFile /u01/app/oracle/product/12.1.0/dbhome_1/oui/clusterparam.ini -silent -ignoreSysPrereqs -updateNodeList -bigCluster ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 CLUSTER_NODES=<Local Node> "NODES_TO_SET={ocmnode2,ocmnode1}" -invPtrLoc "/u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc" -local
.................................................. 100% Done.
Update Inventory successful.
Successfully Setup Software.
Execute root.sh script on new node.
[root@ocmnode1 app]# /u01/app/oracle/product/12.1.0/dbhome_1/root.sh
Check /u01/app/oracle/product/12.1.0/dbhome_1/install/root_ocmnode1.localdomain_2020-08-10_15-27-42.log for the output of root script
[root@ocmnode1 app]# cat /u01/app/oracle/product/12.1.0/dbhome_1/install/root_ocmnode1.localdomain_2020-08-10_15-27-42.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/12.1.0/dbhome_1
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
During adding a Node, I faced number of issues. I mentioned in below and it may help you.
Issue # 1: ORA-00845: MEMORY_TARGET not supported on this system
[grid@ocmnode2 addnode]$ ./addnode.sh -silent CLUSTER_NEW_NODES={ocmnode1} CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip} -ignoreSysPrereqs -ignorePrereq
Got below error which running root.sh script.
ORA-00845: MEMORY_TARGET not supported on this system . For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/ocmnode1/crs/trace/ohasd_oraagent_grid.trc". CRS-2883: Resource 'ora.asm' failed during Clusterware stack start. CRS-4406: Oracle High Availability Services synchronous start failed. CRS-4000: Command Start failed, or completed with errors. 2020/08/08 22:55:25 CLSRSC-117: Failed to start Oracle Clusterware stack Died at /u01/app/12.1.0/grid/crs/install/crsinstall.pm line 914. The command '/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/rootcrs.pl ' execution failed
Solution: Need to extend /tmp filesystem size.
[root@ocmnode1 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_racprodn1-lv_root 45G 20G 23G 47% / tmpfs 1001M 702M 30% /dev/shm /dev/sda1 477M 55M 397M 13% /boot /dev/sr0 56M 56M 0 100% /media/VBox_GAs_5.2.10
[root@ocmnode1 ~]# cat /etc/fstab # /dev/mapper/vg_racprodn1-lv_root / ext4 defaults 1 1 UUID=31e76520-7f50-4592-9ced-d4f03d6d95cc /boot ext4 defaults 1 2 /dev/mapper/vg_racprodn1-lv_swap swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0
Modify Below line on fstab:
tmpfs /dev/shm tmpfs size=2g 0 0
[root@ocmnode1 ~]# cat /etc/fstab # /dev/mapper/vg_racprodn1-lv_root / ext4 defaults 1 1 UUID=31e76520-7f50-4592-9ced-d4f03d6d95cc /boot ext4 defaults 1 2 /dev/mapper/vg_racprodn1-lv_swap swap swap defaults 0 0 tmpfs /dev/shm tmpfs size=2g 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0
Reboot the system.
Run the root.sh script.
Issue # 2: One or more node names “ocmnode1.localdomain” contain one or more of the following invalid characters “.”
[grid@ocmnode2 addnode]$ . oraenv
ORACLE_SID = [grid] ? +ASM2
The Oracle base has been set to /u01/app/grid
[grid@ocmnode2 addnode]$ pwd
/u01/app/12.1.0/grid/addnode
[grid@ocmnode2 addnode]$ ./addnode.sh -silent CLUSTER_NEW_NODES={ocmnode1.localdomain} CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip.localdomain} -ignoreSysPrereqs -ignorePrereq
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 16183 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4031 MB Passed
[FATAL] [INS-30131] Initial setup required for the execution of installer validations failed.
CAUSE: Failed to access the temporary location.
ACTION: Ensure that the current user has required permissions to access the temporary location.
*ADDITIONAL INFORMATION:*
Exception details
 - PRVG-11322 : One or more node names "ocmnode1.localdomain" contain one or more of the following invalid characters "."
Solution:
Don’t use the domain name for silent mode. Correct syntax will be:
[grid@ocmnode2 addnode]$ ./addnode.sh -silent CLUSTER_NEW_NODES={ocmnode1} CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip} -ignoreSysPrereqs -ignorePrereq
Issue # 3: [FATAL] [INS-40906] Duplicate host name ocmnode1 found in the node information table for Oracle Clusterware install
[grid@ocmnode2 addnode]$ pwd
/u01/app/12.1.0/grid/addnode
[grid@ocmnode2 addnode]$ ls -l addnode.sh
-rwxr-xr-x 1 grid oinstall 3575 Aug 7 22:18 addnode.sh
[grid@ocmnode2 addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={ocmnode1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 15562 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4019 MB Passed
[FATAL] [INS-40906] Duplicate host name ocmnode1 found in the node information table for Oracle Clusterware install.
CAUSE: Duplicate entries have been made in the node information table for Oracle Clusterware installation.
ACTION: Remove duplicate entries in the Oracle Clusterware node information table.
Cause: The problematic node information still exist on other node’s inventory file.
[grid@ocmnode2 addnode]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
...
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0/grid" TYPE="O" IDX="1" CRS="true">
<NODE_LIST>
<NODE NAME="ocmnode1"/>
<NODE NAME="ocmnode2"/>
</NODE_LIST>
</HOME>
<HOME NAME="OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0/dbhome_1" TYPE="O" IDX="2">
<NODE_LIST>
<NODE NAME="ocmnode1"/>
<NODE NAME="ocmnode2"/>
</NODE_LIST>
</HOME>
...Need to update inventory with Good Nodes (currently working) list.
[grid@ocmnode2 bin]$ pwd
/u01/app/12.1.0/grid/oui/bin
[grid@ocmnode2 bin]$ export ORACLE_HOME=/u01/app/12.1.0/grid/
[grid@ocmnode2 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={ocmnode2}" CRS=TRUE
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4019 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
[grid@ocmnode2 addnode]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml | grep NODE
<NODE_LIST>
<NODE NAME="ocmnode2"/>
</NODE_LIST>
<NODE_LIST>
<NODE NAME="ocmnode2"/>
</NODE_LIST>Issue # 4:
[grid@ocmnode2 addnode]$ ./addnode.sh -silent CLUSTER_NEW_NODES={ocmnode1} CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip} -ignoreSysPrereqs -ignorePrereq
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 16182 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4029 MB Passed
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 8% Done.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2020-08-08_10-28-51PM.log
SEVERE:Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.
[FATAL] [INS-10008] Session initialization failed
CAUSE: An unexpected error occured while initializing the session.
ACTION: Contact Oracle Support Services or refer logs
SUMMARY:
[grid@ocmnode2 addnode]$ more /u01/app/oraInventory/logs/addNodeActions2020-08-08_10-28-51PM.log
Status of node 'ocmnode1':
Node is okay
--------------------------------------------------------------------------
INFO: Setting variable 'REMOTE_CLEAN_MACHINES' to 'ocmnode1'. Received the value from a code block.
INFO: /u01/app/12.1.0/grid/oui/bin/../bin/lsnodes: struct size 0
INFO: Vendor clusterware is not detected.
INFO: Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.
SEVERE: Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.
INFO: Alert Handler not registered, using Super class functionality
INFO: Alert Handler not registered, using Super class functionality
INFO: User Selected: Yes/OK
INFO: Shutting down OUISetupDriver.JobExecutorThread
SEVERE: [FATAL] [INS-10008] Session initialization failed
CAUSE: An unexpected error occured while initializing the session.
ACTION: Contact Oracle Support Services or refer logs
SUMMARY:
- .
Refer associated stacktrace #oracle.install.commons.util.exception.DefaultErrorAdvisor:299
INFO: Advice is ABORT
SEVERE: Unconditional Exit
INFO: Adding ExitStatus FAILURE to the exit status set
INFO: Finding the most appropriate exit status for the current application
INFO: Exit Status is -1
INFO: Shutdown Oracle Grid Infrastructure 12c Release 1 Installer
INFO: Unloading Setup Driver
Cause: Check on problematic Node if oracleasm is working properly. Run below command to verify.
$ oracleasm init # If get any error, you may try with re-installation oracle asm packages. $ oracleasm scandisks # If not return any records by below query, you have to fix the oracleasm. $ oracleasm listdisks
Solution: Fix the oracleasm issue and make sure all disks are accessible from new node.
Issue # 5: On Oracle Home.
[FATAL] [INS-30160] Installer has detected that the nodes [ocmnode1] specified for addnode operation have uncleaned inventory
[oracle@ocmnode2 addnode]$ pwd
/u01/app/oracle/product/12.1.0/dbhome_1/addnode
[oracle@ocmnode2 addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={ocmnode1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={ocmnode1-vip}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 12822 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4031 MB Passed
[FATAL] [INS-30160] Installer has detected that the nodes [ocmnode1] specified for addnode operation have uncleaned inventory.
ACTION: Ensure that the inventory location /u01/app/oraInventory is cleaned before performing addnode procedure.
Solution: Remove or Rename oraInventory directory from New Node.
[root@ocmnode1 app]# pwd /u01/app [root@ocmnode1 app]# mv oraInventory oraInventory_bck [root@ocmnode1 app]# ls -lrt total 20 drwxr-xr-x. 3 root oinstall 4096 Aug 7 00:27 12.1.0 drwxrwxr-x 2 grid oinstall 4096 Aug 8 17:40 grid:oinstall drwxrwx--- 5 grid oinstall 4096 Aug 9 20:53 oraInventory_bck drwxrwxr-x. 8 grid oinstall 4096 Aug 10 01:34 grid drwxrwxr-x. 5 oracle oinstall 4096 Aug 10 15:19 oracle
