Oracle Database 12c RAC on Oracle Linux 6.5

Oracle RAC (Real Application Cluster) is used for complex application to increase database performance, high availability and scalability.

Below high level steps will perform to configure / install Oracle RAC:

  • Configure 2 Virtual Machine (VM) for 2 Nodes RAC
  • Install Oracle Linux 6.5
  • Install GI (Grid Infrastructure)
  • Configure ASM
  • Install Oracle Database Binary
  • Create Database on RAC

Configure 2 Virtual Machine (VM) for 2 Nodes RAC

Install Oracle Linux 6.5

Install GI (Grid Infrastructure)

Oracle GI and Database Installation Prerequisites:

Create Group and User – Oracle for DB (For RAC – recommended to use GRID user for GI)

groupadd -g 54321 oinstall
groupadd -g 54322 dba
groupadd -g 54323 oper
groupadd -g 54324 backupdba
groupadd -g 54325 dgdba
groupadd -g 54326 asmdba
groupadd -g 54327 asmoper
groupadd -g 54328 asmadmin
/usr/sbin/useradd -u 54321 -g oinstall -G asmadmin,asmdba,asmoper grid
/usr/sbin/useradd -u 1000 -g oinstall -G dba,oper,asmdba,asmadmin,asmoper oracle

To create home directory for GI and Oracle DB, enter the following commands:

#From root user
mkdir -p  /u01/app/12.1.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/

# To create the Oracle Base directory, enter the following commands:
mkdir -p /u01/app/oracle
mkdir /u01/app/oracle/cfgtoollogs 
chown -R oracle:oinstall /u01/app/oracle/cfgtoollogs
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle

# To create the Oracle Inventory directory, enter the following commands:
mkdir -p /u01/app/oraInventory
chown -R grid:oinstall /u01/app/oraInventory
chmod -R 775 /u01/app/oraInventory

#To create the Oracle RDBMS Home directory, enter the following commands:
mkdir -p /u01/app/oracle/product/12.1.0/dbhome_1
chown -R oracle:oinstall /u01/app/oracle/product/12.1.0/dbhome_1
chmod -R 775 /u01/app/oracle/product/12.1.0/dbhome_1

Network Setting:

  • eth0 -> will be public IP
  • eth1 -> will be private IP.

IP Setting on Ethernet:

eth0:
Static IP: 192.168.56.10
Network: 255.255.255.0
eth1:
Static IP: 192.168.10.10
Network: 255.255.254.0

/etc/hosts file should be same for both nodes.

[root@racdr1 software]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1              localhost.localdomain localhost
#::1             localhost6.localdomain6 localhost6

#PROD
# Public IP
192.168.56.10   ocmnode1.localdomain    ocmnode1
192.168.56.11   ocmnode2.localdomain    ocmnode2
#Private IP
192.168.10.10   ocmnode1-priv.localdomain    ocmnode1-priv
192.168.10.11   ocmnode2-priv.localdomain    ocmnode2-priv
#VIP
192.168.56.12   ocmnode1-vip.localdomain    ocmnode1-vip
192.168.56.13   ocmnode2-vip.localdomain    ocmnode2-vip
#SCAN IP
192.168.56.14   ocmnode-scan.localdomain    ocmnode-scan
192.168.56.15   ocmnode-scan.localdomain    ocmnode-scan
192.168.56.16   ocmnode-scan.localdomain    ocmnode-scan

Ping on both nodes:

[root@ocmnode2 ~]# ping ocmnode1
[root@ocmnode1 ~]# ping ocmnode2
[root@ocmnode1 ~]# ping ocmnode-vip
[root@ocmnode2 ~]# ping ocmnode1-vip
[root@ocmnode1 ~]# ping ocmnode-scan
[root@ocmnode2 ~]# ping ocmnode-scan

Set Environment Variable for Oracle user on both Nodes:

$ vi /home/oracle/.bash_profile

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=ocmnode1; export ORACLE_HOSTNAME
ORACLE_UNQNAME=ORCL; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
GRID_HOME=/u01/app/12.1.0/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/12.1.0/dbhome_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_SID=ORCL1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'

Note::: For Node 2 : All will be same except – ORACLE_SID=ORCL2; export ORACLE_SID

$ vim /home/oracle/grid_env
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

Note::: For Node 2 : All will be same except – ORACLE_SID=+ASM2; export ORACLE_SID

$ vim /home/oracle/db_env

ORACLE_SID=ORCL1; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

Note::: For Node 2 : All will be same except – ORACLE_SID=ORCL2; export ORACLE_SID

OS Kernel Parameter Setting for Oracle:

Add following line to /etc/sysctl.conf

# Oracle
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

Verify Kernel Parameter:

$ /sbin/sysctl -p

Add the following lines to the /etc/security/limits.conf file:

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

Below mentioned packages need to install. You can install package either using yum (if you have installed) otherwise manually.

Package Installation by using yum repository:

yum install binutils-2.15.92.0.2
yum install compat-libstdc++-33-3.2.3
yum install    elfutils-libelf-0.97
yum install    elfutils-libelf-devel-0.97
yum install    expat-1.95.7
yum install    gcc-3.4.6
yum install    gcc-c++-3.4.6
yum install    glibc-2.3.4-2.41
yum install    glibc-common-2.3.4
yum install    glibc-devel-2.3.4
yum install    glibc-headers-2.3.4
yum install    libaio-0.3.105
yum install    libaio-devel-0.3.105
yum install    libgcc-3.4.6
yum install    libstdc++-3.4.6
yum install    libstdc++-devel 3.4.6
yum install    compat-libcap1-1.10*
yum install    libaio-devel-0.3*
yum install ksh
yum install    make-3.80
yum install    pdksh-5.2.14
yum install    sysstat-5.0.5
yum install    unixODBC-2.2.11
yum install    unixODBC-devel-2.2.11
yum install    compat-libstdc++-33-3.2.3
yum install    glibc-2.3.4-2.41
yum install    libaio-0.3.105
yum install    libaio-devel-0.3.105
yum install    libgcc-3.4.6
yum install    libstdc++-3.4.6
yum install    unixODBC-2.2.11

Install package by using rmp command:

ISO Disk Mount:

[root@ocmnode1~]# mkdir -p /mnt/disk
[root@ocmnode1 ~]# mount -o loop /u01/software/V41362-01.iso /mnt/disk

Use below command if yum repository doesn’t setup.

$cd /mnt/disk/Server
$ rpm -Uvh binutils-2.*
rpm -Uvh compat-libstdc++-33*
rpm -Uvh elfutils-libelf-0.*
rpm -Uvh elfutils-libelf-devel-*
rpm -Uvh gcc-4.*
rpm -Uvh gcc-c++-4.*
rpm -Uvh glibc-2.*
rpm -Uvh glibc-common-2.*
rpm -Uvh glibc-devel-2.*
rpm -Uvh glibc-headers-2.*
rpm -Uvh ksh-2*
rpm -Uvh libaio-0.*
rpm -Uvh libaio-devel-0.*
rpm -Uvh libgcc-4.*
rpm -Uvh libstdc++-4.*
rpm -Uvh libstdc++-devel-4.*
rpm -Uvh make-3.*
rpm -Uvh sysstat-7.*
rpm -Uvh unixODBC-2.*
rpm -Uvh unixODBC-devel-2.*

The following command can be run on the system to list the currently installed packages:

Command to verify installed package list:
[root@racdr1 Server]# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils \
compat-libstdc++-33 \
elfutils-libelf \
elfutils-libelf-devel \
gcc \
gcc-c++ \
glibc \
glibc-common \
glibc-devel \
glibc-headers \
ksh \
libaio \
libaio-devel \
libgcc \
libstdc++ \
libstdc++-devel \
make \
sysstat \
unixODBC \
unixODBC-devel
OUTPUT: (Show all list and find which packages are not installed. 
binutils-2.17.50.0.6-20.el5 (x86_64)
compat-libstdc++-33-3.2.3-61 (x86_64)
compat-libstdc++-33-3.2.3-61 (i386)
elfutils-libelf-0.137-3.el5 (x86_64)
package elfutils-libelf-devel is not installed
gcc-4.1.2-52.el5 (x86_64)
package gcc-c++ is not installed
glibc-2.5-81 (x86_64)
glibc-2.5-81 (i686)
glibc-common-2.5-81 (x86_64)
glibc-devel-2.5-81 (x86_64)
glibc-headers-2.5-81 (x86_64)
ksh-20100621-5.el5 (x86_64)
libaio-0.3.106-5 (x86_64)
libaio-0.3.106-5 (i386)
libaio-devel-0.3.106-5 (x86_64)
libaio-devel-0.3.106-5 (i386)
libgcc-4.1.2-52.el5 (x86_64)
libgcc-4.1.2-52.el5 (i386)
libstdc++-4.1.2-52.el5 (x86_64)
libstdc++-4.1.2-52.el5 (i386)
libstdc++-devel-4.1.2-52.el5 (x86_64)
libstdc++-devel-4.1.2-52.el5 (i386)
make-3.81-3.el5 (x86_64)
sysstat-7.0.2-11.el5 (x86_64)
package unixODBC is not installed
package unixODBC-devel is not installed

Install Oracle ASM related package:

[root@ocmnode1 software]# rpm -Uvh oracleasmlib-2.0.12-1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:oracleasmlib           ########################################### [100%]
[root@ocmnode1 software]# rpm -Uvh oracleasm-support-2.1.8-1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
        package oracleasm-support-2.1.8-1.el6.x86_64 is already installed

Configure Disk For ASM on RAC

Create necessary shareable disks for ASM. How to create shareable disk on VM

Create Partition on ROW Disk:

[root@ocmnode1 ~]# cd /dev/
[root@ocmnode1 dev]# ls sd*
[root@ocmnode1 dev]# ls sd*
[root@ocmnode1 ~]# ls  /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sdb  /dev/sdc  /dev/sdd  /dev/sde  /dev/sdf  /dev/sdg  /dev/sdh

To verify which ROW disks are already mapping with filesystem: use ($ mount or $df -h) command.

[root@ocmnode1 ~]# mount
/dev/mapper/vg_racprodn1-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
oracleasmfs on /dev/oracleasm type oracleasmfs (rw)

[root@ocmnode1 ~]# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/vg_racprodn1-lv_root   45G   11G   32G  26% /
tmpfs                            1002M   72K 1002M   1% /dev/shm
/dev/sda1                         477M   55M  397M  13% /boot

User fdisk command to format a row disk. Follow below steps for each shareable disks (sdb to sdh).

[root@ocmnode dev]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-522, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-522, default 522):
Using default value 522

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

After partitioning all disk it will show like below:

[root@ocmnode1 ~]# ls -l /dev/sd*
brw-rw---- 1 root disk 8,   0 Aug  7 13:11 /dev/sda
brw-rw---- 1 root disk 8,   1 Aug  7 13:11 /dev/sda1
brw-rw---- 1 root disk 8,   2 Aug  7 13:11 /dev/sda2
brw-rw---- 1 root disk 8,  16 Aug  7 13:28 /dev/sdb
brw-rw---- 1 root disk 8,  17 Aug  7 13:28 /dev/sdb1
brw-rw---- 1 root disk 8,  32 Aug  7 13:28 /dev/sdc
brw-rw---- 1 root disk 8,  33 Aug  7 13:28 /dev/sdc1
brw-rw---- 1 root disk 8,  48 Aug  7 13:29 /dev/sdd
brw-rw---- 1 root disk 8,  49 Aug  7 13:29 /dev/sdd1
brw-rw---- 1 root disk 8,  64 Aug  7 13:29 /dev/sde
brw-rw---- 1 root disk 8,  65 Aug  7 13:29 /dev/sde1
brw-rw---- 1 root disk 8,  80 Aug  7 13:29 /dev/sdf
brw-rw---- 1 root disk 8,  81 Aug  7 13:29 /dev/sdf1
brw-rw---- 1 root disk 8,  96 Aug  7 13:29 /dev/sdg
brw-rw---- 1 root disk 8,  97 Aug  7 13:29 /dev/sdg1
brw-rw---- 1 root disk 8, 112 Aug  7 13:29 /dev/sdh
brw-rw---- 1 root disk 8, 113 Aug  7 13:29 /dev/sdh1

Create ASM Disk:

oracleasm help:

[root@ocmnode1 ~]# /usr/sbin/oracleasm -h
Usage: oracleasm [--exec-path=<exec_path>] <command> [ <args> ]
       oracleasm --exec-path
       oracleasm -h
       oracleasm -V

The basic oracleasm commands are:
    configure        Configure the Oracle Linux ASMLib driver
    init             Load and initialize the ASMLib driver
    exit             Stop the ASMLib driver
    scandisks        Scan the system for Oracle ASMLib disks
    status           Display the status of the Oracle ASMLib driver
    listdisks        List known Oracle ASMLib disks
    querydisk        Determine if a disk belongs to Oracle ASMlib
    createdisk       Allocate a device for Oracle ASMLib use
    deletedisk       Return a device to the operating system
    renamedisk       Change the label of an Oracle ASMlib disk
    update-driver    Download the latest ASMLib driver
[root@ocmnode1 software]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
[root@ocmnode1 software]# /usr/sbin/oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

Run below command for all disks [sdb1 – sdh1]

Instantiating disk: done
[root@ocmnode1 ~]# oracleasm createdisk OCRDISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@ocmnode1 ~]# oracleasm createdisk DATADISK01 /dev/sde1
Writing disk header: done
Instantiating disk: done

If need to delete any ASM disks, use below command:

Delete existing Disk:
[root@ocmnode1 dev]# oracleasm deletedisk OCRDISK1 
[root@ocmnode1 dev]# oracleasm deletedisk OCRDISK2

Verify Disks:

[root@ocmnode1 ~]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@ocmnode1 ~]# /usr/sbin/oracleasm listdisks
DATADISK01
DATADISK02
DATADISK03
DATADISK04
OCRDISK1
OCRDISK2
OCRDISK3

This VM is almost ready. We can use this VM for Node 1 and Clone this VM for Node 2.

Note:::Once we have Node 2, We will check some prerequisites.

On Node 2:

Verify Disks:

[root@ocmnode1 dev]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@ocmnode1 dev]# /usr/sbin/oracleasm listdisks
DATADISK01
DATADISK02
DATADISK03
DATADISK04
OCRDISK1
OCRDISK2
OCRDISK3

Step 1: unzip GI binary file on one node

# From Oracle USER. If you have GRID user for GI, then run below command from GRID
[grid@ocmnode1 software]$ pwd
/u01/software
[grid@ocmnode1 software]$ ls -lrt V4609*
-rw-r--r-- 1 grid oinstall 1747043545 Aug  5 23:48 V46096-01_1of2.zip
-rw-r--r-- 1 grid oinstall  646972897 Aug  5 23:48 V46096-01_2of2.zip

[grid@ocmnode1 software]$ unzip V46096-01_1of2.zip
[grid@ocmnode1 software]$ unzip V46096-01_2of2.zip

Step 2: run below command (runcluvfy.sh) for prerequisite check before final installation process start.

[oracle@racdr1 grid]$ pwd
/u01/software/grid
[oracle@racdr1 grid]$ ls
install  response  rpm  runcluvfy.sh  runInstaller  sshsetup  stage  welcome.html
[oracle@racdr1 grid]$ ./runcluvfy.sh stage -pre crsinst -n racdr1,racdr2 -verbose
Prerequisites Check (runcluvfy.sh) Log:
[grid@ocmnode1 grid]$ pwd
/u01/software/grid
[grid@ocmnode1 grid]$ ls -l
total 44
drwxr-xr-x  4 grid oinstall 4096 Aug  7 17:53 install
drwxrwxr-x  2 grid oinstall 4096 Jul  7  2014 response
drwxr-xr-x  2 grid oinstall 4096 Jul  7  2014 rpm
-rwxr-xr-x  1 grid oinstall 5085 Dec 20  2013 runcluvfy.sh
-rwxr-xr-x  1 grid oinstall 8534 Jul  7  2014 runInstaller
drwxrwxr-x  2 grid oinstall 4096 Jul  7  2014 sshsetup
drwxr-xr-x 14 grid oinstall 4096 Jul  7  2014 stage
-rwxr-xr-x  1 grid oinstall  500 Feb  6  2013 welcome.html

[grid@ocmnode1 grid]$ ./runcluvfy.sh stage -pre crsinst -n ocmnode1,ocmnode2 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "ocmnode1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  ocmnode1                              yes
  ocmnode2                              yes
Result: Node reachability check passed from node "ocmnode1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  ocmnode2                              passed
  ocmnode1                              passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  ocmnode1                              passed
  ocmnode2                              passed

Verification of the hosts config file successful


Interface information for node "ocmnode1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.56.10   192.168.56.0    0.0.0.0         192.168.1.254   08:00:27:41:B9:8C 1500
 eth1   192.168.10.10   192.168.10.0    0.0.0.0         192.168.1.254   08:00:27:B9:5F:D0 1500
 eth2   192.168.1.220   192.168.1.0     0.0.0.0         192.168.1.254   08:00:27:0B:AC:FB 1500
 virbr0 192.168.122.1   192.168.122.0   0.0.0.0         192.168.1.254   52:54:00:01:C9:C6 1500


Interface information for node "ocmnode2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.56.11   192.168.56.0    0.0.0.0         192.168.1.254   08:00:27:41:B9:8C 1500
 eth1   192.168.10.11   192.168.10.0    0.0.0.0         192.168.1.254   08:00:27:B9:5F:D0 1500
 eth2   192.168.1.220   192.168.1.0     0.0.0.0         192.168.1.254   08:00:27:0B:AC:FB 1500
 virbr0 192.168.122.1   192.168.122.0   0.0.0.0         192.168.1.254   52:54:00:01:C9:C6 1500
 eth2   2600:1700:8930:4760:a00:27ff:fe0b:acfb 2600:1700:8930:4760:: UNKNOWN         fe80::9662:69ff:fe44:5cb0 08:00:27:0B:AC:FB 1500


Check: Node connectivity of subnet "192.168.56.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode1[192.168.56.10]         ocmnode2[192.168.56.11]         yes
Result: Node connectivity passed for subnet "192.168.56.0" with node(s) ocmnode1,ocmnode2


Check: TCP connectivity of subnet "192.168.56.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode1 : 192.168.56.10        ocmnode1 : 192.168.56.10        passed
  ocmnode2 : 192.168.56.11        ocmnode1 : 192.168.56.10        passed
  ocmnode1 : 192.168.56.10        ocmnode2 : 192.168.56.11        passed
  ocmnode2 : 192.168.56.11        ocmnode2 : 192.168.56.11        passed
Result: TCP connectivity check passed for subnet "192.168.56.0"


Check: Node connectivity of subnet "192.168.10.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode1[192.168.10.10]         ocmnode2[192.168.10.11]         yes
Result: Node connectivity passed for subnet "192.168.10.0" with node(s) ocmnode1,ocmnode2


Check: TCP connectivity of subnet "192.168.10.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode1 : 192.168.10.10        ocmnode1 : 192.168.10.10        passed
  ocmnode2 : 192.168.10.11        ocmnode1 : 192.168.10.10        passed
  ocmnode1 : 192.168.10.10        ocmnode2 : 192.168.10.11        passed
  ocmnode2 : 192.168.10.11        ocmnode2 : 192.168.10.11        passed
Result: TCP connectivity check passed for subnet "192.168.10.0"


Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode1[192.168.1.220]         ocmnode2[192.168.1.220]         yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) ocmnode1,ocmnode2


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode1 : 192.168.1.220        ocmnode1 : 192.168.1.220        failed

ERROR:
PRVG-11850 : The system call "connect" failed with error "111" while executing exectask on node "ocmnode1"
Connection refused
  ocmnode2 : 192.168.1.220        ocmnode1 : 192.168.1.220        passed
  ocmnode1 : 192.168.1.220        ocmnode2 : 192.168.1.220        failed

ERROR:
PRVG-11850 : The system call "connect" failed with error "111" while executing exectask on node "ocmnode1"
Connection refused
  ocmnode2 : 192.168.1.220        ocmnode2 : 192.168.1.220        passed
Result: TCP connectivity check failed for subnet "192.168.1.0"


Check: Node connectivity of subnet "192.168.122.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode1[192.168.122.1]         ocmnode2[192.168.122.1]         yes
Result: Node connectivity passed for subnet "192.168.122.0" with node(s) ocmnode1,ocmnode2


Check: TCP connectivity of subnet "192.168.122.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode1 : 192.168.122.1        ocmnode1 : 192.168.122.1        failed

ERROR:
PRVG-11850 : The system call "connect" failed with error "111" while executing exectask on node "ocmnode1"
Connection refused
  ocmnode2 : 192.168.122.1        ocmnode1 : 192.168.122.1        passed
  ocmnode1 : 192.168.122.1        ocmnode2 : 192.168.122.1        failed

ERROR:
PRVG-11850 : The system call "connect" failed with error "111" while executing exectask on node "ocmnode1"
Connection refused
  ocmnode2 : 192.168.122.1        ocmnode2 : 192.168.122.1        passed
Result: TCP connectivity check failed for subnet "192.168.122.0"


Check: Node connectivity of subnet "2600:1700:8930:4760::"
Result: Node connectivity passed for subnet "2600:1700:8930:4760::" with node(s) ocmnode2


Check: TCP connectivity of subnet "2600:1700:8930:4760::"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  ocmnode2 : 2600:1700:8930:4760:a00:27ff:fe0b:acfb  ocmnode2 : 2600:1700:8930:4760:a00:27ff:fe0b:acfb  passed
Result: TCP connectivity check passed for subnet "2600:1700:8930:4760::"


Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
ocmnode1 eth2:192.168.1.220
ocmnode2 eth2:192.168.1.220

Interfaces found on subnet "192.168.56.0" that are likely candidates for a private interconnect are:
ocmnode1 eth0:192.168.56.10
ocmnode2 eth0:192.168.56.11

Interfaces found on subnet "192.168.10.0" that are likely candidates for a private interconnect are:
ocmnode1 eth1:192.168.10.10
ocmnode2 eth1:192.168.10.11
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed for subnet "192.168.122.0".
Subnet mask consistency check passed for subnet "2600:1700:8930:4760::".
Subnet mask consistency check passed.

ERROR:
PRVG-1172 : The IP address "192.168.122.1" is on multiple interfaces "virbr0" on nodes "ocmnode1,ocmnode2"

ERROR:
PRVG-1172 : The IP address "192.168.1.220" is on multiple interfaces "eth2" on nodes "ocmnode1,ocmnode2"

Result: Node connectivity check failed

Checking multicast communication...

Checking subnet "192.168.56.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.56.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  ocmnode1                              passed
  ocmnode2                              passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      3.8613GB (4048820.0KB)    4GB (4194304.0KB)         failed
  ocmnode1      3.8613GB (4048820.0KB)    4GB (4194304.0KB)         failed
Result: Total memory check failed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      3.7128GB (3893200.0KB)    50MB (51200.0KB)          passed
  ocmnode1      3.576GB (3749756.0KB)     50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      3.9375GB (4128764.0KB)    3.8613GB (4048820.0KB)    passed
  ocmnode1      3.9375GB (4128764.0KB)    3.8613GB (4048820.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "ocmnode2:/usr,ocmnode2:/var,ocmnode2:/etc,ocmnode2:/sbin,ocmnode2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              ocmnode2      /             33.2168GB     1.0635GB      passed
  /var              ocmnode2      /             33.2168GB     1.0635GB      passed
  /etc              ocmnode2      /             33.2168GB     1.0635GB      passed
  /sbin             ocmnode2      /             33.2168GB     1.0635GB      passed
  /tmp              ocmnode2      /             33.2168GB     1.0635GB      passed
Result: Free disk space check passed for "ocmnode2:/usr,ocmnode2:/var,ocmnode2:/etc,ocmnode2:/sbin,ocmnode2:/tmp"

Check: Free disk space for "ocmnode1:/usr,ocmnode1:/var,ocmnode1:/etc,ocmnode1:/sbin,ocmnode1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              ocmnode1      /             30.2419GB     1.0635GB      passed
  /var              ocmnode1      /             30.2419GB     1.0635GB      passed
  /etc              ocmnode1      /             30.2419GB     1.0635GB      passed
  /sbin             ocmnode1      /             30.2419GB     1.0635GB      passed
  /tmp              ocmnode1      /             30.2419GB     1.0635GB      passed
Result: Free disk space check passed for "ocmnode1:/usr,ocmnode1:/var,ocmnode1:/etc,ocmnode1:/sbin,ocmnode1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  ocmnode2      passed                    exists(54321)
  ocmnode1      passed                    exists(54321)

Checking for multiple users with UID value 54321
Result: Check for multiple users with UID value 54321 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  ocmnode2      passed                    exists
  ocmnode1      passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  ocmnode2      passed                    exists
  ocmnode1      passed                    exists
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode2          yes           yes           yes           yes           passed
  ocmnode1          yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  ocmnode2          yes           yes           no            failed
  ocmnode1          yes           yes           no            failed
Result: Membership check for user "grid" in group "dba" failed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      5                         3,5                       passed
  ocmnode1      5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  ocmnode2          hard          65536         65536         passed
  ocmnode1          hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  ocmnode2          soft          1024          1024          passed
  ocmnode1          soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  ocmnode2          hard          16384         16384         passed
  ocmnode1          hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  ocmnode2          soft          2047          2047          passed
  ocmnode1          soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      x86_64                    x86_64                    passed
  ocmnode1      x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      3.8.13-16.2.1.el6uek.x86_64  2.6.39                    passed
  ocmnode1      3.8.13-16.2.1.el6uek.x86_64  2.6.39                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          250           250           250           passed
  ocmnode2          250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          32000         32000         32000         passed
  ocmnode2          32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          100           100           100           passed
  ocmnode2          100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          128           128           128           passed
  ocmnode2          128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          68719476736   68719476736   2072995840    passed
  ocmnode2          68719476736   68719476736   2072995840    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          4096          4096          4096          passed
  ocmnode2          4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          4294967296    4294967296    404882        passed
  ocmnode2          4294967296    4294967296    404882        passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          6815744       6815744       6815744       passed
  ocmnode2          6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  ocmnode2          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          262144        262144        262144        passed
  ocmnode2          262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          4194304       4194304       4194304       passed
  ocmnode2          4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          262144        262144        262144        passed
  ocmnode2          262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          1048576       1048576       1048576       passed
  ocmnode2          1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          1048576       unknown       1048576       failed (ignorable)  Configured value incorrect.
  ocmnode2          1048576       unknown       1048576       failed (ignorable)  Configured value incorrect.

PRVG-1206 : Check cannot be performed for configured value of kernel parameter "aio-max-nr" on node "ocmnode1"
PRVG-1206 : Check cannot be performed for configured value of kernel parameter "aio-max-nr" on node "ocmnode2"

Result: Kernel parameter check passed for "aio-max-nr"

Check: Kernel parameter for "panic_on_oops"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  ocmnode1          1             unknown       1             failed (ignorable)  Configured value incorrect.
  ocmnode2          1             unknown       1             failed (ignorable)  Configured value incorrect.

PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "ocmnode1"
PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "ocmnode2"

Result: Kernel parameter check passed for "panic_on_oops"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  ocmnode1      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      missing                   compat-libcap1-1.10       failed
  ocmnode1      missing                   compat-libcap1-1.10       failed
Result: Package existence check failed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  ocmnode1      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      libgcc(x86_64)-4.4.7-23.0.1.el6  libgcc(x86_64)-4.4.4      passed
  ocmnode1      libgcc(x86_64)-4.4.7-23.0.1.el6  libgcc(x86_64)-4.4.4      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      libstdc++(x86_64)-4.4.7-23.0.1.el6  libstdc++(x86_64)-4.4.4   passed
  ocmnode1      libstdc++(x86_64)-4.4.7-23.0.1.el6  libstdc++(x86_64)-4.4.4   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      libstdc++-devel(x86_64)-4.4.7-23.0.1.el6  libstdc++-devel(x86_64)-4.4.4  passed
  ocmnode1      libstdc++-devel(x86_64)-4.4.7-23.0.1.el6  libstdc++-devel(x86_64)-4.4.4  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      sysstat-9.0.4-22.el6      sysstat-9.0.4             passed
  ocmnode1      sysstat-9.0.4-22.el6      sysstat-9.0.4             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      gcc-4.4.7-23.0.1.el6      gcc-4.4.4                 passed
  ocmnode1      gcc-4.4.7-23.0.1.el6      gcc-4.4.4                 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      gcc-c++-4.4.7-23.0.1.el6  gcc-c++-4.4.4             passed
  ocmnode1      gcc-c++-4.4.7-23.0.1.el6  gcc-c++-4.4.4             passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      missing                   ksh                       failed
  ocmnode1      missing                   ksh                       failed
Result: Package existence check failed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      make-3.81-20.el6          make-3.81                 passed
  ocmnode1      make-3.81-20.el6          make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      glibc(x86_64)-2.12-1.132.el6  glibc(x86_64)-2.12        passed
  ocmnode1      glibc(x86_64)-2.12-1.132.el6  glibc(x86_64)-2.12        passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      glibc-devel(x86_64)-2.12-1.132.el6  glibc-devel(x86_64)-2.12  passed
  ocmnode1      glibc-devel(x86_64)-2.12-1.132.el6  glibc-devel(x86_64)-2.12  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  ocmnode1      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      missing                   libaio-devel(x86_64)-0.3.107  failed
  ocmnode1      missing                   libaio-devel(x86_64)-0.3.107  failed
Result: Package existence check failed for "libaio-devel(x86_64)"

Check: Package existence for "nfs-utils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      nfs-utils-1.2.3-39.el6    nfs-utils-1.2.3-15        passed
  ocmnode1      nfs-utils-1.2.3-39.el6    nfs-utils-1.2.3-15        passed
Result: Package existence check passed for "nfs-utils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  ocmnode2          6200          TCP           yes           successful
  ocmnode1          6200          TCP           yes           successful
  ocmnode2          6100          TCP           yes           successful
  ocmnode1          6100          TCP           yes           successful
Result: Port availability check passed for ports "6200,6100"

Checking availability of ports "42424" required for component "Oracle Cluster Synchronization Services (CSSD)"
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  ocmnode2          42424         TCP           yes           successful
  ocmnode1          42424         TCP           yes           successful
Result: Port availability check passed for ports "42424"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  ocmnode2                              passed
  ocmnode1                              passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
  Node Name                             File exists?
  ------------------------------------  ------------------------
  ocmnode2                              yes
  ocmnode1                              yes
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP configuration file "/etc/ntp.conf" existence check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
ocmnode2,ocmnode1
Result: Clock synchronization check using Network Time Protocol(NTP) failed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  ocmnode2      passed                    does not exist
  ocmnode1      passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  ocmnode2      0022                      0022                      passed
  ocmnode1      0022                      0022                      passed
Result: Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of 'domain' and 'search' entries is defined

WARNING:
PRVF-5640 : Both 'search' and 'domain' entries are present in file "/etc/resolv.conf" on the following nodes: ocmnode1,ocmnode2

Check for integrity of file "/etc/resolv.conf" passed

Check: Time zone consistency
Result: Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running

Check: Daemon "avahi-daemon" not configured
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  ocmnode2      no                        passed
  ocmnode1      no                        passed
Daemon not configured check passed for process "avahi-daemon"

Check: Daemon "avahi-daemon" not running
  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  ocmnode2      no                        passed
  ocmnode1      no                        passed
Daemon not running check passed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Starting check for /boot mount ...

Check for /boot mount passed

Starting check for zeroconf check ...

ERROR:

PRVE-10077 : NOZEROCONF parameter was not  specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "ocmnode2.localdomain"
PRVE-10077 : NOZEROCONF parameter was not  specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "ocmnode1.localdomain"

Check for zeroconf check failed

Pre-check for cluster services setup was unsuccessful on all the nodes.

Note:::If you are happy with prerequisites validation, then Install GI.

Step 3: Install GI

Run runInstaller to start installation process.

[oracle@ocmnode1 grid]$ pwd
/u01/software/grid
[oracle@ocmnode1 grid]$ ls
install  response  rpm  runcluvfy.sh  runInstaller  sshsetup  stage  welcome.html
[oracle@ocmnode1 grid]$ ./runInstaller
Starting Oracle Universal Installer...

Select “Install and Configure Oracle Grid Infrastructure for a Cluster” option and Click Next…

Choose Configure a Standard cluster option and Click Next…

Select Installation Type (Advanced Installation) and Click Next…

Language will be default (English) and Click Next…

Enter Cluster Name, Scan Name and port and Click Next…

Default it shows local Node information but need to add all Node information by Clicking Add Button.

Node list will be like below… You may configure SSH connectivity by Click SSH connectivity button or Manually set SSH for Oracle (Or, if you use GRID user for GI).

Select appropriate Interface for Public and Private and Click Next…

For this practice, I used “Standard ASM for storage” but you may explore other option or select appropriate one for you environment. Click Next….

Enter the DISK group name for VotingDisk/OCR. Highly recommend to use Normal or High Redundant for VotingDisk. Click Next…

Provide the password for sys and asmsnp users. If don’t meet the recommendation then it will show below message

Highly recommended to use the strong password.

Choose Do not use IPMI and Click Next…

You may leave it unchecked and install OEM agent later. Click Next..

Select appropriate group for each categorizes and Click Next…

Provide the Software location for GI Home and Click Next…

Click Next…

You may select this option but I think better to run script manually… Click Next…

Prerequisites check

It will show list of warning or failed from minimum acquirement list. Click “Fix & Check Again”.

Run below script on both nodes to fix the requirement.

[root@ocmnode1 ~]# cd /tmp/CVU_12.1.0.2.0_oracle/
[root@ocmnode1 CVU_12.1.0.2.0_oracle]# ./runfixup.sh
All Fix-up operations were completed successfully.
[root@ocmnode2 ~]# cd /tmp/CVU_12.1.0.2.0_oracle/
[root@ocmnode2 CVU_12.1.0.2.0_oracle]# ./runfixup.sh
All Fix-up operations were completed successfully.

Highly recommended to meet all requirements otherwise installation may failed. For VM setup, NTP and DNS option may ignore but all prerequisites should be resolved in Production database in real life. Click Yes for confirmation to proceed the installation process.

Summary report will show all Settings. If you are happy, Click Install otherwise may go back to change the options.

Monitor the Installation Process

Below two scripts need to run on all nodes one by one.

Need to run orainstRoot.sh script on both nodes:

#Node 1:
[root@ocmnode1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

#Note 2:
[root@ocmnode2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

Need to run root.sh script on both node one by one.

[root@ocmnode1 ~]# /u01/app/12.1.0/grid/root.sh
Log for root.sh script on Node 1
[root@ocmnode1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@ocmnode1 ~]# /u01/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2020/08/07 22:28:19 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2020/08/07 22:28:45 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2020/08/07 22:28:46 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2020/08/07 22:29:36 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'ocmnode1'
CRS-2676: Start of 'ora.mdnsd' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.evmd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ocmnode1'
CRS-2676: Start of 'ora.gpnpd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.gipcd' on 'ocmnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'ocmnode1'
CRS-2676: Start of 'ora.diskmon' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'ocmnode1' succeeded

ASM created and started successfully.

Disk Group OCR created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.storage' on 'ocmnode1'
CRS-2676: Start of 'ora.storage' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.crf' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ocmnode1'
CRS-2676: Start of 'ora.crsd' on 'ocmnode1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk cbbbfa3f83074fe9bf67da431a51cb6e.
Successful addition of voting disk bc0fd34d64d44f26bf44ceb14758e21c.
Successful addition of voting disk 4af143af9e234f13bf4cb9b8ef999d88.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   cbbbfa3f83074fe9bf67da431a51cb6e (ORCL:OCRDISK1) [OCR]
 2. ONLINE   bc0fd34d64d44f26bf44ceb14758e21c (ORCL:OCRDISK2) [OCR]
 3. ONLINE   4af143af9e234f13bf4cb9b8ef999d88 (ORCL:OCRDISK3) [OCR]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.crsd' on 'ocmnode1'
CRS-2677: Stop of 'ora.crsd' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'ocmnode1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'ocmnode1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'ocmnode1'
CRS-2677: Stop of 'ora.storage' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'ocmnode1'
CRS-2677: Stop of 'ora.gpnpd' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'ocmnode1' succeeded
CRS-2677: Stop of 'ora.asm' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'ocmnode1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ocmnode1'
CRS-2677: Stop of 'ora.cssd' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'ocmnode1'
CRS-2677: Stop of 'ora.crf' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'ocmnode1'
CRS-2677: Stop of 'ora.gipcd' on 'ocmnode1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ocmnode1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.evmd' on 'ocmnode1'
CRS-2676: Start of 'ora.mdnsd' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.evmd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ocmnode1'
CRS-2676: Start of 'ora.gpnpd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'ocmnode1'
CRS-2676: Start of 'ora.gipcd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ocmnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'ocmnode1'
CRS-2676: Start of 'ora.diskmon' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.ctssd' on 'ocmnode1'
CRS-2676: Start of 'ora.ctssd' on 'ocmnode1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ocmnode1'
CRS-2676: Start of 'ora.asm' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'ocmnode1'
CRS-2676: Start of 'ora.storage' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'ocmnode1'
CRS-2676: Start of 'ora.crf' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ocmnode1'
CRS-2676: Start of 'ora.crsd' on 'ocmnode1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: ocmnode1
CRS-6016: Resource auto-start has completed for server ocmnode1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2020/08/07 22:35:14 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.asm' on 'ocmnode1'
CRS-2676: Start of 'ora.asm' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.OCR.dg' on 'ocmnode1'
CRS-2676: Start of 'ora.OCR.dg' on 'ocmnode1' succeeded
2020/08/07 22:36:27 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Log for root.sh script on Node 2
[root@ocmnode2 ~]# /u01/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2020/08/07 22:40:25 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2020/08/07 22:40:51 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2020/08/07 22:40:52 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2020/08/07 22:42:06 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ocmnode2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'ocmnode2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'ocmnode2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ocmnode2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'ocmnode2'
CRS-2672: Attempting to start 'ora.evmd' on 'ocmnode2'
CRS-2676: Start of 'ora.mdnsd' on 'ocmnode2' succeeded
CRS-2676: Start of 'ora.evmd' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ocmnode2'
CRS-2676: Start of 'ora.gpnpd' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'ocmnode2'
CRS-2676: Start of 'ora.gipcd' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ocmnode2'
CRS-2676: Start of 'ora.cssdmonitor' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ocmnode2'
CRS-2672: Attempting to start 'ora.diskmon' on 'ocmnode2'
CRS-2676: Start of 'ora.diskmon' on 'ocmnode2' succeeded
CRS-2676: Start of 'ora.cssd' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ocmnode2'
CRS-2672: Attempting to start 'ora.ctssd' on 'ocmnode2'
CRS-2676: Start of 'ora.ctssd' on 'ocmnode2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ocmnode2'
CRS-2676: Start of 'ora.asm' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'ocmnode2'
CRS-2676: Start of 'ora.storage' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'ocmnode2'
CRS-2676: Start of 'ora.crf' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ocmnode2'
CRS-2676: Start of 'ora.crsd' on 'ocmnode2' succeeded
CRS-6017: Processing resource auto-start for servers: ocmnode2
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode1'
CRS-2672: Attempting to start 'ora.net1.network' on 'ocmnode2'
CRS-2676: Start of 'ora.net1.network' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'ocmnode2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'ocmnode1'
CRS-2677: Stop of 'ora.scan1.vip' on 'ocmnode1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'ocmnode2'
CRS-2676: Start of 'ora.scan1.vip' on 'ocmnode2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode2'
CRS-2676: Start of 'ora.ons' on 'ocmnode2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'ocmnode2' succeeded
CRS-6016: Resource auto-start has completed for server ocmnode2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2020/08/07 22:46:04 CLSRSC-343: Successfully started Oracle Clusterware stack

2020/08/07 22:46:18 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

After executed above scripts in all Nodes, Click OK. Monitor the process.

Ignore this error and Click OK to continue Install process.

Ignore this error.

Click Yes to confirm. Installation process are done.

Installation completed successfully.

Verify all services:

[root@ocmnode1 ~]# su - grid
[grid@ocmnode1 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@ocmnode1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       ocmnode1                 STABLE
               ONLINE  ONLINE       ocmnode2                 STABLE
ora.OCR.dg
               ONLINE  ONLINE       ocmnode1                 STABLE
               ONLINE  ONLINE       ocmnode2                 STABLE
ora.asm
               ONLINE  ONLINE       ocmnode1                 Started,STABLE
               ONLINE  ONLINE       ocmnode2                 Started,STABLE
ora.net1.network
               ONLINE  ONLINE       ocmnode1                 STABLE
               ONLINE  ONLINE       ocmnode2                 STABLE
ora.ons
               ONLINE  ONLINE       ocmnode1                 STABLE
               ONLINE  ONLINE       ocmnode2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       ocmnode2                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       ocmnode1                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       ocmnode1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       ocmnode1                 169.254.73.37 192.16
                                                             8.10.10,STABLE
ora.cvu
      1        ONLINE  ONLINE       ocmnode1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       ocmnode1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       ocmnode1                 STABLE
ora.ocmnode1.vip
      1        ONLINE  ONLINE       ocmnode1                 STABLE
ora.ocmnode2.vip
      1        ONLINE  ONLINE       ocmnode2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       ocmnode2                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       ocmnode1                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       ocmnode1                 STABLE
--------------------------------------------------------------------------------

Configure ASM

To Configure ASM Disk Group using asmca utility, set GI environment and run asmca command

[grid@ocmnode1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/grid

Check all ASM instances are up and running or not.

Select Disk Groups Tab and Click on Create Button.

Note::: Check why is Usable (GB) showing negative values OCR DiskGroup: Usable (GB) Negative Values

Enter Disk Group Name (DATA) and Choose Redundancy and Select Disks. Click OK

DISK Group Creation is progressing..

Confirmation message

Enter Disk Group Name (FRA) and Choose Redundancy and Select Disks. Click OK

Make sure that all DiskGroups are mounted for all Instances

Install Oracle Database Software Only on RAC

Unzip Oracle Database Binary

[oracle@ocmnode1 software]$ ls -l V*
-rw-r--r-- 1 oracle oinstall 1673544724 Aug  5 23:49 V46095-01_1of2.zip
-rw-r--r-- 1 oracle oinstall 1014530602 Aug  5 23:38 V46095-01_2of2.zip
[oracle@ocmnode1 software]$ unzip V46095-01_1of2.zip
[oracle@ocmnode1 software]$ unzip V46095-01_2of2.zip

Start Installation

Run runInstaller to start installation process.

[oracle@ocmnode1 database]$ pwd
/u01/software/database
[oracle@ocmnode1 database]$ ls -l
total 36
drwxr-xr-x  4 oracle oinstall 4096 Aug  8 00:43 install
drwxrwxr-x  2 oracle oinstall 4096 Jul  7  2014 response
drwxr-xr-x  2 oracle oinstall 4096 Jul  7  2014 rpm
-rwxr-xr-x  1 oracle oinstall 8533 Jul  7  2014 runInstaller
drwxrwxr-x  2 oracle oinstall 4096 Jul  7  2014 sshsetup
drwxr-xr-x 14 oracle oinstall 4096 Jul  7  2014 stage
-rwxr-xr-x  1 oracle oinstall  500 Feb  6  2013 welcome.html
[oracle@ocmnode1 database]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 22268 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 3865 MB    Passed
......

You can unchecked for receiving the security update via MOS. Click Next…

Click Yes to confirm.

Select Installation Option- Create and configure a database and Click Next…

Select “Oracle Real Application Clusters Database Installation” and Click Next.

Select all nodes from list. If you didn’t configure the ssh between Nodes, then provide OS user and password then Click on Setup. If already configured then Test.

SSH configured successfully.

Select language and click on Next.

Choose Enterprise Edition and Click Next…

Provide location for Oracle Base and Software Installation Location. Click Next..

Select Group and Click Next…

Try to resolve all required prerequisites. Click “Fix & Check Again”

Run the fix script on all nodes from root user.

I am ignoring below warning by checking “Ignore All”. But for Production, we may not get these warning or should not ignore. Click Next…

Click Yes to confirm.

Review Summary of all selection and Click Install. If want to change any option then Click on Back.

Installation is progressing.

Installation is progressing.

Execute these scripts from root user on nodes.

Execute the orainstRoot.sh script

[root@ocmnode1 software]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

[root@ocmnode2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

Execute the root.sh script

[root@ocmnode1 software]# /u01/app/oracle/product/12.1.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.1.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

[root@ocmnode2 ~]# /u01/app/oracle/product/12.1.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.1.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

Installation completed successfully.

Configure Database on RAC using DBCA

Run dbca a command from oracle user with GUI mode.

[oracle@ocmnode1 database]$ pwd
/u01/software/database
[oracle@ocmnode1 database]$ ls -l
total 36
drwxr-xr-x  4 oracle oinstall 4096 Aug 10 17:02 install
drwxrwxr-x  2 oracle oinstall 4096 Jul  7  2014 response
drwxr-xr-x  2 oracle oinstall 4096 Jul  7  2014 rpm
-rwxr-xr-x  1 oracle oinstall 8533 Jul  7  2014 runInstaller
drwxrwxr-x  2 oracle oinstall 4096 Jul  7  2014 sshsetup
drwxr-xr-x 14 oracle oinstall 4096 Jul  7  2014 stage
-rwxr-xr-x  1 oracle oinstall  500 Feb  6  2013 welcome.html
[oracle@ocmnode1 database]$ echo $ORACLE_HOME
/u01/app/oracle/product/12.1.0/dbhome_1/

Choose “Create Database” and Click Next.

Click Advanced Mode if want more configuration options. Click Next..

Select Database Type and Configuration Type. Select template for the database. Click Next…

Provide Database Name. If you have plan to create PDB (PLUGGABLE DATABASE) or use Multitenant Architecture then select Create As Container Database Option other ignore. Click Next.

Select all nodes. Click Next..

Unchecked EM related options and can be installed agent for OEM later. Click Next…

Provide password For below users. Click Next..

As this is RAC Database, Select ASM for all options. Enable Archiving option if database is critical. Click next…

Click Next..

Checked on “Use Automatic Memory Management”. Click Other Tab and change accordingly..

Select Appropriate Character Sets for you database based on requirement by application. Click Next.

Click Next.

Recommended to meet all requirements. SCAN related warning may not get if you have DNS setting properly and expect it would be in Real PROD database. Click Ignore All and Click Next..

Recommended to meet all requirements. SCAN related warning may not get if you have DNS setting properly and expect it would be in Real PROD database. Click Ignore All and Click Next..

Database Creation Summary. If happy with setting then Click Finish otherwise Click Back and change settings.

Database Creation is progressing. Can monitor by clicking Activity Log and Alert Log.

Can change the password for other default schema except SYS and SYSTEM by clicking Password Management Button or can be changed at later. Click Close.

[grid@ocmnode1 ~]$ srvctl status database -d orcl
Instance ORCL1 is running on node ocmnode1
Instance ORCL2 is running on node ocmnode2
[oracle@ocmnode1 ~]$ . ./.bash_profile
[oracle@ocmnode1 ~]$ echo $ORACLE_HOME
/u01/app/oracle/product/12.1.0/dbhome_1
[oracle@ocmnode1 ~]$ echo $ORACLE_SID
ORCL1
[oracle@racdr1 ~]$ srvctl status database -d ORCL
Instance ORCL1 is running on node racdr1
Instance ORCL2 is running on node racdr2
[oracle@ocmnode1 ~]$ srvctl config database -db ORCL
Database unique name: ORCL
Database name: ORCL
Oracle home: /u01/app/oracle/product/12.1.0/dbhome_2
Oracle user: oracle
Spfile: +DATA/ORCL/PARAMETERFILE/spfile.269.1048095383
Password file: +DATA/ORCL/PASSWORD/pwdorcl.256.1048094823
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: ORCL1,ORCL2
Configured nodes: ocmnode1,ocmnode2
Database is administrator managed

Clusterware Control (CRSCTL) & Server Control (SRVCTL) to mange Grid and DB:

NOTE::: You don’t need to memorize these command. If you know what command does then can find the required command by your self.

Clusterware Control (CRSCTL) Utility:

[oracle@racdr1 ~]$ crsctl -help
Usage: crsctl add       - add a resource, type or other entity
       crsctl check     - check the state or operating status of a service, resource, or other entity
       crsctl config    - display automatic startup configuration
       crsctl debug     - display or modify debug state
       crsctl delete    - delete a resource, type or other entity
       crsctl disable   - disable automatic startup
       crsctl discover  - discover DHCP server
       crsctl enable    - enable automatic startup
       crsctl eval      - evaluate operations on resource or other entity without performing them
       crsctl get       - get an entity value
       crsctl getperm   - get entity permissions
       crsctl lsmodules - list debug modules
       crsctl modify    - modify a resource, type or other entity
       crsctl query     - query service state
       crsctl pin       - make the leases of specified nodes immutable
       crsctl relocate  - relocate a resource, server or other entity
       crsctl replace   - change the location of voting files
       crsctl release   - release a DHCP lease
       crsctl request   - request a DHCP lease or an action entry point
       crsctl setperm   - set entity permissions
       crsctl set       - set an entity value
       crsctl start     - start a resource, server or other entity
       crsctl status    - get status of a resource or other entity
       crsctl stop      - stop a resource, server or other entity
       crsctl unpin     - make the leases of previously pinned nodes mutable
       crsctl unset     - unset an entity value, restoring its default

More Specific:

[oracle@racdr1 ~]$ crsctl start -help
Usage:
  crsctl start resource {<resName> [...]|-w <filter>]|-all} [-n <server> | -s <server_pools>] [-k <cid>] [-d <did>] [-env "env1=val1,env2=val2,..."] [-begin|-end] [-f] [-i] [-l]
     Start designated resources
more....

Server Control (SRVCTL):

[oracle@racdr1 ~]$ srvctl start -help

The SRVCTL start command starts, Oracle Clusterware enabled, non-running objects.

Usage: srvctl start database -db <db_unique_name> [-startoption <start_options>] [-startconcurrency <start_concurrency>] [-node <node>] [-eval] [-verbose]
Usage: srvctl start instance -db <db_unique_name> {-node <node_name> [-instance <inst_name>] | -instance <inst_name_list>} [-startoption <start_options>]
Usage: srvctl start service -db <db_unique_name> [-service  "<service_name_list>"] [-serverpool <pool_name>] [-node <node_name> | -instance <inst_name>] [-pq] [-global_override] [-startoption <start_options>] [-eval] [-verbose]
more....

Leave a Reply