Oracle Database 12c R1 (12.1) RAC installation

Oracle RAC (Real Application Cluster) is used for complex application to increase database performance, high availability and scalability.

Below high level steps will perform to configure / install Oracle RAC:

  • Configure 2 Virtual Machine (VM) for 2 Nodes RAC
  • Install Oracle Linux 5.7
  • Install GI (Grid Infrastructure)
  • Configure ASM
  • Install Oracle Database Binary
  • Create Database on RAC

Configure 2 Virtual Machine (VM) for 2 Nodes RAC

Install Oracle Linux 5.7

Install GI (Grid Infrastructure)

Oracle GI and Database Installation Prerequisites:

Create Group and User – Oracle for DB (For RAC – recommended to use GRID user for GI)

Note::: But with this setup, we will use Oracle user for Database as well as GI

# From root user:
/usr/sbin/groupadd -g 503 oinstall
/usr/sbin/groupadd -g 504 asmadmin
/usr/sbin/groupadd -g 505 asmdba
/usr/sbin/groupadd -g 506 asmoper
/usr/sbin/groupadd -g 507 dba
/usr/sbin/groupadd -g 508 oper
[root@racdr1 ~]# /usr/sbin/useradd -u 501 -g oinstall -G dba,oper,asmdba,asmadmin,asmoper oracle
[root@racdr1 ~]# id oracle
uid=501(oracle) gid=503(oinstall) groups=503(oinstall),504(asmadmin),505(asmdba),506(asmoper),507(dba),508(oper) context=root:system_r:unconfined_t:SystemLow-SystemHigh

To create home directory for GI and Oracle DB, enter the following commands:

#From root user
mkdir -p /u01/app/12.1.0/grid
chown -R oracle:oinstall /u01/app/12.1.0/grid
chmod -R 775 /u01/app/12.1.0/grid

# To create the Oracle Base directory, enter the following commands:
mkdir -p /u01/app/oracle
mkdir /u01/app/oracle/cfgtoollogs 
chown -R oracle:oinstall /u01/app/oracle/cfgtoollogs
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle

# To create the Oracle Inventory directory, enter the following commands:
mkdir -p /u01/app/oraInventory
chown -R oracle:oinstall /u01/app/oraInventory
chmod -R 775 /u01/app/oraInventory

#To create the Oracle RDBMS Home directory, enter the following commands:
mkdir -p /u01/app/oracle/product/12.1.0/dbhome_1
chown -R oracle:oinstall /u01/app/oracle/product/12.1.0/dbhome_1
chmod -R 775 /u01/app/oracle/product/12.1.0/dbhome_1

Network Setting:

  • eth0 -> will be public IP
  • eth1 -> will be private IP.

IP Setting on Ethernet:

eth0:
Static IP: 192.168.56.220
Network: 255.255.255.0
eth1:
Static IP: 192.168.2.220
Network: 255.255.254.0

/etc/hosts file should be same for both nodes.

[root@racdr1 software]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1              localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
#DR
#Public IP
192.168.56.220   racdr1.localdomain    racdr1
192.168.56.221   racdr2.localdomain    racdr2
#Private IP
192.168.2.220   racdr1-priv.localdomain    racdr1-priv
192.168.2.221   racdr2-priv.localdomain    racdr2-priv
#VIP
192.168.56.222   racdr1-vip.localdomain    racdr1-vip
192.168.56.223   racdr2-vip.localdomain    racdr2-vip
#SCAN IP
192.168.56.224   racdr-scan.localdomain    racdr-scan
192.168.56.225   racdr-scan.localdomain    racdr-scan
192.168.56.226   racdr-scan.localdomain    racdr-scan

#PROD
# Public IP
192.168.56.120   ocmnode1.localdomain    ocmnode1
192.168.56.121   ocmnode2.localdomain    ocmnode2
#Private IP
192.168.2.120   ocmnode1-priv.localdomain    ocmnode1-priv
192.168.2.121   ocmnode2-priv.localdomain    ocmnode2-priv
#VIP
192.168.56.122   ocmnode1-vip.localdomain    ocmnode1-vip
192.168.56.123   ocmnode2-vip.localdomain    ocmnode2-vip
#SCAN IP
192.168.56.124   ocmnode-scan.localdomain    ocmnode-scan
192.168.56.125   ocmnode-scan.localdomain    ocmnode-scan
192.168.56.126   ocmnode-scan.localdomain    ocmnode-scan

Ping on both nodes:

[root@racdr2 ~]# ping racdr1
[root@racdr1 ~]# ping racdr2
[root@racdr1 ~]# ping racdr2-vip
[root@racdr2 ~]# ping racdr1-vip
[root@racdr1 ~]# ping racdr-scan
[root@racdr2 ~]# ping racdr-scan

Set Environment Variable for Oracle user on both Nodes:

$ vi /home/oracle/.bash_profile

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=racdr1; export ORACLE_HOSTNAME
ORACLE_UNQNAME=ORCL; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
GRID_HOME=/u01/app/12.1.0/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/12.1.0/dbhome_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_SID=ORCL1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'

Note::: For Node 2 : All will be same except – ORACLE_SID=ORCL2; export ORACLE_SID

$ vim /home/oracle/grid_env
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

Note::: For Node 2 : All will be same except – ORACLE_SID=+ASM2; export ORACLE_SID

$ vim /home/oracle/db_env

ORACLE_SID=ORCL1; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

Note::: For Node 2 : All will be same except – ORACLE_SID=ORCL2; export ORACLE_SID

OS Kernel Parameter Setting for Oracle:

Add following line to /etc/sysctl.conf

# Oracle
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

Verify Kernel Parameter:

$ /sbin/sysctl -p

Add the following lines to the /etc/security/limits.conf file:

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

Below mentioned packages need to install. You can install package either using yum (if you have installed) otherwise manually.

Package Installation by using yum repository:

yum install binutils-2.15.92.0.2
yum install compat-libstdc++-33-3.2.3
yum install    elfutils-libelf-0.97
yum install    elfutils-libelf-devel-0.97
yum install    expat-1.95.7
yum install    gcc-3.4.6
yum install    gcc-c++-3.4.6
yum install    glibc-2.3.4-2.41
yum install    glibc-common-2.3.4
yum install    glibc-devel-2.3.4
yum install    glibc-headers-2.3.4
yum install    libaio-0.3.105
yum install    libaio-devel-0.3.105
yum install    libgcc-3.4.6
yum install    libstdc++-3.4.6
yum install    libstdc++-devel 3.4.6
yum install    make-3.80
yum install    pdksh-5.2.14
yum install    sysstat-5.0.5
yum install    unixODBC-2.2.11
yum install    unixODBC-devel-2.2.11
yum install    compat-libstdc++-33-3.2.3
yum install    glibc-2.3.4-2.41
yum install    libaio-0.3.105
yum install    libaio-devel-0.3.105
yum install    libgcc-3.4.6
yum install    libstdc++-3.4.6
yum install    unixODBC-2.2.11

Install package by using rmp command:

ISO Disk Mount:

[root@racdr1 ~]# mkdir -p /mnt/disk
[root@racdr1 ~]# mount -o loop /u01/software/V31120-01.iso /mnt/disk

Use below command if yum repository doesn’t setup.

$cd /mnt/disk/Server
$ rpm -Uvh binutils-2.*
rpm -Uvh compat-libstdc++-33*
rpm -Uvh elfutils-libelf-0.*
rpm -Uvh elfutils-libelf-devel-*
rpm -Uvh gcc-4.*
rpm -Uvh gcc-c++-4.*
rpm -Uvh glibc-2.*
rpm -Uvh glibc-common-2.*
rpm -Uvh glibc-devel-2.*
rpm -Uvh glibc-headers-2.*
rpm -Uvh ksh-2*
rpm -Uvh libaio-0.*
rpm -Uvh libaio-devel-0.*
rpm -Uvh libgcc-4.*
rpm -Uvh libstdc++-4.*
rpm -Uvh libstdc++-devel-4.*
rpm -Uvh make-3.*
rpm -Uvh sysstat-7.*
rpm -Uvh unixODBC-2.*
rpm -Uvh unixODBC-devel-2.*

The following command can be run on the system to list the currently installed packages:

Command to verify installed package list:
[root@racdr1 Server]# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils \
compat-libstdc++-33 \
elfutils-libelf \
elfutils-libelf-devel \
gcc \
gcc-c++ \
glibc \
glibc-common \
glibc-devel \
glibc-headers \
ksh \
libaio \
libaio-devel \
libgcc \
libstdc++ \
libstdc++-devel \
make \
sysstat \
unixODBC \
unixODBC-devel
OUTPUT: (Show all list and find which packages are not installed. 
binutils-2.17.50.0.6-20.el5 (x86_64)
compat-libstdc++-33-3.2.3-61 (x86_64)
compat-libstdc++-33-3.2.3-61 (i386)
elfutils-libelf-0.137-3.el5 (x86_64)
package elfutils-libelf-devel is not installed
gcc-4.1.2-52.el5 (x86_64)
package gcc-c++ is not installed
glibc-2.5-81 (x86_64)
glibc-2.5-81 (i686)
glibc-common-2.5-81 (x86_64)
glibc-devel-2.5-81 (x86_64)
glibc-headers-2.5-81 (x86_64)
ksh-20100621-5.el5 (x86_64)
libaio-0.3.106-5 (x86_64)
libaio-0.3.106-5 (i386)
libaio-devel-0.3.106-5 (x86_64)
libaio-devel-0.3.106-5 (i386)
libgcc-4.1.2-52.el5 (x86_64)
libgcc-4.1.2-52.el5 (i386)
libstdc++-4.1.2-52.el5 (x86_64)
libstdc++-4.1.2-52.el5 (i386)
libstdc++-devel-4.1.2-52.el5 (x86_64)
libstdc++-devel-4.1.2-52.el5 (i386)
make-3.81-3.el5 (x86_64)
sysstat-7.0.2-11.el5 (x86_64)
package unixODBC is not installed
package unixODBC-devel is not installed

Install Oracle ASM related package:

[root@racdr1 software]# rpm -Uvh oracleasmlib-2.0.4-1.el5.x86_64.rpm
warning: oracleasmlib-2.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:oracleasmlib           ########################################### [100%]
[root@racdr1 software]# rpm -Uvh oracleasm-support-2.1.3-1.el5.x86_64.rpm
warning: oracleasm-support-2.1.3-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:oracleasm-support      ########################################### [100%]
[root@racdr1 software]# rpm -Uvh oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64*
warning: oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:oracleasm-2.6.18-164.el########################################### [100%]

Configure Disk For ASM on RAC

Create necessary shareable disks for ASM. How to create shareable disk on VM

Create Partition on ROW Disk:

[root@racdr1 ~]# cd /dev/
[root@racdr1 dev]# ls sd*
[root@racdr1 dev]# ls sd*
sda   sda2  sda4  sda6  sdb  sdd  sdf  sdh
sda1  sda3  sda5  sda7  sdc  sde  sdg  sdi

To verify which ROW disk are already mapping with filesystem: use ($ mount or $df -h) command.

[root@racdr1 dev]# mount
[root@racdr1 dev]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             2.0G  486M  1.4G  26% /
/dev/mapper/VolGroup00-LogVol00
                       70G  3.8G   62G   6% /u01
/dev/sda6             494M   11M  458M   3% /home
/dev/sda3             2.0G   68M  1.9G   4% /tmp
/dev/sda2             3.9G  1.9G  1.9G  51% /usr
/dev/sda1              99M   22M   73M  23% /boot
tmpfs                 1.5G     0  1.5G   0% /dev/shm
/dev/sr0              3.7G  3.7G     0 100% /media/OL5.8 x86_64 dvd 20120229

User fdisk command to format a row disk. Follow below steps for each shareable disks (sdb to sdi).

[root@racdr1 dev]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-522, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-522, default 522):
Using default value 522

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

After partitioning all disk it will show like below:

[root@racdr1 dev]# ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sda4  /dev/sda6  /dev/sdb   /dev/sdc   /dev/sdd   /dev/sde   /dev/sdf   /dev/sdg   /dev/sdh   /dev/sdi
/dev/sda1  /dev/sda3  /dev/sda5  /dev/sda7  /dev/sdb1  /dev/sdc1  /dev/sdd1  /dev/sde1  /dev/sdf1  /dev/sdg1  /dev/sdh1  /dev/sdi1

Create ASM Disk:

[root@racdr1 dev]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library driver.  The following questions will determine whether the driver is loaded on boot and what permissions it will have.  The current values will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
[root@racdr1 dev]# /usr/sbin/oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm

Run below command for all disks [sdb1 – sdi1]

[root@racdr1 dev]# oracleasm createdisk OCRDISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@racdr1 dev]# oracleasm createdisk OCRDISK2 /dev/sdc1
Writing disk header: done
Instantiating disk: done
[root@racdr1 dev]# oracleasm createdisk OCRDISK3 /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@racdr1 dev]# oracleasm createdisk DATADISK1 /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@racdr1 dev]# oracleasm createdisk DATADISK2 /dev/sdf1
Writing disk header: done
Instantiating disk: done
[root@racdr1 dev]# oracleasm createdisk DATADISK3 /dev/sdg1
Writing disk header: done
Instantiating disk: done
[root@racdr1 dev]# oracleasm createdisk DATADISK4 /dev/sdh1
Writing disk header: done
Instantiating disk: done
[root@racdr1 dev]# oracleasm createdisk DATADISK5 /dev/sdi1
Writing disk header: done
Instantiating disk: done

If need to delete any ASM disks, use below command:

Delete existing Disk:
[root@racdr1 dev]# oracleasm deletedisk OCRDISK1 
Clearing disk header: done
Dropping disk: done
[root@racdr1 dev]# oracleasm deletedisk OCRDISK2
Clearing disk header: done
Dropping disk: done

Verify Disks:

[root@racdr1 dev]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@racdr1 dev]# /usr/sbin/oracleasm listdisks
DATADISK1
DATADISK2
DATADISK3
DATADISK4
DATADISK5
OCRDISK1
OCRDISK2
OCRDISK3

This VM is almost ready. We can use this VM for Node 1 and Clone this VM for Node 2.

Note:::Once we have Node 2, We will check some prerequisites.

On Node 2:

Verify Disks:

[root@racdr1 dev]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@racdr1 dev]# /usr/sbin/oracleasm listdisks
DATADISK1
DATADISK2
DATADISK3
DATADISK4
DATADISK5
OCRDISK1
OCRDISK2
OCRDISK3

Step 1: unzip GI binary file on one node

# From Oracle USER. If you have GRID user for GI, then run below command from GRID
[oracle@racdr1 ~]$ cd /u01/software/
[oracle@racdr1 software]$ ls -lrt *grid*
-rw-r--r-- 1 oracle oinstall  646972897 Feb  7  2019 linuxamd64_12102_grid_2of2.zip
-rw-r--r-- 1 oracle oinstall 1747043545 Feb  7  2019 linuxamd64_12102_grid_1of2.zip

[oracle@racdr1 software]$ unzip linuxamd64_12102_grid_1of2.zip
[oracle@racdr1 software]$ unzip linuxamd64_12102_grid_2of2.zip

Step 2: run below command (runcluvfy.sh) for prerequisite check before final installation process start.

[oracle@racdr1 grid]$ pwd
/u01/software/grid
[oracle@racdr1 grid]$ ls
install  response  rpm  runcluvfy.sh  runInstaller  sshsetup  stage  welcome.html
[oracle@racdr1 grid]$ ./runcluvfy.sh stage -pre crsinst -n racdr1,racdr2 -verbose
Prerequisites Check (runcluvfy.sh) Log:
[oracle@racdr1 grid]$ ./runcluvfy.sh stage -pre crsinst -n racdr1,racdr2 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "racdr1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  racdr1                                yes
  racdr2                                yes
Result: Node reachability check passed from node "racdr1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  racdr2                                passed
  racdr1                                passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  racdr1                                passed
  racdr2                                passed

Verification of the hosts config file successful


Interface information for node "racdr1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.56.220  192.168.56.0    0.0.0.0         UNKNOWN         08:00:27:0D:F4:E3 1500


Interface information for node "racdr2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.56.221  192.168.56.0    0.0.0.0         UNKNOWN         08:00:27:46:D3:12 1500


Check: Node connectivity of subnet "192.168.56.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  racdr1[192.168.56.220]          racdr2[192.168.56.221]          yes
Result: Node connectivity passed for subnet "192.168.56.0" with node(s) racdr1,racdr2


Check: TCP connectivity of subnet "192.168.56.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  racdr1 : 192.168.56.220         racdr1 : 192.168.56.220         passed
  racdr2 : 192.168.56.221         racdr1 : 192.168.56.220         passed
  racdr1 : 192.168.56.220         racdr2 : 192.168.56.221         passed
  racdr2 : 192.168.56.221         racdr2 : 192.168.56.221         passed
Result: TCP connectivity check passed for subnet "192.168.56.0"


Interfaces found on subnet "192.168.56.0" that are likely candidates for a private interconnect are:
racdr1 eth0:192.168.56.220
racdr2 eth0:192.168.56.221

WARNING:
Could not find a suitable set of interfaces for VIPs
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.56.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.56.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  racdr1                                passed
  racdr2                                passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        2.9634GB (3107372.0KB)    4GB (4194304.0KB)         failed
  racdr1        2.9634GB (3107372.0KB)    4GB (4194304.0KB)         failed
Result: Total memory check failed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        2.5547GB (2678772.0KB)    50MB (51200.0KB)          passed
  racdr1        2.4122GB (2529368.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        0.0 bytes                 2.9634GB (3107372.0KB)    failed
  racdr1        0.0 bytes                 2.9634GB (3107372.0KB)    failed
Result: Swap space check failed

Check: Free disk space for "racdr2:/usr"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              racdr2        /usr          1.9043GB      25MB          passed
Result: Free disk space check passed for "racdr2:/usr"

Check: Free disk space for "racdr1:/usr"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              racdr1        /usr          1.9043GB      25MB          passed
Result: Free disk space check passed for "racdr1:/usr"

Check: Free disk space for "racdr2:/var,racdr2:/etc,racdr2:/sbin"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /var              racdr2        /             1.4277GB      40MB          passed
  /etc              racdr2        /             1.4277GB      40MB          passed
  /sbin             racdr2        /             1.4277GB      40MB          passed
Result: Free disk space check passed for "racdr2:/var,racdr2:/etc,racdr2:/sbin"

Check: Free disk space for "racdr1:/var,racdr1:/etc,racdr1:/sbin"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /var              racdr1        /             1.4287GB      40MB          passed
  /etc              racdr1        /             1.4287GB      40MB          passed
  /sbin             racdr1        /             1.4287GB      40MB          passed
Result: Free disk space check passed for "racdr1:/var,racdr1:/etc,racdr1:/sbin"

Check: Free disk space for "racdr2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              racdr2        /tmp          1.8926GB      1GB           passed
Result: Free disk space check passed for "racdr2:/tmp"

Check: Free disk space for "racdr1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              racdr1        /tmp          1.1335GB      1GB           passed
Result: Free disk space check passed for "racdr1:/tmp"

Check: User existence for "oracle"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  racdr2        passed                    exists(501)
  racdr1        passed                    exists(501)

Checking for multiple users with UID value 501
Result: Check for multiple users with UID value 501 passed
Result: User existence check passed for "oracle"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  racdr2        passed                    exists
  racdr1        passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  racdr2        passed                    exists
  racdr1        passed                    exists
Result: Group existence check passed for "dba"

Check: Membership of user "oracle" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr2            yes           yes           yes           yes           passed
  racdr1            yes           yes           yes           yes           passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed

Check: Membership of user "oracle" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  racdr2            yes           yes           yes           passed
  racdr1            yes           yes           yes           passed
Result: Membership check for user "oracle" in group "dba" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        5                         3,5                       passed
  racdr1        5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  racdr2            hard          65536         65536         passed
  racdr1            hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  racdr2            soft          1024          1024          passed
  racdr1            soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  racdr2            hard          16384         16384         passed
  racdr1            hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  racdr2            soft          2047          2047          passed
  racdr1            soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        x86_64                    x86_64                    passed
  racdr1        x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        2.6.32-300.10.1.el5uek    2.6.18                    passed
  racdr1        2.6.32-300.10.1.el5uek    2.6.18                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            250           250           250           passed
  racdr2            250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            32000         32000         32000         passed
  racdr2            32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            100           100           100           passed
  racdr2            100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            128           128           128           passed
  racdr2            128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            68719476736   68719476736   1590974464    passed
  racdr2            68719476736   68719476736   1590974464    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            4096          4096          4096          passed
  racdr2            4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            4294967296    4294967296    310737        passed
  racdr2            4294967296    4294967296    310737        passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            6815744       6815744       6815744       passed
  racdr2            6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  racdr2            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            262144        262144        262144        passed
  racdr2            262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            4194304       4194304       4194304       passed
  racdr2            4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            262144        262144        262144        passed
  racdr2            262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            1048576       1048576       1048576       passed
  racdr2            1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            65536         unknown       1048576       failed        Current value incorrect. Configured value incorrect.
  racdr2            65536         unknown       1048576       failed        Current value incorrect. Configured value incorrect.

PRVG-1206 : Check cannot be performed for configured value of kernel parameter "aio-max-nr" on node "racdr1"
PRVG-1206 : Check cannot be performed for configured value of kernel parameter "aio-max-nr" on node "racdr2"

Result: Kernel parameter check failed for "aio-max-nr"

Check: Kernel parameter for "panic_on_oops"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  racdr1            1             unknown       1             failed (ignorable)  Configured value incorrect.
  racdr2            1             unknown       1             failed (ignorable)  Configured value incorrect.

PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "racdr1"
PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "racdr2"

Result: Kernel parameter check passed for "panic_on_oops"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        make-3.81-3.el5           make-3.81                 passed
  racdr1        make-3.81-3.el5           make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        binutils-2.17.50.0.6-20.el5  binutils-2.17.50.0.6      passed
  racdr1        binutils-2.17.50.0.6-20.el5  binutils-2.17.50.0.6      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        gcc(x86_64)-4.1.2-52.el5  gcc(x86_64)-4.1.2         passed
  racdr1        gcc(x86_64)-4.1.2-52.el5  gcc(x86_64)-4.1.2         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
  racdr1        libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        glibc(x86_64)-2.5-81      glibc(x86_64)-2.5-58      passed
  racdr1        glibc(x86_64)-2.5-81      glibc(x86_64)-2.5-58      passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
  racdr1        compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        glibc-devel(x86_64)-2.5-81  glibc-devel(x86_64)-2.5   passed
  racdr1        glibc-devel(x86_64)-2.5-81  glibc-devel(x86_64)-2.5   passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        missing                   gcc-c++(x86_64)-4.1.2     failed
  racdr1        missing                   gcc-c++(x86_64)-4.1.2     failed
Result: Package existence check failed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        libaio-devel(x86_64)-0.3.106-5  libaio-devel(x86_64)-0.3.106  passed
  racdr1        libaio-devel(x86_64)-0.3.106-5  libaio-devel(x86_64)-0.3.106  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        libgcc(x86_64)-4.1.2-52.el5  libgcc(x86_64)-4.1.2      passed
  racdr1        libgcc(x86_64)-4.1.2-52.el5  libgcc(x86_64)-4.1.2      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        libstdc++(x86_64)-4.1.2-52.el5  libstdc++(x86_64)-4.1.2   passed
  racdr1        libstdc++(x86_64)-4.1.2-52.el5  libstdc++(x86_64)-4.1.2   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        libstdc++-devel(x86_64)-4.1.2-52.el5  libstdc++-devel(x86_64)-4.1.2  passed
  racdr1        libstdc++-devel(x86_64)-4.1.2-52.el5  libstdc++-devel(x86_64)-4.1.2  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        sysstat-7.0.2-11.el5      sysstat-7.0.2             passed
  racdr1        sysstat-7.0.2-11.el5      sysstat-7.0.2             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        ksh                       ksh                       passed
  racdr1        ksh                       ksh                       passed
Result: Package existence check passed for "ksh"

Check: Package existence for "nfs-utils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        nfs-utils-1.0.9-60.0.1.el5  nfs-utils-1.0.9-60        passed
  racdr1        nfs-utils-1.0.9-60.0.1.el5  nfs-utils-1.0.9-60        passed
Result: Package existence check passed for "nfs-utils"

Check: Package existence for "coreutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  racdr2        coreutils-5.97-34.el5     coreutils-5.97-23         passed
  racdr1        coreutils-5.97-34.el5     coreutils-5.97-23         passed
Result: Package existence check passed for "coreutils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  racdr2            6200          TCP           yes           successful
  racdr1            6200          TCP           yes           successful
  racdr2            6100          TCP           yes           successful
  racdr1            6100          TCP           yes           successful
Result: Port availability check passed for ports "6200,6100"

Checking availability of ports "42424" required for component "Oracle Cluster Synchronization Services (CSSD)"
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  racdr2            42424         TCP           yes           successful
  racdr1            42424         TCP           yes           successful
Result: Port availability check passed for ports "42424"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  racdr2                                passed
  racdr1                                passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
  Node Name                             File exists?
  ------------------------------------  ------------------------
  racdr2                                yes
  racdr1                                yes
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP configuration file "/etc/ntp.conf" existence check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
racdr2,racdr1
Result: Clock synchronization check using Network Time Protocol(NTP) failed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  racdr2        passed                    does not exist
  racdr1        passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  racdr2        0022                      0022                      passed
  racdr1        0022                      0022                      passed
Result: Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

PRVF-5600 : On node "racdr1" The following lines in file "/etc/resolv.conf" could not be parsed as they are not in proper format: racdr1
PRVF-5600 : On node "racdr2" The following lines in file "/etc/resolv.conf" could not be parsed as they are not in proper format: racdr2

Check for integrity of file "/etc/resolv.conf" failed

Check: Time zone consistency
Result: Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running

Check: Daemon "avahi-daemon" not configured
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  racdr2        yes                       failed
  racdr1        yes                       failed
Daemon not configured check failed for process "avahi-daemon"

Check: Daemon "avahi-daemon" not running
  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  racdr2        yes                       failed
  racdr1        yes                       failed
Daemon not running check failed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Starting check for /boot mount ...

Check for /boot mount passed

Starting check for zeroconf check ...

ERROR:

PRVE-10077 : NOZEROCONF parameter was not  specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "racdr2"
PRVE-10077 : NOZEROCONF parameter was not  specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "racdr1"

Check for zeroconf check failed

Pre-check for cluster services setup was unsuccessful on all the nodes.
[oracle@racdr1 grid]$

Note:::If you are happy with prerequisites validation, then Install GI.

Step 3: Install GI

Run runInstaller to start installation process.

[oracle@racdr1 grid]$ pwd
/u01/software/grid
[oracle@racdr1 grid]$ ls
install  response  rpm  runcluvfy.sh  runInstaller  sshsetup  stage  welcome.html
[oracle@racdr1 grid]$ ./runInstaller
Starting Oracle Universal Installer...

Select “Install and Configure Oracle Grid Infrastructure for a Cluster” option and Click Next…

Choose Configure a Standard cluster option and Click Next…

Select Installation Type (Advanced Installation) and Click Next…

Language will be default (English) and Click Next…

Enter Cluster Name, Scan Name and port and Click Next…

Default it shows local Node information but need to add all Node information by Clicking Add Button.

Node list will be like below… You may configure SSH connectivity by Click SSH connectivity button or Manually set SSH for Oracle (Or, if you use GRID user for GI).

Select appropriate Interface for Public and Private and Click Next…

For this practice, I used “Standard ASM for storage” but you may explore other option or select appropriate one for you environment. Click Next….

Enter the DISK group name for VotingDisk/OCR. Highly recommend to use Normal or High Redundant for VotingDisk. Click Next…

Provide the password for sys and asmsnp users. If don’t meet the recommendation then it will show below message

Highly recommended to use the strong password.

Choose Do not use IPMI and Click Next…

You may leave it unchecked and install OEM agent later. Click Next..

Select appropriate group for each categorizes and Click Next…

Provide the Software location for GI Home and Click Next…

Click Next…

You may select this option but I think better to run script manually… Click Next…

Prerequisites check

It will show list of warning or failed from minimum acquirement list. Click “Fix & Check Again”.

Run below script on both nodes to fix the requirement.

Run above recommended script on both nodes.

[root@racdr1 ~]# cd /tmp/CVU_12.1.0.2.0_oracle/
[root@racdr1 CVU_12.1.0.2.0_oracle]# ./runfixup.sh
All Fix-up operations were completed successfully.
[root@racdr2 ~]# cd /tmp/CVU_12.1.0.2.0_oracle/
[root@racdr2 CVU_12.1.0.2.0_oracle]# ./runfixup.sh
All Fix-up operations were completed successfully.

Highly recommended to meet all requirements otherwise installation may failed. For VM setup, NTP and DNS option may ignore but in really life it should be resolved.

Click Yes for confirmation to proceed the installation process.

Summary report will show all Settings. If you are happy, Click Install otherwise may go back to change the options.

Monitor the Installation Process

Below two scripts need to run on all nodes one by one.

Need to run orainstRoot.sh script on both nodes:

#Node 1:
[root@racdr1 CVU_12.1.0.2.0_oracle]# cd /u01/app/oraInventory/
[root@racdr1 oraInventory]# ls 
backup  ContentsXML  logs  oraInst.loc  orainstRoot.sh
[root@racdr1 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

#Note 2:
[root@racdr1 oraInventory]# ssh racdr2
root@racdr2's password:
Last login: Tue Jun 23 13:24:35 2020 from racdr1.localdomain
[root@racdr2 ~]# cd /u01/app/oraInventory/
[root@racdr2 oraInventory]# ls 
backup  ContentsXML  logs  oraInst.loc  orainstRoot.sh
[root@racdr2 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

Need to run root.sh script on both node one by one.

[root@racdr1 grid]# pwd
/u01/app/12.1.0/grid
[root@racdr1 grid]# ls -lrt root.sh
-rwxr-x--- 1 oracle oinstall 471 Jun 23 13:39 root.sh
[root@racdr1 grid]# ./root.sh
Log for root.sh script on Node 1

Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2020/06/23 17:55:23 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2020/06/23 17:55:49 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2020/06/23 17:55:50 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2020/06/23 17:56:40 CLSRSC-330: Adding Clusterware entries to file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'racdr1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'racdr1'
CRS-2676: Start of 'ora.mdnsd' on 'racdr1' succeeded
CRS-2676: Start of 'ora.evmd' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racdr1'
CRS-2676: Start of 'ora.gpnpd' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racdr1'
CRS-2672: Attempting to start 'ora.gipcd' on 'racdr1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racdr1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racdr1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racdr1'
CRS-2676: Start of 'ora.diskmon' on 'racdr1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racdr1' succeeded

ASM created and started successfully.

Disk Group OCR created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'racdr1'
CRS-2672: Attempting to start 'ora.storage' on 'racdr1'
CRS-2676: Start of 'ora.storage' on 'racdr1' succeeded
CRS-2676: Start of 'ora.crf' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racdr1'
CRS-2676: Start of 'ora.crsd' on 'racdr1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 583fa0bac67c4fb2bf8c7573c78c9ac5.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1245fd1fdf6e4f0dbf507f6e6abb1061 (ORCL:OCR_DISK01) [OCR]
 2. ONLINE   a8594f4803f64ff5bff9114decda1b01 (ORCL:OCR_DISK02) [OCR]
 3. ONLINE   98084ddf39354f19bfa787e6c55f3214 (ORCL:OCR_DISK03) [OCR]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racdr1'
CRS-2673: Attempting to stop 'ora.crsd' on 'racdr1'
CRS-2677: Stop of 'ora.crsd' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'racdr1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racdr1'
CRS-2673: Attempting to stop 'ora.crf' on 'racdr1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racdr1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racdr1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racdr1'
CRS-2677: Stop of 'ora.storage' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.crf' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'racdr1'
CRS-2673: Attempting to stop 'ora.asm' on 'racdr1'
CRS-2677: Stop of 'ora.evmd' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.asm' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'racdr1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racdr1'
CRS-2677: Stop of 'ora.cssd' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racdr1'
CRS-2677: Stop of 'ora.gipcd' on 'racdr1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racdr1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'racdr1'
CRS-2672: Attempting to start 'ora.evmd' on 'racdr1'
CRS-2676: Start of 'ora.mdnsd' on 'racdr1' succeeded
CRS-2676: Start of 'ora.evmd' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racdr1'
CRS-2676: Start of 'ora.gpnpd' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racdr1'
CRS-2676: Start of 'ora.gipcd' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racdr1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racdr1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racdr1'
CRS-2676: Start of 'ora.diskmon' on 'racdr1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'racdr1'
CRS-2672: Attempting to start 'ora.ctssd' on 'racdr1'
CRS-2676: Start of 'ora.ctssd' on 'racdr1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racdr1'
CRS-2676: Start of 'ora.asm' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'racdr1'
CRS-2676: Start of 'ora.storage' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'racdr1'
CRS-2676: Start of 'ora.crf' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racdr1'
CRS-2676: Start of 'ora.crsd' on 'racdr1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: racdr1
CRS-6016: Resource auto-start has completed for server racdr1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2020/06/23 18:01:00 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.asm' on 'racdr1'
CRS-2676: Start of 'ora.asm' on 'racdr1' succeeded
CRS-2672: Attempting to start 'ora.OCR.dg' on 'racdr1'
CRS-2676: Start of 'ora.OCR.dg' on 'racdr1' succeeded
2020/06/23 18:02:01 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Log for root.sh script on Node 2

[root@racdr2 ~]# cd /u01/app/12.1.0/grid/
[root@racdr2 grid]# ./root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2020/06/23 18:05:36 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2020/06/23 18:06:02 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2020/06/23 18:06:03 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2020/06/23 18:07:15 CLSRSC-330: Adding Clusterware entries to file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racdr2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racdr2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'racdr2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racdr2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'racdr2'
CRS-2672: Attempting to start 'ora.evmd' on 'racdr2'
CRS-2676: Start of 'ora.mdnsd' on 'racdr2' succeeded
CRS-2676: Start of 'ora.evmd' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racdr2'
CRS-2676: Start of 'ora.gpnpd' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racdr2'
CRS-2676: Start of 'ora.gipcd' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racdr2'
CRS-2676: Start of 'ora.cssdmonitor' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racdr2'
CRS-2672: Attempting to start 'ora.diskmon' on 'racdr2'
CRS-2676: Start of 'ora.diskmon' on 'racdr2' succeeded
CRS-2676: Start of 'ora.cssd' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'racdr2'
CRS-2672: Attempting to start 'ora.ctssd' on 'racdr2'
CRS-2676: Start of 'ora.ctssd' on 'racdr2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racdr2'
CRS-2676: Start of 'ora.asm' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'racdr2'
CRS-2676: Start of 'ora.storage' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'racdr2'
CRS-2676: Start of 'ora.crf' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racdr2'
CRS-2676: Start of 'ora.crsd' on 'racdr2' succeeded
CRS-6017: Processing resource auto-start for servers: racdr2
CRS-2672: Attempting to start 'ora.net1.network' on 'racdr2'
CRS-2676: Start of 'ora.net1.network' on 'racdr2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'racdr2'
CRS-2676: Start of 'ora.ons' on 'racdr2' succeeded
CRS-6016: Resource auto-start has completed for server racdr2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2020/06/23 18:10:04 CLSRSC-343: Successfully started Oracle Clusterware stack

2020/06/23 18:10:18 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

After executed above scripts in all Nodes, Click OK. Monitor the process.

Ignore this error and Click OK to continue Install process.

Ignore this error.

Click Yes to confirm. Installation process are done.

Installation completed successfully.

Verify all services:

[root@racdr1 grid]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@racdr1 grid]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       racdr1                   STABLE
               ONLINE  ONLINE       racdr2                   STABLE
ora.OCR.dg
               ONLINE  ONLINE       racdr1                   STABLE
               ONLINE  ONLINE       racdr2                   STABLE
ora.asm
               ONLINE  ONLINE       racdr1                   STABLE
               ONLINE  ONLINE       racdr2                   Started,STABLE
ora.net1.network
               ONLINE  ONLINE       racdr1                   STABLE
               ONLINE  ONLINE       racdr2                   STABLE
ora.ons
               ONLINE  ONLINE       racdr1                   STABLE
               ONLINE  ONLINE       racdr2                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racdr1                   STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       racdr1                   169.254.9.200 192.16
                                                             8.2.220,STABLE
ora.cvu
      1        ONLINE  ONLINE       racdr1                   STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       racdr1                   Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       racdr1                   STABLE
ora.racdr1.vip
      1        ONLINE  ONLINE       racdr1                   STABLE
ora.racdr2.vip
      1        ONLINE  ONLINE       racdr2                   STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       racdr1                   STABLE
--------------------------------------------------------------------------------

Configure ASM

To Configure ASM Disk Group using asmca utility, set GI environment and run asmca command

You may check all ASM instances are up and running or not. Select Disk Groups Tab

Click Create Button

Enter Disk Group Name (FRA) and Choose Redundancy and Select Disks. Click OK

Confirmation message

If Disk Group is not mounted in all Instances, then Click Mount All Button

Install Oracle Database Software and Configure the Database

Unzip Oracle Database Binary

[oracle@racdr1 software]$ ls -lrt
-rw-r--r-- 1 oracle oinstall 1014530602 Feb  7  2019 linuxamd64_12102_database_2of2.zip
-rw-r--r-- 1 oracle oinstall 1673544724 Feb  7  2019 linuxamd64_12102_database_1of2.zip

[oracle@racdr1 software]$ unzip linuxamd64_12102_database_1of2.zip
[oracle@racdr1 software]$ unzip linuxamd64_12102_database_2of2.zip
[oracle@racdr1 software]$ ls
database  linuxamd64_12102_database_1of2.zip  linuxamd64_12102_database_2of2.zip

Start Installation

run runInstaller to start installation process.

[oracle@racdr1 database]$ pwd
/u01/software/database
[oracle@racdr1 database]$ ls
install  response  rpm  runInstaller  sshsetup  stage  welcome.html
[oracle@racdr1 database]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 1341 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 4491 MB    Passed

You can unchecked for receiving the security update via MOS. Click Next…

Select Installation Option- Create and configure a database and Click Next…

Select Server Class and Click Next…

Choose GI option – It should be Oracle Real Application Cluster database installation. Click Next…

Select RAC Database Configuration Type and Click Next…

Select all Nodes.

Select Languages.. Click Next…

Select the Database Edition. Based on your license, select the Edition and Click Next…

Provide Oracle Base and Home Location. Click Next…

Select the Database Configuration Type. Very important to choose right one based on your purpose. Click Next…

Provide the Database Name. Click Next…

Check “Enable Automatic Memory Management” and assign the total memory for Oracle.

Select Character Sets based on your requirement. This is very important for your environment otherwise you may face issue later. Click Next…

Select Storage Options. For RAC, It should be ASM. Click Next.

You may provide OMS details during installation and configure agent later. Click Next…

Select the Recovery Options and Click Next…

Select the ASM Group for database. Click Next.

Enter the password for sys, system and dbsnmp users. Click Next…

Assign the OS GROUP for each database group and click Next…

[root@racdr1 CVU_12.1.0.2.0_oracle]# ./runfixup.sh
All Fix-up operations were completed successfully.

Below list of prerequisite did not resolved by running the fixup script. Recommended to resolve. Below prerequisite I ignored for this setup.

Summary of all setting. If you are good with all setting then Click Install

Installation is in progress

Open a new terminal or connect from putty and execute the root.sh script on both nodes from root user and once execution is done, Click OK.

Execute the root.sh script

[root@racdr1 CVU_12.1.0.2.0_oracle]# cd /u01/app/oracle/product/12.1.0/dbhome_1/
[root@racdr1 dbhome_1]# ./root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.1.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@racdr1 dbhome_1]# pwd
/u01/app/oracle/product/12.1.0/dbhome_1

Database creation is progressing

Installation completed successfully.

[oracle@racdr1 ~]$ srvctl status database -d ORCL
Instance ORCL1 is running on node racdr1
Instance ORCL2 is running on node racdr2
[oracle@racdr1 ~]$ srvctl config database -db ORCL
Database unique name: ORCL
Database name: ORCL
Oracle home: /u01/app/oracle/product/12.1.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/ORCL/PARAMETERFILE/spfile.273.1043890263
Password file: +DATA/ORCL/PASSWORD/pwdorcl.256.1043889945
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oinstall
Database instances: ORCL1,ORCL2
Configured nodes: racdr1,racdr2
Database is administrator managed

Clusterware Control (CRSCTL) & Server Control (SRVCTL) to mange Grid and DB:

NOTE::: You don’t need to memorize these command. If you know what command does then can find the required command by your self.

Clusterware Control (CRSCTL) Utility:

[oracle@racdr1 ~]$ crsctl -help
Usage: crsctl add       - add a resource, type or other entity
       crsctl check     - check the state or operating status of a service, resource, or other entity
       crsctl config    - display automatic startup configuration
       crsctl debug     - display or modify debug state
       crsctl delete    - delete a resource, type or other entity
       crsctl disable   - disable automatic startup
       crsctl discover  - discover DHCP server
       crsctl enable    - enable automatic startup
       crsctl eval      - evaluate operations on resource or other entity without performing them
       crsctl get       - get an entity value
       crsctl getperm   - get entity permissions
       crsctl lsmodules - list debug modules
       crsctl modify    - modify a resource, type or other entity
       crsctl query     - query service state
       crsctl pin       - make the leases of specified nodes immutable
       crsctl relocate  - relocate a resource, server or other entity
       crsctl replace   - change the location of voting files
       crsctl release   - release a DHCP lease
       crsctl request   - request a DHCP lease or an action entry point
       crsctl setperm   - set entity permissions
       crsctl set       - set an entity value
       crsctl start     - start a resource, server or other entity
       crsctl status    - get status of a resource or other entity
       crsctl stop      - stop a resource, server or other entity
       crsctl unpin     - make the leases of previously pinned nodes mutable
       crsctl unset     - unset an entity value, restoring its default

More Specific:

[oracle@racdr1 ~]$ crsctl start -help
Usage:
  crsctl start resource {<resName> [...]|-w <filter>]|-all} [-n <server> | -s <server_pools>] [-k <cid>] [-d <did>] [-env "env1=val1,env2=val2,..."] [-begin|-end] [-f] [-i] [-l]
     Start designated resources
more....

Server Control (SRVCTL):

[oracle@racdr1 ~]$ srvctl start -help

The SRVCTL start command starts, Oracle Clusterware enabled, non-running objects.

Usage: srvctl start database -db <db_unique_name> [-startoption <start_options>] [-startconcurrency <start_concurrency>] [-node <node>] [-eval] [-verbose]
Usage: srvctl start instance -db <db_unique_name> {-node <node_name> [-instance <inst_name>] | -instance <inst_name_list>} [-startoption <start_options>]
Usage: srvctl start service -db <db_unique_name> [-service  "<service_name_list>"] [-serverpool <pool_name>] [-node <node_name> | -instance <inst_name>] [-pq] [-global_override] [-startoption <start_options>] [-eval] [-verbose]
more....

1 ping

  1. […] Install Oracle Grid Infrastructure and Oracle Software: Step by Step […]

Leave a Reply