Exadata Simulation (X3) on VM Part – II

In this topic I will demonstrate how to configure compute node for Exadata and how to configure connection between Cell and Compute Nodes on VM.

Very few companies are using Exadata Engineering system so this is very difficult for a DBA to get an environment to explore. So Exadata simulation will be a good option to learn.

Configure Storage Server: Exadata Simulation Part I

Configure Compute Node or Database Node

Host Name and IP Details for Compute Node:

  • Compute Node 1:Host Name: exadatadb01
    • Public IP: 192.168.56.80
    • Private IP: 192.168.2.80
  • Compute Node 2:Host Name: exadatadb02
    • Public IP: 192.168.56.90
    • Private IP: 192.168.2.80
  • Scan IP:
    • 192.168.56.30
    • 192.168.56.40
    • 192.168.56.50

Create a Virtual Machine

Install Oracle Linux on VM

Prerequisites for Grid Infrastructure and Oracle Database

Changes Kernel Parameter:

Add below mentioned Kernel Parameter to /etc/sysctl.conf

vi /etc/sysctl.conf
#Oracle
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=4194304
net.core.rmem_max=4194304
net.core.wmem_default=4194304
net.core.wmem_max=4194304
kernel.core_uses_pid = 1
fs.suid_dumpable = 1
kernel.panic_on_oops = 1

Add below line to /etc/security/limits.conf

$ vi /etc/security/limits.conf
# Oracle related parameters 
oracle  soft    nofile  131072
oracle  hard    nofile  131072
oracle  soft    nproc   131072
oracle  hard    nproc   131072
oracle  soft    core    unlimited
oracle  hard    core    unlimited
oracle  soft    memlock 3500000
oracle  hard    memlock 3500000
grid    soft    nofile  131072
grid    hard    nofile  131072
grid    soft    nproc   131072
grid    hard    nproc   131072
grid    soft    core    unlimited
grid    hard    core    unlimited
grid    soft    memlock 3500000
grid    hard    memlock 3500000
grid soft nproc 2047

Add below parameter to /etc/pam.d/login

$ /etc/pam.d/login
session required pam_limits.so
$

Change below parameter and these lines to /etc/profile.

[root@exadatadb01 ~]# cat /etc/profile | grep limit
ulimit -S -c 0 > /dev/null 2>&1
[root@exadatadb01 ~]# vi /etc/profile
After changed
[root@exadatadb01 ~]# cat /etc/profile | grep limit
ulimit -S -c unlimited > /dev/null 2>&1

if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
       umask 022
       if [ $SHELL = "/bin/ksh" ]; then
            ulimit -p 16384
            ulimit -n 65536
       else
            ulimit -u 16384 -n 65536
       fi
fi

Network Setting Check:

SELinux should be disabled or permissive

[root@exadatadb01 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing - SELinux security policy is enforced.
#       permissive - SELinux prints warnings instead of enforcing.
#       disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
#       targeted - Only targeted network daemons are protected.
#       strict - Full SELinux protection.

Disable Firewall:

[root@exadatadb01 ~]# iptables -F
[root@exadatadb01 ~]# service iptables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]
[root@exadatadb01 ~]# chkconfig iptables off
[root@exadatadb01 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.localdomain localhost
192.168.56.80   exadatadb01.localdomain    exadatadb01
192.168.56.90   exadatadb02.localdomain    exadatadb02

192.168.2.80   exadatadb01-priv.localdomain    exadatadb01-priv
192.168.2.90   exadatadb02-priv.localdomain    exadatadb02-priv

192.168.56.10   exadatadb01-vip.localdomain    exadatadb01-vip
192.168.56.20   exadatadb02-vip.localdomain    exadatadb02-vip

192.168.56.30   exadatadb-scan.localdomain    exadatadb-scan
192.168.56.40   exadatadb-scan.localdomain    exadatadb-scan
192.168.56.50   exadatadb-scan.localdomain    exadatadb-scan

Create User Group, User and Directory:

/usr/sbin/groupadd -g 503 oinstall
/usr/sbin/groupadd -g 504 asmadmin
/usr/sbin/groupadd -g 505 asmdba
/usr/sbin/groupadd -g 506 asmoper
/usr/sbin/groupadd -g 507 dba
/usr/sbin/groupadd -g 508 oper
/usr/sbin/useradd -u 502 -g oinstall -G asmadmin,asmdba,asmoper grid
/usr/sbin/useradd -u 501 -g oinstall -G dba,oper,asmdba,asmadmin,asmoper oracle
[root@exadatadb01 ~]# passwd oracle
Changing password for user oracle.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@exadatadb01 ~]# passwd grid
Changing password for user grid.
New UNIX password:
BAD PASSWORD: it is too short
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
mkdir -p /u01/app/grid    
mkdir -p /u01/app/11.2.0/grid

chown grid:oinstall /u01/app/grid
chown -R grid:oinstall /u01/app/11.2.0/grid
chmod -R 775 /u01/app/11.2.0/grid

mkdir -p /u01/app/oracle
--needed to ensure that dbca is able to run after the rdbms installation.
mkdir /u01/app/oracle/cfgtoollogs 
chown -R oracle:oinstall /u01/app/oracle/cfgtoollogs
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle

mkdir -p /u01/app/oraInventory
chown -R oracle:oinstall /u01/app/oraInventory
chmod -R 775 /u01/app/oraInventory

mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/dbhome_1
chmod -R 775 /u01/app/oracle/product/11.2.0/dbhome_1

chmod -R 775 /u01/
chown -R grid:oinstall /u01
[root@exadatadb01 ~]# cd /etc
[root@exadatadb01 etc]# mkdir oracle
[root@exadatadb01 etc]# chmod 775 oracle
[root@exadatadb01 etc]# chown grid:oinstall oracle

Set Environment Variable:

$ vi /home/oracle/.bash_profile

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=exadatadb01; export ORACLE_HOSTNAME
ORACLE_UNQNAME=ORCL; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_SID=ORCL1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'
$ vim /home/oracle/grid_env

ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
$ vim /home/oracle/db_env

ORACLE_SID=ORCL1; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

Install required packages for Grid Infrastructure and Oracle Database:

List of Packages

yum install net-snmp
yum install sysstat
yum install libaio-devel

The following command can be run on the system to list the currently installed packages:

rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' libaio-devel \
net-snmp \
sysstat
[root@exadatadb01 rpm]# pwd
/u01/software/grid/rpm
[root@exadatadb01 rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
   1:cvuqdisk               ########################################### [100%]
[root@exadatadb01 ~]# yum install net-snmp
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
You can use up2date --register to register.
ULN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
--> Processing Dependency: net-snmp = 1:5.3.2.2-20.0.2.el5 for package: net-snmp-perl
--> Processing Dependency: net-snmp = 1:5.3.2.2-20.0.2.el5 for package: net-snmp-utils
---> Package net-snmp.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated
--> Processing Dependency: net-snmp-libs = 1:5.3.2.2-25.0.2.el5_11 for package: net-snmp
--> Running transaction check
---> Package net-snmp-libs.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated
---> Package net-snmp-perl.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated
---> Package net-snmp-utils.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================
 Package                         Arch                    Version                                     Repository                   Size
=======================================================================================================================================
Updating:
 net-snmp                        x86_64                  1:5.3.2.2-25.0.2.el5_11                     el5_latest                  708 k
Updating for dependencies:
 net-snmp-libs                   x86_64                  1:5.3.2.2-25.0.2.el5_11                     el5_latest                  1.3 M
 net-snmp-perl                   x86_64                  1:5.3.2.2-25.0.2.el5_11                     el5_latest                  203 k
 net-snmp-utils                  x86_64                  1:5.3.2.2-25.0.2.el5_11                     el5_latest                  194 k

Transaction Summary
=======================================================================================================================================
Install       0 Package(s)
Upgrade       4 Package(s)

Total download size: 2.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): net-snmp-utils-5.3.2.2-25.0.2.el5_11.x86_64.rpm                                                          | 194 kB     00:00
(2/4): net-snmp-perl-5.3.2.2-25.0.2.el5_11.x86_64.rpm                                                           | 203 kB     00:00
(3/4): net-snmp-5.3.2.2-25.0.2.el5_11.x86_64.rpm                                                                | 708 kB     00:00
(4/4): net-snmp-libs-5.3.2.2-25.0.2.el5_11.x86_64.rpm                                                           | 1.3 MB     00:00
---------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                  358 kB/s | 2.4 MB     00:06
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : net-snmp-libs                                                                                                   1/8
  Updating       : net-snmp                                                                                                        2/8
  Updating       : net-snmp-perl                                                                                                   3/8
  Updating       : net-snmp-utils                                                                                                  4/8
  Cleanup        : net-snmp                                                                                                        5/8
  Cleanup        : net-snmp-utils                                                                                                  6/8
  Cleanup        : net-snmp-perl                                                                                                   7/8
  Cleanup        : net-snmp-libs                                                                                                   8/8

Updated:
  net-snmp.x86_64 1:5.3.2.2-25.0.2.el5_11

Dependency Updated:
  net-snmp-libs.x86_64 1:5.3.2.2-25.0.2.el5_11                       net-snmp-perl.x86_64 1:5.3.2.2-25.0.2.el5_11
  net-snmp-utils.x86_64 1:5.3.2.2-25.0.2.el5_11

Complete!

Load RDS Kernel Modules:

[root@exadatadb01 ~]# modprobe rds;modprobe rds_tcp;modprobe rds_rdma
[root@exadatadb01 ~]# modprobe rds
[root@exadatadb01 ~]# modprobe rds_tcp
[root@exadatadb01 ~]# modprobe rds_rdma
# Need to add following line to /etc/modprobe.d/rds.conf 
[root@exadatadb01 ~]# vi /etc/modprobe.d/rds.conf
install rds /sbin/modprobe --ignore-install rds && /sbin/modprobe rds_tcp && /sbin/modprobe rds_rdma

Remove the conflicting installation package if it already exists

Note::: Check point# 5.6 on Part I

[root@exadatadb01 ~]# lsmod |grep rds
rds_rdma               82580  0
rds_tcp                10455  0
rds                    96059  2 rds_rdma,rds_tcp
rdma_cm                36793  2 rds_rdma,ib_iser
ib_core                66577  7 rds_rdma,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad

Configure Network File for connecting Cell Node or Storage Server.

[root@exadatadb01 etc]# su - grid
[grid@exadatadb01 ~]$ cd /etc/oracle
[grid@exadatadb01 oracle]$ mkdir -p cell/network-config
[grid@exadatadb01 oracle]$ cd cell/network-config/
[grid@exadatadb01 network-config]$ pwd
/etc/oracle/cell/network-config

# Add Storage Server IP (Infiniband IP) 
[grid@exadatadb01 network-config]$ vi cellip.ora
cell="192.168.2.60"
cell="192.168.2.70"

[grid@exadatadb01 network-config]$ cat cellip.ora
cell="192.168.2.60"
cell="192.168.2.70"

#Create file and Add DB Server IP
[grid@exadatadb01 network-config]$ vi cellinit.ora
# Compute Node Public IP. 
[grid@exadatadb01 network-config]$ cat cellinit.ora
ipaddress1=192.168.56.80/24
_cell_print_all_params=true
_skgxp_gen_rpc_timeout_in_sec=90
_skgxp_gen_ant_off_rpc_timeout_in_sec=300
_skgxp_udp_interface_detection_time_secs=15
_skgxp_udp_use_tcb_client=true
_skgxp_udp_use_tcb=false
_reconnect_to_cell_attempts=5
# grid binary or software unzip location is /u01/software/grid.
[grid@exadatadb01 bin]$ export LD_LIBRARY_PATH=/u01/software/grid/stage/ext/lib
[grid@exadatadb01 bin]$ /u01/software/grid/stage/ext/bin/kfod disks=all op=disks
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:        720 Mb o/192.168.2.60/DATA_CD_DISK00_exadatacell01 < unknown> < unknown>
   2:        720 Mb o/192.168.2.60/DATA_CD_DISK01_exadatacell01 < unknown> < unknown>
   3:        720 Mb o/192.168.2.60/DATA_CD_DISK02_exadatacell01 < unknown> < unknown>
   4:        720 Mb o/192.168.2.60/DATA_CD_DISK03_exadatacell01 < unknown> < unknown>
   5:        720 Mb o/192.168.2.60/DATA_CD_DISK04_exadatacell01 < unknown> < unknown>
   6:        720 Mb o/192.168.2.60/DATA_CD_DISK05_exadatacell01 < unknown> < unknown>
   7:        720 Mb o/192.168.2.60/DATA_CD_DISK06_exadatacell01 < unknown> < unknown>
   8:        720 Mb o/192.168.2.60/DATA_CD_DISK07_exadatacell01 < unknown> < unknown>
   9:        720 Mb o/192.168.2.60/DATA_CD_DISK08_exadatacell01 < unknown> < unknown>
  10:        720 Mb o/192.168.2.60/DATA_CD_DISK09_exadatacell01 < unknown> < unknown>
  11:        176 Mb o/192.168.2.60/FRA_CD_DISK00_exadatacell01 < unknown> < unknown>
  12:        176 Mb o/192.168.2.60/FRA_CD_DISK01_exadatacell01 < unknown> < unknown>
  13:        176 Mb o/192.168.2.60/FRA_CD_DISK02_exadatacell01 < unknown> < unknown>
  14:        176 Mb o/192.168.2.60/FRA_CD_DISK03_exadatacell01 < unknown> < unknown>
  15:        176 Mb o/192.168.2.60/FRA_CD_DISK04_exadatacell01 < unknown> < unknown>
  16:        176 Mb o/192.168.2.60/FRA_CD_DISK05_exadatacell01 < unknown> < unknown>
  17:        176 Mb o/192.168.2.60/FRA_CD_DISK06_exadatacell01 < unknown> < unknown>
  18:        176 Mb o/192.168.2.60/FRA_CD_DISK07_exadatacell01 < unknown> < unknown>
  19:        176 Mb o/192.168.2.60/FRA_CD_DISK08_exadatacell01 < unknown> < unknown>
  20:        176 Mb o/192.168.2.60/FRA_CD_DISK09_exadatacell01 < unknown> < unknown>
  21:        720 Mb o/192.168.2.70/DATA_CD_DISK00_exadatacell02 < unknown> < unknown>
  22:        720 Mb o/192.168.2.70/DATA_CD_DISK01_exadatacell02 < unknown> < unknown>
  23:        720 Mb o/192.168.2.70/DATA_CD_DISK02_exadatacell02 < unknown> < unknown>
  24:        720 Mb o/192.168.2.70/DATA_CD_DISK03_exadatacell02 < unknown> < unknown>
  25:        720 Mb o/192.168.2.70/DATA_CD_DISK04_exadatacell02 < unknown> < unknown>
  26:        720 Mb o/192.168.2.70/DATA_CD_DISK05_exadatacell02 < unknown> < unknown>
  27:        720 Mb o/192.168.2.70/DATA_CD_DISK06_exadatacell02 < unknown> < unknown>
  28:        720 Mb o/192.168.2.70/DATA_CD_DISK07_exadatacell02 < unknown> < unknown>
  29:        720 Mb o/192.168.2.70/DATA_CD_DISK08_exadatacell02 < unknown> < unknown>
  30:        720 Mb o/192.168.2.70/DATA_CD_DISK09_exadatacell02 < unknown> < unknown>
  31:         80 Mb o/192.168.2.70/DBFS_CD_DISK00_exadatacell02 < unknown> < unknown>
  32:         80 Mb o/192.168.2.70/DBFS_CD_DISK01_exadatacell02 < unknown> < unknown>
  33:         80 Mb o/192.168.2.70/DBFS_CD_DISK02_exadatacell02 < unknown> < unknown>
  34:         80 Mb o/192.168.2.70/DBFS_CD_DISK03_exadatacell02 < unknown> < unknown>
  35:         80 Mb o/192.168.2.70/DBFS_CD_DISK04_exadatacell02 < unknown> < unknown>
  36:         80 Mb o/192.168.2.70/DBFS_CD_DISK05_exadatacell02 < unknown> < unknown>
  37:         80 Mb o/192.168.2.70/DBFS_CD_DISK06_exadatacell02 < unknown> < unknown>
  38:         80 Mb o/192.168.2.70/DBFS_CD_DISK07_exadatacell02 < unknown> < unknown>
  39:         80 Mb o/192.168.2.70/DBFS_CD_DISK08_exadatacell02 < unknown> < unknown>
  40:         80 Mb o/192.168.2.70/DBFS_CD_DISK09_exadatacell02 < unknown> < unknown>
  41:        176 Mb o/192.168.2.70/FRA_CD_DISK00_exadatacell02 < unknown> < unknown>
  42:        176 Mb o/192.168.2.70/FRA_CD_DISK01_exadatacell02 < unknown> < unknown>
  43:        176 Mb o/192.168.2.70/FRA_CD_DISK02_exadatacell02 < unknown> < unknown>
  44:        176 Mb o/192.168.2.70/FRA_CD_DISK03_exadatacell02 < unknown> < unknown>
  45:        176 Mb o/192.168.2.70/FRA_CD_DISK04_exadatacell02 < unknown> < unknown>
  46:        176 Mb o/192.168.2.70/FRA_CD_DISK05_exadatacell02 < unknown> < unknown>
  47:        176 Mb o/192.168.2.70/FRA_CD_DISK06_exadatacell02 < unknown> < unknown>
  48:        176 Mb o/192.168.2.70/FRA_CD_DISK07_exadatacell02 < unknown> < unknown>
  49:        176 Mb o/192.168.2.70/FRA_CD_DISK08_exadatacell02 < unknown> < unknown>
  50:        176 Mb o/192.168.2.70/FRA_CD_DISK09_exadatacell02 < unknown> < unknown>

Clone this VM for Compute Node 2. You may follow this link to clone: VM Clone

Note:::Two compute nodes are ready.

For RAC, Validate below requirement:

  • Should be pingable both nodes using Host Name and IP

Change or Add below IP on Node 2:

[grid@exadatadb02 network-config]$ pwd
/etc/oracle/cell/network-config
[grid@exadatadb02 network-config]$ vi cellinit.ora
# Compute Node Public IP. 
[grid@exadatadb02 network-config]$ cat cellinit.ora
ipaddress1 = 192.168.56.90/24

Grid Infrastructure Installation on Compute Node Step by Step

Oracle Database Software Installation and Create Database on Compute Node Step by Step

Configure ASM DiskGroup on Exadata: ASM DiskGroup

Create Oracle RAC Database using DBCA: RAC DB

Leave a Reply