In this topic I will demonstrate how to configure compute node for Exadata and how to configure connection between Cell and Compute Nodes on VM.
Very few companies are using Exadata Engineering system so this is very difficult for a DBA to get an environment to explore. So Exadata simulation will be a good option to learn.
Configure Storage Server: Exadata Simulation Part I
Configure Compute Node or Database Node
Host Name and IP Details for Compute Node:
- Compute Node 1:Host Name: exadatadb01
- Public IP: 192.168.56.80
- Private IP: 192.168.2.80
- Compute Node 2:Host Name: exadatadb02
- Public IP: 192.168.56.90
- Private IP: 192.168.2.80
- Scan IP:
- 192.168.56.30
- 192.168.56.40
- 192.168.56.50
Prerequisites for Grid Infrastructure and Oracle Database
Changes Kernel Parameter:
Add below mentioned Kernel Parameter to /etc/sysctl.conf
vi /etc/sysctl.conf #Oracle fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default=4194304 net.core.rmem_max=4194304 net.core.wmem_default=4194304 net.core.wmem_max=4194304 kernel.core_uses_pid = 1 fs.suid_dumpable = 1 kernel.panic_on_oops = 1
Add below line to /etc/security/limits.conf
$ vi /etc/security/limits.conf # Oracle related parameters oracle soft nofile 131072 oracle hard nofile 131072 oracle soft nproc 131072 oracle hard nproc 131072 oracle soft core unlimited oracle hard core unlimited oracle soft memlock 3500000 oracle hard memlock 3500000 grid soft nofile 131072 grid hard nofile 131072 grid soft nproc 131072 grid hard nproc 131072 grid soft core unlimited grid hard core unlimited grid soft memlock 3500000 grid hard memlock 3500000 grid soft nproc 2047
Add below parameter to /etc/pam.d/login
$ /etc/pam.d/login session required pam_limits.so $
Change below parameter and these lines to /etc/profile.
[root@exadatadb01 ~]# cat /etc/profile | grep limit
ulimit -S -c 0 > /dev/null 2>&1
[root@exadatadb01 ~]# vi /etc/profile
After changed
[root@exadatadb01 ~]# cat /etc/profile | grep limit
ulimit -S -c unlimited > /dev/null 2>&1
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
umask 022
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fiNetwork Setting Check:
SELinux should be disabled or permissive
[root@exadatadb01 ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - SELinux is fully disabled. SELINUX=disabled # SELINUXTYPE= type of policy in use. Possible values are: # targeted - Only targeted network daemons are protected. # strict - Full SELinux protection.
Disable Firewall:
[root@exadatadb01 ~]# iptables -F [root@exadatadb01 ~]# service iptables stop Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ OK ] Unloading iptables modules: [ OK ] [root@exadatadb01 ~]# chkconfig iptables off
[root@exadatadb01 ~]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 192.168.56.80 exadatadb01.localdomain exadatadb01 192.168.56.90 exadatadb02.localdomain exadatadb02 192.168.2.80 exadatadb01-priv.localdomain exadatadb01-priv 192.168.2.90 exadatadb02-priv.localdomain exadatadb02-priv 192.168.56.10 exadatadb01-vip.localdomain exadatadb01-vip 192.168.56.20 exadatadb02-vip.localdomain exadatadb02-vip 192.168.56.30 exadatadb-scan.localdomain exadatadb-scan 192.168.56.40 exadatadb-scan.localdomain exadatadb-scan 192.168.56.50 exadatadb-scan.localdomain exadatadb-scan
Create User Group, User and Directory:
/usr/sbin/groupadd -g 503 oinstall /usr/sbin/groupadd -g 504 asmadmin /usr/sbin/groupadd -g 505 asmdba /usr/sbin/groupadd -g 506 asmoper /usr/sbin/groupadd -g 507 dba /usr/sbin/groupadd -g 508 oper
/usr/sbin/useradd -u 502 -g oinstall -G asmadmin,asmdba,asmoper grid /usr/sbin/useradd -u 501 -g oinstall -G dba,oper,asmdba,asmadmin,asmoper oracle
[root@exadatadb01 ~]# passwd oracle Changing password for user oracle. New UNIX password: BAD PASSWORD: it is based on a dictionary word Retype new UNIX password: passwd: all authentication tokens updated successfully. [root@exadatadb01 ~]# passwd grid Changing password for user grid. New UNIX password: BAD PASSWORD: it is too short Retype new UNIX password: passwd: all authentication tokens updated successfully.
mkdir -p /u01/app/grid mkdir -p /u01/app/11.2.0/grid chown grid:oinstall /u01/app/grid chown -R grid:oinstall /u01/app/11.2.0/grid chmod -R 775 /u01/app/11.2.0/grid mkdir -p /u01/app/oracle --needed to ensure that dbca is able to run after the rdbms installation. mkdir /u01/app/oracle/cfgtoollogs chown -R oracle:oinstall /u01/app/oracle/cfgtoollogs chown -R oracle:oinstall /u01/app/oracle chmod -R 775 /u01/app/oracle mkdir -p /u01/app/oraInventory chown -R oracle:oinstall /u01/app/oraInventory chmod -R 775 /u01/app/oraInventory mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1 chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/dbhome_1 chmod -R 775 /u01/app/oracle/product/11.2.0/dbhome_1 chmod -R 775 /u01/ chown -R grid:oinstall /u01
[root@exadatadb01 ~]# cd /etc [root@exadatadb01 etc]# mkdir oracle [root@exadatadb01 etc]# chmod 775 oracle [root@exadatadb01 etc]# chown grid:oinstall oracle
Set Environment Variable:
$ vi /home/oracle/.bash_profile
# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=exadatadb01; export ORACLE_HOSTNAME
ORACLE_UNQNAME=ORCL; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_SID=ORCL1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'
$ vim /home/oracle/grid_env ORACLE_SID=+ASM1; export ORACLE_SID ORACLE_HOME=$GRID_HOME; export ORACLE_HOME PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
$ vim /home/oracle/db_env ORACLE_SID=ORCL1; export ORACLE_SID ORACLE_HOME=$DB_HOME; export ORACLE_HOME PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
Install required packages for Grid Infrastructure and Oracle Database:
yum install net-snmp yum install sysstat yum install libaio-devel
The following command can be run on the system to list the currently installed packages:
rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' libaio-devel \
net-snmp \
sysstat
[root@exadatadb01 rpm]# pwd /u01/software/grid/rpm [root@exadatadb01 rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm Preparing... ########################################### [100%] 1:cvuqdisk ########################################### [100%]
[root@exadatadb01 ~]# yum install net-snmp Loaded plugins: rhnplugin, security This system is not registered with ULN. You can use up2date --register to register. ULN support will be disabled. Setting up Install Process Resolving Dependencies --> Running transaction check --> Processing Dependency: net-snmp = 1:5.3.2.2-20.0.2.el5 for package: net-snmp-perl --> Processing Dependency: net-snmp = 1:5.3.2.2-20.0.2.el5 for package: net-snmp-utils ---> Package net-snmp.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated --> Processing Dependency: net-snmp-libs = 1:5.3.2.2-25.0.2.el5_11 for package: net-snmp --> Running transaction check ---> Package net-snmp-libs.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated ---> Package net-snmp-perl.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated ---> Package net-snmp-utils.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================= Updating: net-snmp x86_64 1:5.3.2.2-25.0.2.el5_11 el5_latest 708 k Updating for dependencies: net-snmp-libs x86_64 1:5.3.2.2-25.0.2.el5_11 el5_latest 1.3 M net-snmp-perl x86_64 1:5.3.2.2-25.0.2.el5_11 el5_latest 203 k net-snmp-utils x86_64 1:5.3.2.2-25.0.2.el5_11 el5_latest 194 k Transaction Summary ======================================================================================================================================= Install 0 Package(s) Upgrade 4 Package(s) Total download size: 2.4 M Is this ok [y/N]: y Downloading Packages: (1/4): net-snmp-utils-5.3.2.2-25.0.2.el5_11.x86_64.rpm | 194 kB 00:00 (2/4): net-snmp-perl-5.3.2.2-25.0.2.el5_11.x86_64.rpm | 203 kB 00:00 (3/4): net-snmp-5.3.2.2-25.0.2.el5_11.x86_64.rpm | 708 kB 00:00 (4/4): net-snmp-libs-5.3.2.2-25.0.2.el5_11.x86_64.rpm | 1.3 MB 00:00 --------------------------------------------------------------------------------------------------------------------------------------- Total 358 kB/s | 2.4 MB 00:06 Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : net-snmp-libs 1/8 Updating : net-snmp 2/8 Updating : net-snmp-perl 3/8 Updating : net-snmp-utils 4/8 Cleanup : net-snmp 5/8 Cleanup : net-snmp-utils 6/8 Cleanup : net-snmp-perl 7/8 Cleanup : net-snmp-libs 8/8 Updated: net-snmp.x86_64 1:5.3.2.2-25.0.2.el5_11 Dependency Updated: net-snmp-libs.x86_64 1:5.3.2.2-25.0.2.el5_11 net-snmp-perl.x86_64 1:5.3.2.2-25.0.2.el5_11 net-snmp-utils.x86_64 1:5.3.2.2-25.0.2.el5_11 Complete!
Load RDS Kernel Modules:
[root@exadatadb01 ~]# modprobe rds;modprobe rds_tcp;modprobe rds_rdma
[root@exadatadb01 ~]# modprobe rds
[root@exadatadb01 ~]# modprobe rds_tcp
[root@exadatadb01 ~]# modprobe rds_rdma
# Need to add following line to /etc/modprobe.d/rds.conf
[root@exadatadb01 ~]# vi /etc/modprobe.d/rds.conf
install rds /sbin/modprobe --ignore-install rds && /sbin/modprobe rds_tcp && /sbin/modprobe rds_rdmaRemove the conflicting installation package if it already exists
Note::: Check point# 5.6 on Part I
[root@exadatadb01 ~]# lsmod |grep rds rds_rdma 82580 0 rds_tcp 10455 0 rds 96059 2 rds_rdma,rds_tcp rdma_cm 36793 2 rds_rdma,ib_iser ib_core 66577 7 rds_rdma,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
Configure Network File for connecting Cell Node or Storage Server.
[root@exadatadb01 etc]# su - grid [grid@exadatadb01 ~]$ cd /etc/oracle [grid@exadatadb01 oracle]$ mkdir -p cell/network-config [grid@exadatadb01 oracle]$ cd cell/network-config/ [grid@exadatadb01 network-config]$ pwd /etc/oracle/cell/network-config # Add Storage Server IP (Infiniband IP) [grid@exadatadb01 network-config]$ vi cellip.ora cell="192.168.2.60" cell="192.168.2.70" [grid@exadatadb01 network-config]$ cat cellip.ora cell="192.168.2.60" cell="192.168.2.70" #Create file and Add DB Server IP [grid@exadatadb01 network-config]$ vi cellinit.ora # Compute Node Public IP. [grid@exadatadb01 network-config]$ cat cellinit.ora ipaddress1=192.168.56.80/24 _cell_print_all_params=true _skgxp_gen_rpc_timeout_in_sec=90 _skgxp_gen_ant_off_rpc_timeout_in_sec=300 _skgxp_udp_interface_detection_time_secs=15 _skgxp_udp_use_tcb_client=true _skgxp_udp_use_tcb=false _reconnect_to_cell_attempts=5
# grid binary or software unzip location is /u01/software/grid.
[grid@exadatadb01 bin]$ export LD_LIBRARY_PATH=/u01/software/grid/stage/ext/lib
[grid@exadatadb01 bin]$ /u01/software/grid/stage/ext/bin/kfod disks=all op=disks
--------------------------------------------------------------------------------
Disk Size Path User Group
================================================================================
1: 720 Mb o/192.168.2.60/DATA_CD_DISK00_exadatacell01 < unknown> < unknown>
2: 720 Mb o/192.168.2.60/DATA_CD_DISK01_exadatacell01 < unknown> < unknown>
3: 720 Mb o/192.168.2.60/DATA_CD_DISK02_exadatacell01 < unknown> < unknown>
4: 720 Mb o/192.168.2.60/DATA_CD_DISK03_exadatacell01 < unknown> < unknown>
5: 720 Mb o/192.168.2.60/DATA_CD_DISK04_exadatacell01 < unknown> < unknown>
6: 720 Mb o/192.168.2.60/DATA_CD_DISK05_exadatacell01 < unknown> < unknown>
7: 720 Mb o/192.168.2.60/DATA_CD_DISK06_exadatacell01 < unknown> < unknown>
8: 720 Mb o/192.168.2.60/DATA_CD_DISK07_exadatacell01 < unknown> < unknown>
9: 720 Mb o/192.168.2.60/DATA_CD_DISK08_exadatacell01 < unknown> < unknown>
10: 720 Mb o/192.168.2.60/DATA_CD_DISK09_exadatacell01 < unknown> < unknown>
11: 176 Mb o/192.168.2.60/FRA_CD_DISK00_exadatacell01 < unknown> < unknown>
12: 176 Mb o/192.168.2.60/FRA_CD_DISK01_exadatacell01 < unknown> < unknown>
13: 176 Mb o/192.168.2.60/FRA_CD_DISK02_exadatacell01 < unknown> < unknown>
14: 176 Mb o/192.168.2.60/FRA_CD_DISK03_exadatacell01 < unknown> < unknown>
15: 176 Mb o/192.168.2.60/FRA_CD_DISK04_exadatacell01 < unknown> < unknown>
16: 176 Mb o/192.168.2.60/FRA_CD_DISK05_exadatacell01 < unknown> < unknown>
17: 176 Mb o/192.168.2.60/FRA_CD_DISK06_exadatacell01 < unknown> < unknown>
18: 176 Mb o/192.168.2.60/FRA_CD_DISK07_exadatacell01 < unknown> < unknown>
19: 176 Mb o/192.168.2.60/FRA_CD_DISK08_exadatacell01 < unknown> < unknown>
20: 176 Mb o/192.168.2.60/FRA_CD_DISK09_exadatacell01 < unknown> < unknown>
21: 720 Mb o/192.168.2.70/DATA_CD_DISK00_exadatacell02 < unknown> < unknown>
22: 720 Mb o/192.168.2.70/DATA_CD_DISK01_exadatacell02 < unknown> < unknown>
23: 720 Mb o/192.168.2.70/DATA_CD_DISK02_exadatacell02 < unknown> < unknown>
24: 720 Mb o/192.168.2.70/DATA_CD_DISK03_exadatacell02 < unknown> < unknown>
25: 720 Mb o/192.168.2.70/DATA_CD_DISK04_exadatacell02 < unknown> < unknown>
26: 720 Mb o/192.168.2.70/DATA_CD_DISK05_exadatacell02 < unknown> < unknown>
27: 720 Mb o/192.168.2.70/DATA_CD_DISK06_exadatacell02 < unknown> < unknown>
28: 720 Mb o/192.168.2.70/DATA_CD_DISK07_exadatacell02 < unknown> < unknown>
29: 720 Mb o/192.168.2.70/DATA_CD_DISK08_exadatacell02 < unknown> < unknown>
30: 720 Mb o/192.168.2.70/DATA_CD_DISK09_exadatacell02 < unknown> < unknown>
31: 80 Mb o/192.168.2.70/DBFS_CD_DISK00_exadatacell02 < unknown> < unknown>
32: 80 Mb o/192.168.2.70/DBFS_CD_DISK01_exadatacell02 < unknown> < unknown>
33: 80 Mb o/192.168.2.70/DBFS_CD_DISK02_exadatacell02 < unknown> < unknown>
34: 80 Mb o/192.168.2.70/DBFS_CD_DISK03_exadatacell02 < unknown> < unknown>
35: 80 Mb o/192.168.2.70/DBFS_CD_DISK04_exadatacell02 < unknown> < unknown>
36: 80 Mb o/192.168.2.70/DBFS_CD_DISK05_exadatacell02 < unknown> < unknown>
37: 80 Mb o/192.168.2.70/DBFS_CD_DISK06_exadatacell02 < unknown> < unknown>
38: 80 Mb o/192.168.2.70/DBFS_CD_DISK07_exadatacell02 < unknown> < unknown>
39: 80 Mb o/192.168.2.70/DBFS_CD_DISK08_exadatacell02 < unknown> < unknown>
40: 80 Mb o/192.168.2.70/DBFS_CD_DISK09_exadatacell02 < unknown> < unknown>
41: 176 Mb o/192.168.2.70/FRA_CD_DISK00_exadatacell02 < unknown> < unknown>
42: 176 Mb o/192.168.2.70/FRA_CD_DISK01_exadatacell02 < unknown> < unknown>
43: 176 Mb o/192.168.2.70/FRA_CD_DISK02_exadatacell02 < unknown> < unknown>
44: 176 Mb o/192.168.2.70/FRA_CD_DISK03_exadatacell02 < unknown> < unknown>
45: 176 Mb o/192.168.2.70/FRA_CD_DISK04_exadatacell02 < unknown> < unknown>
46: 176 Mb o/192.168.2.70/FRA_CD_DISK05_exadatacell02 < unknown> < unknown>
47: 176 Mb o/192.168.2.70/FRA_CD_DISK06_exadatacell02 < unknown> < unknown>
48: 176 Mb o/192.168.2.70/FRA_CD_DISK07_exadatacell02 < unknown> < unknown>
49: 176 Mb o/192.168.2.70/FRA_CD_DISK08_exadatacell02 < unknown> < unknown>
50: 176 Mb o/192.168.2.70/FRA_CD_DISK09_exadatacell02 < unknown> < unknown>Clone this VM for Compute Node 2. You may follow this link to clone: VM Clone
Note:::Two compute nodes are ready.
For RAC, Validate below requirement:
- Should be pingable both nodes using Host Name and IP
Change or Add below IP on Node 2:
[grid@exadatadb02 network-config]$ pwd /etc/oracle/cell/network-config [grid@exadatadb02 network-config]$ vi cellinit.ora # Compute Node Public IP. [grid@exadatadb02 network-config]$ cat cellinit.ora ipaddress1 = 192.168.56.90/24
Grid Infrastructure Installation on Compute Node Step by Step
Oracle Database Software Installation and Create Database on Compute Node Step by Step
Configure ASM DiskGroup on Exadata: ASM DiskGroup
Create Oracle RAC Database using DBCA: RAC DB
