Exadata Simulation on VM (Cell 12.1.1.1.0)

Create a Virtual Machine for Exadata CELL or Storage Server

In this topic will demonstrate how to configure Cell or Storage Server for Exadata. Very few companies are using Exadata Engineering system so this is very difficult for a DBA to get an environment to explore. So Exadata simulation will be a good option to learn.

• Download Required Software
• Hardware Requirements
• Create a Virtual Machine
• Install Oracle Linux
• Prerequisite for Exadata Storage Server
• Install the Exadata Storage Server Software
• Clone Storage Server for 2nd VM of Exadata Storage Server (exadatacell02)
• Create Cell disks and Flash Cache storage for Exadata Storage Server on Node 1
• Create Cell disks and Flash Cache storage for Exadata Storage Server on Node 2
• Install Oracle Grid and Database

Oracle Doc: Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)

Software Version:

• Oracle Storage Server Software: 12.1.1.1.0
• Oracle Linux 6.8
• Oracle GI and Database: 12.1.0.0.0
• JDK 1.7

Step 1: Download Required Software:

• Download Oracle VM and Install: VM Box
• Download Exadata Storage Server Software form Oracle: Download
• Download Oracle Linux: Oracle Linux

1.1. Download Oracle VM and Install: VM Box

1.2 Download Exadata Storage Server Software form Oracle: Download

1.3. Download Oracle Linux: Oracle Linux

Step 2: Hardware Requirements:

• Virtual Machine: At least 2 Storage Server and two Compute Nodes on VM
• Memory: At least 4 GB memory is required for Storage Server and 2-3 GB for DB Server. So for 4 VMs, need around 12-14 GB memory
• Storage: Storage Server -> 40 GB x 2 = 80 GB and Compute Node: 40×2 GB = 80 GB. Total around 160 GB

Environment:

• Storage Server:
  • CELL Node 1:
    • Host Name: exadatacell01
    • Public IP: 192.168.56.60
    • Private IP: 192.168.2.60
  • CELL Node 2:
    • Host Name: exadatacell02
    • Public IP: 192.168.56.70
    • Private IP: 192.168.2.70
• Compute Node:
  • Compute Node 1:
    • Host Name: exadatadb01
    • Public IP: 192.168.56.80
    • Private IP: 192.168.2.80
  • Compute Node 2:
    • Host Name: exadatadb02
    • Public IP: 192.168.56.90
    • Private IP: 192.168.2.80
    • Scan IP:
      • 192.168.56.30
      • 192.168.56.40
      • 192.168.56.50

Step 3: Create a Virtual Machine: exadatacell01

How to create a VM: Create a Virtual Machine

Step 4: Install Oracle Linux on VM

Step 5: Prerequisite for Exadata Storage Server:

5.1. Add IP and Hostname on /etc/hosts file.

[root@exadatacell01 ~]# cat /etc/hosts
127.0.0.1   localhost.localdomain localhost
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.60 exadatacell01.localdomain exadatacell01
192.168.56.70 exadatacell02.localdomain exadatacell02
192.168.2.60 exadatacell01-ib.localdomain exadatacell01-ib
192.168.2.70 exadatacell02-ib.localdomain exadatacell02-ib

[root@exadatacell01 ~]# ping exadatacell01
PING exadatacell01.localdomain (192.168.56.60) 56(84) bytes of data.
64 bytes from exadatacell01.localdomain (192.168.56.60): icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from exadatacell01.localdomain (192.168.56.60): icmp_seq=2 ttl=64 time=0.035 ms

5.2. Set Kernel Parameters:

[root@exadatacell01 ~]# vi /etc/sysctl.conf

##### Exadata################
fs.file-max = 655360
fs.aio-max-nr=50000000
net.core.rmem_default=4194304
net.core.rmem_max=4194304
net.core.wmem_default=4194304
net.core.wmem_max=4194304
##### Exadata##################

[root@exadatacell01 media]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max = 65536
fs.aio-max-nr = 50000000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 4194304
net.core.wmem_max = 4194304

[root@exadatacell01 ~]# vi /etc/security/limits.conf
* soft nofile 655360
* hard nofile 655360
#End of file

[root@exadatacell01 media]# cat  /etc/grub.conf | grep default
default=0
#Change value for default to 1
[root@exadatacell01 media]# vi  /etc/grub.conf
[root@exadatacell01 media]# cat  /etc/grub.conf | grep default
default=1

#Add below line:
[root@exadatacell01 media]# vi /etc/bashrc
export DISPLAY=:0
[root@exadatacell01 media]# cat /etc/bashrc | grep DISPLAY
export DISPLAY=:0

5.3. Disable Firewall and SELinux

[root@exadatacell01 ~]# chkconfig ip6tables off
[root@exadatacell01 ~]# service ip6tables stop
ip6tables: Setting chains to policy ACCEPT: filter         [  OK  ]
ip6tables: Flushing firewall rules:                        [  OK  ]
ip6tables: Unloading modules:                              [  OK  ]
[root@exadatacell01 ~]# chkconfig iptables off
[root@exadatacell01 ~]# service iptables stop
[root@exadatacell01 ~]# cat /etc/selinux/config

5.4 Create Directories:

[root@exadatacell01 ~]# mkdir /var/log/oracle
[root@exadatacell01 ~]# chmod 775 /var/log/oracle
[root@exadatacell01 ~]# cd /
[root@exadatacell01 /]# mkdir storage_server_software

5.5. Load RDS Kernel Modules:

[root@exadatacell01 /]# modprobe rds
[root@exadatacell01 /]# modprobe rds_tcp
[root@exadatacell01 /]# modprobe rds_rdma
#Add below line to /etc/modprobe.d/rds.conf
[root@exadatacell01 /]# cat /etc/modprobe.d/rds.conf
install rds /sbin/modprobe --ignore-install rds && /sbin/modprobe rds_tcp && /sbin/modprobe rds_rdma

5.6. Remove the conflicting installation package if it already exists

[root@exadatacell01 /]# rpm -qa | egrep 'rds|rdma'
rdma-6.8_4.1-1.el6.noarch
words-3.0-17.el6.noarch
[root@exadatacell01 /]# rpm -e rdma
[root@exadatacell01 /]# rpm -qa | egrep 'rds|rdma'
words-3.0-17.el6.noarch

Step 6. Install the Exadata Storage Server Software

6.1. Copy software from your local disk to VM and unzip the software. You may use winscp software to copy on VM.

[root@exadatacell01 storage_server_software]# pwd
/storage_server_software
[root@exadatacell01 storage_server_software]# ls -l
-rw-r--r-- 1 root root 1849304714 Jul  5 05:13 V42777-01.zip
[root@exadatacell01 storage_server_software]# unzip V42777-01.zip
Archive:  V42777-01.zip
  inflating: README.txt
  inflating: cellImageMaker_12.1.1.1.0_LINUX.X64_131219-1.x86_64.tar

[root@exadatacell01 storage_server_software]# tar -xvf cellImageMaker_12.1.1.1.0_LINUX.X64_131219-1.x86_64.tar
...
dl180/initrd/lib/udev/edd_id
dl180/initrd/lib/libpthread-2.5.so
dl180/initrd/lib/libgcc_s-4.1.2-20080825.so.1
dl180/initrd/lib/libpthread.so.0
dl180/initrd/lib/libc-2.5.so

6.2. Unzip cell.bin

[root@exadatacell01 cellbits]# pwd
/storage_server_software/dl180/boot/cellbits

[root@exadatacell01 cellbits]# unzip cell.bin
Archive:  cell.bin
warning [cell.bin]:  20118 extra bytes at beginning or within zipfile
  (attempting to process anyway)
  inflating: cell-12.1.1.1.0_LINUX.X64_131219-1.x86_64.rpm

[root@exadatacell01 cellbits]# pwd
/storage_server_software/dl180/boot/cellbits
[root@exadatacell01 cellbits]# ls -l
total 1818352
-rw-rw-r-- 1 root root 321067509 Dec 19  2013 cell-12.1.1.1.0_LINUX.X64_131219-1.x86_64.rpm
-rwxrwxr-x 1 root root 310174845 Dec 19  2013 cell.bin
-rw-rw-r-- 1 root root  12754010 Dec 19  2013 cellboot.tbz
-rw-rw-r-- 1 root root  99170935 Dec 19  2013 cellfw.tbz
-rw-rw-r-- 1 root root 188786646 Dec 19  2013 cellrpms.tbz
-rw-rw-r-- 1 root root 159338722 Dec 19  2013 commonos.tbz
-rw-rw-r-- 1 root root 473690391 Dec 19  2013 debugos.tbz
-rw-rw-r-- 1 root root  74258331 Dec 19  2013 doclib.zip
-rw-rw-r-- 1 root root 126400896 Dec 19  2013 exaos.tbz
-rw-rw-r-- 1 root root  11663378 Dec 19  2013 hputils.tbz
-rw-rw-r-- 1 root root  66532883 Dec 19  2013 kernel.tbz
-rw-rw-r-- 1 root root   4799939 Dec 19  2013 ofed.tbz
-rw-rw-r-- 1 root root  13304574 Dec 19  2013 sunutils.tbz

[root@exadatacell01 cellbits]# pwd
/storage_server_software/cellbits
[root@exadatacell01 cellbits]# mkdir cellrpms
[root@exadatacell01 cellbits]# tar -xjf cellrpms.tbz -C cellrpms

[root@exadatacell01 cellbits]# pwd
/storage_server_software/cellbits
[root@exadatacell01 cellbits]# mkdir commonos
[root@exadatacell01 cellbits]# tar -xjf commonos.tbz -C commonos

6.3. Install JDK

[root@exadatacell01 cellrpms]# pwd
/storage_server_software/dl180/boot/cellbits/cellrpms
[root@exadatacell01 cellrpms]# ls -l
.....
-r--r--r-- 1 root root 85414830 Dec 19  2013 jdk-1.7.0_25-fcs.x86_64.rpm
....

[root@exadatacell01 cellrpms]# rpm -ivh jdk-1.7.0_25-fcs.x86_64.rpm
warning: jdk-1.7.0_25-fcs.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 1e5e0159: NOKEY
Preparing...                ########################################### [100%]
   1:jdk                    warning: /etc/init.d/jexec saved as /etc/init.d/jexec.rpmorig
########################################### [100%]
Unpacking JAR files...
        rt.jar...
        jsse.jar...
        charsets.jar...
        tools.jar...
        localedata.jar...

[root@localhost cellbits]# rpm -ivh cell-12.2.1.1.1_LINUX.X64_170419-1.x86_64.rpm
error: Failed dependencies:
        net-snmp is needed by cell-12.2.1.1.1_LINUX.X64_170419-1.x86_64
        net-snmp-utils is needed by cell-12.2.1.1.1_LINUX.X64_170419-1.x86_64
        perl(Env) is needed by cell-12.2.1.1.1_LINUX.X64_170419-1.x86_64
        perl(Time::HiRes) is needed by cell-12.2.1.1.1_LINUX.X64_170419-1.x86_64
        perl(XML::Simple) is needed by cell-12.2.1.1.1_LINUX.X64_170419-1.x86_64

6.4. Install required packages for CELL rpm:

If you have internet connection on VM then you will able to run below command. Otherwise you need to configure yum server manually.

[root@exadatacell01 ~]# yum install net-snmp
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
You can use up2date --register to register.
ULN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
--> Processing Dependency: net-snmp = 1:5.3.2.2-20.0.2.el5 for package: net-snmp-perl
--> Processing Dependency: net-snmp = 1:5.3.2.2-20.0.2.el5 for package: net-snmp-utils
---> Package net-snmp.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated
--> Processing Dependency: net-snmp-libs = 1:5.3.2.2-25.0.2.el5_11 for package: net-snmp
--> Running transaction check
---> Package net-snmp-libs.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated
---> Package net-snmp-perl.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated
---> Package net-snmp-utils.x86_64 1:5.3.2.2-25.0.2.el5_11 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================
 Package                        Arch                   Version                                    Repository                  Size
===================================================================================================================================
Updating:
 net-snmp                       x86_64                 1:5.3.2.2-25.0.2.el5_11                    el5_latest                 708 k
Updating for dependencies:
 net-snmp-libs                  x86_64                 1:5.3.2.2-25.0.2.el5_11                    el5_latest                 1.3 M
 net-snmp-perl                  x86_64                 1:5.3.2.2-25.0.2.el5_11                    el5_latest                 203 k
 net-snmp-utils                 x86_64                 1:5.3.2.2-25.0.2.el5_11                    el5_latest                 194 k

Transaction Summary
===================================================================================================================================
Install       0 Package(s)
Upgrade       4 Package(s)

Total download size: 2.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): net-snmp-utils-5.3.2.2-25.0.2.el5_11.x86_64.rpm                                                      | 194 kB     00:00
(2/4): net-snmp-perl-5.3.2.2-25.0.2.el5_11.x86_64.rpm                                                       | 203 kB     00:00
(3/4): net-snmp-5.3.2.2-25.0.2.el5_11.x86_64.rpm                                                            | 708 kB     00:00
(4/4): net-snmp-libs-5.3.2.2-25.0.2.el5_11.x86_64.rpm                                                       | 1.3 MB     00:00
-----------------------------------------------------------------------------------------------------------------------------------
Total                                                                                              335 kB/s | 2.4 MB     00:07
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 1e5e0159
el5_latest/gpgkey                                                                                           | 1.4 kB     00:00
Importing GPG key 0x1E5E0159 "Oracle OSS group (Open Source Software group) <[email protected]>" from http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : net-snmp-libs                                                                                               1/8
  Updating       : net-snmp                                                                                                    2/8
  Updating       : net-snmp-perl                                                                                               3/8
  Updating       : net-snmp-utils                                                                                              4/8
  Cleanup        : net-snmp                                                                                                    5/8
  Cleanup        : net-snmp-utils                                                                                              6/8
  Cleanup        : net-snmp-perl                                                                                               7/8
  Cleanup        : net-snmp-libs                                                                                               8/8

Updated:
  net-snmp.x86_64 1:5.3.2.2-25.0.2.el5_11

Dependency Updated:
  net-snmp-libs.x86_64 1:5.3.2.2-25.0.2.el5_11                     net-snmp-perl.x86_64 1:5.3.2.2-25.0.2.el5_11
  net-snmp-utils.x86_64 1:5.3.2.2-25.0.2.el5_11

Complete!

Note::: Node one is ready to install Exadata Storage Software. As we are planing to configure two Nodes for Storage, So Better to make clone of Node 1 for Node 2 then don’t need to prepare Node 2 again.

Step 7. Clone Storage Server for 2nd VM of Exadata Storage Server (exadatacell02)

7.1. Clone the Server of (exadatacell01)

7.2. Change the hostname

7.3. Change the IP for 3 Network Interface

How to Clone VM

Step 6. Install the Exadata Storage Server Software (Continue…)

6.5. Set environment variable for correct version of Java

[root@exadatacell01 jdk1.7.0_25]# export JAVA_HOME=/usr/java/jdk1.7.0_25
[root@exadatacell01 jdk1.7.0_25]# export PATH=$JAVA_HOME/bin:$PATH
[root@exadatacell01 jdk1.7.0_25]# java -version
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

6.6. Install Exadata Storage Server RPM

[root@exadatacell01 cellbits]# pwd
/storage_server_software/dl180/boot/cellbits
[root@exadatacell01 cellbits]# ls
cell-12.1.1.1.0_LINUX.X64_131219-1.x86_64.rpm  cellfw.tbz    commonos      doclib.zip   kernel.tbz
cell.bin                                       cellrpms      commonos.tbz  exaos.tbz    ofed.tbz
cellboot.tbz                                   cellrpms.tbz  debugos.tbz   hputils.tbz  sunutils.tbz
[root@exadatacell01 cellbits]# rpm -ivh cell-12.1.1.1.0_LINUX.X64_131219-1.x86_64.rpm
Preparing...                ########################################### [100%]
Pre Installation steps in progress ...
   1:cell                   ########################################### [100%]
Post Installation steps in progress ...
Set cellusers group for /opt/oracle/cell12.1.1.1.0_LINUX.X64_131219/cellsrv/deploy/log directory
Set 775 permissions for /opt/oracle/cell12.1.1.1.0_LINUX.X64_131219/cellsrv/deploy/log directory
/opt/oracle/cell12.1.1.1.0_LINUX.X64_131219/cellsrv/deploy
/opt/oracle/cell12.1.1.1.0_LINUX.X64_131219/cellsrv/deploy
/opt/oracle/cell12.1.1.1.0_LINUX.X64_131219
Installation SUCCESSFUL.
Starting RS and MS... as user celladmin
Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration.
If this is a manual installation, please stop and restart ExaWatcher to pick up newly installed binaries.
You can run "/opt/oracle.ExaWatcher/ExaWatcher.sh --stop" and then "/opt/oracle.ExaWatcher/ExaWatcher.sh --fromconf" to stop and restart ExaWatcher.
Logout and then re-login to use the new cell environment.

6.7. Validation

[root@exadatacell01 cellbits]# su - celladmin
[celladmin@exadatacell01 ~]$
[celladmin@exadatacell01 ~]$
[celladmin@exadatacell01 ~]$ cellcli
CellCLI: Release 12.1.1.1.0 - Production on Tue Jul 07 14:27:03 EDT 2020

Copyright (c) 2007, 2013, Oracle.  All rights reserved.
Cell Efficiency Ratio: 1

CellCLI> alter cell restart services all

Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services...  running
Starting CELLSRV services...
The STARTUP of CELLSRV services was not successful.
CELL-01553: Incorrect IP in cellinit.ora. IP is invalid or has incorrect/missing netmask. Please refer to RS incident log.
Starting MS services...
The STARTUP of MS services was successful.

CELL-01533: Incorrect IP in cellinit.ora. IP is invalid or has incorrect/missing netmask: This error is generating because of missing the ipaddress1 parameter on cellint.ora file.

[celladmin@exadatacell01 config]$ pwd
/opt/oracle/cell12.1.1.1.0_LINUX.X64_131219/cellsrv/deploy/config
[celladmin@exadatacell01 config]$ cat cellinit.ora
#CELL Initialization Parameters
version=0.0
DEPLOYED=TRUE
HTTP_PORT=8888
RMI_PORT=23791
SSL_PORT=23943
JMS_PORT=9127
BMC_SNMP_PORT=162

[celladmin@exadatacell01 config]$ cat cellinit.ora
#CELL Initialization Parameters
version=0.0
DEPLOYED=TRUE
HTTP_PORT=8888
RMI_PORT=23791
SSL_PORT=23943
JMS_PORT=9127
BMC_SNMP_PORT=162
ipaddress1=192.168.2.60/24

6.7.1. RS, CELLSRV and MS Services have been started successfully

[celladmin@exadatacell01 config]$ cellcli
CellCLI: Release 12.1.1.1.0 - Production on Tue Jul 07 14:30:19 EDT 2020

Copyright (c) 2007, 2013, Oracle.  All rights reserved.
Cell Efficiency Ratio: 1

CellCLI> alter cell restart services all

Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services...  running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.

Step 8: Create Cell disks and Flash Cache storage for Exadata Storage Server on Node 1

• Create 10 disks of 1024 mb – these are for the Cell disks
• Create 4 disks of 600 mb – these are for the Flash disks

8.1. Add 10 storage for cell disks

VBoxManage createhd --filename "C:\VM\exadatacell01\hd1.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd2.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd3.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd4.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd5.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd6.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd7.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd8.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd9.vdi" --size 1024 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd10.vdi" --size 1024 --format VDI --variant Fixed

8.2. Add 4 storage for Flash Disks

VBoxManage createhd --filename "C:\VM\exadatacell01\hd11.vdi" --size 600 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd12.vdi" --size 600 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd13.vdi" --size 600 --format VDI --variant Fixed
VBoxManage createhd --filename "C:\VM\exadatacell01\hd14.vdi" --size 600 --format VDI --variant Fixed

8.3. Run these command using cmd command prompt

C:\Users\samad>cd "c:\Program Files\Oracle\VirtualBox"

c:\Program Files\Oracle\VirtualBox>VBoxManage createhd --filename "C:\VM\exadatacell01\hd1.vdi" --size 1024 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: 5eac0693-c88f-4b7a-81ef-e5f128b019bb

c:\Program Files\Oracle\VirtualBox>VBoxManage createhd --filename "C:\VM\exadatacell01\hd2.vdi" --size 1024 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: 2d631e61-b514-4806-8dab-ed2ca2811af8
....

8.4. VM Machine

• VM Name: exadatacell01
• Path of VM C:\VM\exadatacell01\

8.4.1. For Cell Disk

Note::: VM Name and Path may need to change for your case for below script.

VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 1 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd1.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 2 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd2.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 3 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd3.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 4 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd4.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 5 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd5.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 6 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd6.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 7 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd7.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 8 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd8.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 9 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd9.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 10 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd10.vdi" --mtype normal

8.4.2. For Flash Disk

VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 11 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd11.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 12 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd12.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 13 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd13.vdi" --mtype normal
VBoxManage storageattach exadatacell01 --storagectl "SATA" --port 14 --device 0 --type hdd --medium "C:\VM\exadatacell01\hd14.vdi" --mtype normal

8.5. Create directories for the Storage Server physical disks:

[root@exadatacell01 cellbits]# cd /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109
[root@exadatacell01 cell12.1.1.1.0_LINUX.X64_130109]# mkdir -p disks/raw
[root@exadatacell01 cell12.1.1.1.0_LINUX.X64_130109]# cd disks/raw
[root@exadatacell01 raw]# pwd
/opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw

8.6. Create Symbolic Disks/Links:

[root@exadatacell01 raw]# pwd
/opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw
[root@exadatacell01 raw]# fdisk -l 2>/dev/null | grep 'MB'
Disk /dev/sdb: 1073 MB, 1073741824 bytes
Disk /dev/sdc: 1073 MB, 1073741824 bytes
Disk /dev/sdd: 1073 MB, 1073741824 bytes
Disk /dev/sde: 1073 MB, 1073741824 bytes
Disk /dev/sdf: 1073 MB, 1073741824 bytes
Disk /dev/sdg: 1073 MB, 1073741824 bytes
Disk /dev/sdh: 1073 MB, 1073741824 bytes
Disk /dev/sdi: 1073 MB, 1073741824 bytes
Disk /dev/sdj: 1073 MB, 1073741824 bytes
Disk /dev/sdk: 1073 MB, 1073741824 bytes
Disk /dev/sdl: 629 MB, 629145600 bytes
Disk /dev/sdm: 629 MB, 629145600 bytes
Disk /dev/sdn: 629 MB, 629145600 bytes
Disk /dev/sdo: 629 MB, 629145600 bytes

[root@exadatacell01 raw]# pwd
/opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw

[root@exadatacell01 raw]# ln -s /dev/sdb exadatacell01_DISK00
[root@exadatacell01 raw]# ln -s /dev/sdc exadatacell01_DISK01
[root@exadatacell01 raw]# ln -s /dev/sdd exadatacell01_DISK02
[root@exadatacell01 raw]# ln -s /dev/sde exadatacell01_DISK03
[root@exadatacell01 raw]# ln -s /dev/sdf exadatacell01_DISK04
[root@exadatacell01 raw]# ln -s /dev/sdg exadatacell01_DISK05
[root@exadatacell01 raw]# ln -s /dev/sdh exadatacell01_DISK06
[root@exadatacell01 raw]# ln -s /dev/sdi exadatacell01_DISK07
[root@exadatacell01 raw]# ln -s /dev/sdj exadatacell01_DISK08
[root@exadatacell01 raw]# ln -s /dev/sdk exadatacell01_DISK09
[root@exadatacell01 raw]# ln -s /dev/sdl exadatacell01_FLASH00
[root@exadatacell01 raw]# ln -s /dev/sdm exadatacell01_FLASH01
[root@exadatacell01 raw]# ln -s /dev/sdn exadatacell01_FLASH02
[root@exadatacell01 raw]# ln -s /dev/sdo exadatacell01_FLASH03

[root@exadatacell01 raw]# ls -lrt
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK00 -> /dev/sdb
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK01 -> /dev/sdc
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK02 -> /dev/sdd
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK03 -> /dev/sde
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK04 -> /dev/sdf
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK05 -> /dev/sdg
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK06 -> /dev/sdh
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK07 -> /dev/sdi
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK08 -> /dev/sdj
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_DISK09 -> /dev/sdk
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_FLASH00 -> /dev/sdl
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_FLASH01 -> /dev/sdm
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_FLASH02 -> /dev/sdn
lrwxrwxrwx. 1 root root 8 Jul  5 18:32 exadatacell01_FLASH03 -> /dev/sdo

8.6.1 Create Script for symbolic links

[root@exadatacell01 raw]# cat symbolic_link.sh
ln -s /dev/sdb /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK00
ln -s /dev/sdc /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK01
ln -s /dev/sdd /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK02
ln -s /dev/sde /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK03
ln -s /dev/sdf /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK04
ln -s /dev/sdg /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK05
ln -s /dev/sdh /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK06
ln -s /dev/sdi /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK07
ln -s /dev/sdj /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK08
ln -s /dev/sdk /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK09
ln -s /dev/sdl /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_FLASH00
ln -s /dev/sdm /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_FLASH01
ln -s /dev/sdn /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_FLASH02
ln -s /dev/sdo /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_FLASH03

8.7. Ready to Create Storage cell

8.7.1. Connect with CELLADMIN and Create the Storage cell

[celladmin@exadatacell01 ~]$ cellcli -e create cell interconnect1=eth1
CELL-01518: Stop CELLSRV. Create Cell cannot continue with CELLSRV running.

Note::: Getting above error or because of CELLSRV already running. So need to stop CELLSRV service

[celladmin@exadatacell01 config]$ cellcli
CellCLI: Release 12.1.1.1.0 - Production on Tue Jul 07 14:30:19 EDT 2020

Copyright (c) 2007, 2013, Oracle.  All rights reserved.
Cell Efficiency Ratio: 1

CellCLI> alter cell shutdown services CELLSRV

Stopping CELLSRV services...
The SHUTDOWN of CELLSRV services was successful.

[celladmin@exadatacell01 ~]$ cellcli -e create cell interconnect1=eth1
Cell exadatacell01 successfully created
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Flash cell disks, FlashCache, and FlashLog will be created...
CellDisk FD_00_exadatacell01 successfully created
CellDisk FD_01_exadatacell01 successfully created
CellDisk FD_02_exadatacell01 successfully created
CellDisk FD_03_exadatacell01 successfully created
Flash log exadatacell01_FLASHLOG successfully created
Flash cache exadatacell01_FLASHCACHE successfully created

8.7.2. Create Cell Disk

CellCLI> create celldisk all
CellDisk CD_DISK00_exadatacell01 successfully created
CellDisk CD_DISK01_exadatacell01 successfully created
CellDisk CD_DISK02_exadatacell01 successfully created
CellDisk CD_DISK03_exadatacell01 successfully created
CellDisk CD_DISK04_exadatacell01 successfully created
CellDisk CD_DISK05_exadatacell01 successfully created
CellDisk CD_DISK06_exadatacell01 successfully created
CellDisk CD_DISK07_exadatacell01 successfully created
CellDisk CD_DISK08_exadatacell01 successfully created
CellDisk CD_DISK09_exadatacell01 successfully created

8.7.3. Show Created Cell Disk

CellCLI> list celldisk
         CD_DISK00_exadatacell01         normal
         CD_DISK01_exadatacell01         normal
         CD_DISK02_exadatacell01         normal
         CD_DISK03_exadatacell01         normal
         CD_DISK04_exadatacell01         normal
         CD_DISK05_exadatacell01         normal
         CD_DISK06_exadatacell01         normal
         CD_DISK07_exadatacell01         normal
         CD_DISK08_exadatacell01         normal
         CD_DISK09_exadatacell01         normal
         FD_00_exadatacell01             normal
         FD_01_exadatacell01             normal
         FD_02_exadatacell01             normal
         FD_03_exadatacell01             normal

8.7.4. Show Created Flash Disk

CellCLI> list celldisk where disktype=flashdisk
         FD_00_exadatacell01     normal
         FD_01_exadatacell01     normal
         FD_02_exadatacell01     normal
         FD_03_exadatacell01     normal

8.7.5. Few important commands

CellCLI> list celldisk where disktype=harddisk
         CD_DISK00_exadatacell01         normal
         CD_DISK01_exadatacell01         normal
         CD_DISK02_exadatacell01         normal
         CD_DISK03_exadatacell01         normal
         CD_DISK04_exadatacell01         normal
         CD_DISK05_exadatacell01         normal
         CD_DISK06_exadatacell01         normal
         CD_DISK07_exadatacell01         normal
         CD_DISK08_exadatacell01         normal
         CD_DISK09_exadatacell01         normal

CellCLI> list physicaldisk where disktype=HardDisk attributes name
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK00
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK01
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK02
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK03
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK04
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK05
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK06
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK07
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK08
         /opt/oracle/cell12.1.1.1.0_LINUX.X64_130109/disks/raw/exadatacell01_DISK09

CellCLI> list flashcache detail
         name:                   exadatacell01_FLASHCACHE
         cellDisk:               FD_02_exadatacell01,FD_01_exadatacell01,FD_03_exadatacell01,FD_00_exadatacell01
         creationTime:           2020-07-07T01:41:55-04:00
         degradedCelldisks:
         effectiveCacheSize:     1.625G
         id:                     c0505a54-7de2-4e87-bad5-e65d71bc76a0
         size:                   1.625G
         status:                 normal