A node in the cluster may crash due to hardware or OS problems and is not recoverable. So DBA needs to delete a problematic node from clusterware. In this topic I will show discuss how to delete a node from two nodes RAC.
Two Nodes: ocmnode1 will be deleted
- ocmnode1
- ocmnode2
Deinstall Oracle Home:
[oracle@ocmnode1 ~]$ . ./.bash_profile [oracle@ocmnode1 ~]$ echo $ORACLE_HOME /u01/app/oracle/product/12.1.0/dbhome_1 [oracle@ocmnode1 ~]$ $ORACLE_HOME/deinstall/deinstall -local Checking for required files and bootstrapping ... Please wait ... Location of logs /u01/app/oraInventory/logs/ ############ ORACLE DECONFIG TOOL START ############ ######################### DECONFIG CHECK OPERATION START ######################### ## [START] Install check configuration ## Checking for existence of the Oracle home location /u01/app/oracle/product/12.1.0/dbhome_1 Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database Oracle Base selected for deinstall is: /u01/app/oracle Checking for existence of central inventory location /u01/app/oraInventory Checking for existence of the Oracle Grid Infrastructure home The following nodes are part of this cluster: ocmnode1 Checking for sufficient temp space availability on node(s) : 'ocmnode1' ## [END] Install check configuration ## Network Configuration check config START Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2020-08-08_11-28-25-AM.log Network Configuration check config END Database Check Configuration START Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2020-08-08_11-28-33-AM.log Use comma as separator when specifying list of values as input Specify the list of database names that are configured locally on this node for this Oracle home. Local configurations of the discovered databases will be removed []: Database Check Configuration END Oracle Configuration Manager check START OCM check log file location : /u01/app/oraInventory/logs//ocm_check8522.log Oracle Configuration Manager check END ######################### DECONFIG CHECK OPERATION END ######################### ####################### DECONFIG CHECK OPERATION SUMMARY ####################### Oracle Grid Infrastructure Home is: The following nodes are part of this cluster: ocmnode1 The cluster node(s) on which the Oracle home deinstallation will be performed are:ocmnode1 Oracle Home selected for deinstall is: /u01/app/oracle/product/12.1.0/dbhome_1 Inventory Location where the Oracle home registered is: /u01/app/oraInventory Checking the config status for CCR Oracle Home exists with CCR directory, but CCR is not configured CCR check is finished Do you want to continue (y - yes, n - no)? [n]: y A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2020-08-08_11-28-24-AM.out' Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2020-08-08_11-28-24-AM.err' ######################## DECONFIG CLEAN OPERATION START ######################## Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2020-08-08_11-36-02-AM.log Network Configuration clean config START Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2020-08-08_11-36-02-AM.log Network Configuration clean config END Oracle Configuration Manager clean START OCM clean log file location : /u01/app/oraInventory/logs//ocm_clean8522.log Oracle Configuration Manager clean END ######################### DECONFIG CLEAN OPERATION END ######################### ####################### DECONFIG CLEAN OPERATION SUMMARY ####################### Cleaning the config for CCR As CCR is not configured, so skipping the cleaning of CCR configuration CCR clean is finished ####################################################################### ############# ORACLE DECONFIG TOOL END ############ Using properties file /tmp/deinstall2020-08-08_11-28-16AM/response/deinstall_2020-08-08_11-28-24-AM.rsp Location of logs /u01/app/oraInventory/logs/ ############ ORACLE DEINSTALL TOOL START ############ ####################### DEINSTALL CHECK OPERATION SUMMARY ####################### A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2020-08-08_11-28-24-AM.out' Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2020-08-08_11-28-24-AM.err' ######################## DEINSTALL CLEAN OPERATION START ######################## ## [START] Preparing for Deinstall ## Setting LOCAL_NODE to ocmnode1 Setting CLUSTER_NODES to ocmnode1 Setting CRS_HOME to false Setting oracle.installer.invPtrLoc to /tmp/deinstall2020-08-08_11-28-16AM/oraInst.loc Setting oracle.installer.local to true ## [END] Preparing for Deinstall ## Setting the force flag to false Setting the force flag to cleanup the Oracle Base Oracle Universal Installer clean START Detach Oracle home '/u01/app/oracle/product/12.1.0/dbhome_1' from the central inventory on the local node : Done Delete directory '/u01/app/oracle/product/12.1.0/dbhome_1' on the local node : Done The Oracle Base directory '/u01/app/oracle' will not be removed on local node. The directory is not empty. Oracle Universal Installer cleanup was successful. Oracle Universal Installer clean END ## [START] Oracle install clean ## Clean install operation removing temporary directory '/tmp/deinstall2020-08-08_11-28-16AM' on node 'ocmnode1' ## [END] Oracle install clean ## ######################### DEINSTALL CLEAN OPERATION END ######################### ####################### DEINSTALL CLEAN OPERATION SUMMARY ####################### Successfully detached Oracle home '/u01/app/oracle/product/12.1.0/dbhome_1' from the central inventory on the local node. Successfully deleted directory '/u01/app/oracle/product/12.1.0/dbhome_1' on the local node. Oracle Universal Installer cleanup was successful. Oracle deinstall tool successfully cleaned up temporary directories. ####################################################################### ############# ORACLE DEINSTALL TOOL END #############
Check the pinned status of nodes which you want to delete from either root user or the user that Installed Oracle Clusterware:
Update Inventory for Oracle Home:
[oracle@ocmnode2 bin]$ pwd
/u01/app/oracle/product/12.1.0/dbhome_1/oui/bin
[oracle@ocmnode2 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 "CLUSTER_NODES={ocmnode2}"
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4031 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
[root@ocmnode1 trace]# olsnodes -s -t PRCO-19: Failure retrieving list of nodes in the cluster PRCO-2: Unable to communicate with the clusterware [grid@ocmnode2 dev]$ olsnodes -s -t ocmnode1 Inactive Unpinned ocmnode2 Active Unpinned
Deconfigure the Clusterware:
On the node you want to delete, run the following command as the user that installed Oracle Clusterware:
[root@ocmnode1 ~]# /u01/app/12.1.0/grid/crs/install/rootcrs.pl -deconfig -force -verbose Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params PRCR-1070 : Failed to check if resource ora.net1.network is registered CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1070 : Failed to check if resource ora.helper is registered CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1070 : Failed to check if resource ora.ons is registered CRS-0184 : Cannot communicate with the CRS daemon. CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ocmnode1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'ocmnode1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'ocmnode1' CRS-2673: Attempting to stop 'ora.gpnpd' on 'ocmnode1' CRS-2673: Attempting to stop 'ora.evmd' on 'ocmnode1' CRS-2673: Attempting to stop 'ora.gipcd' on 'ocmnode1' CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'ocmnode1' CRS-2677: Stop of 'ora.drivers.acfs' on 'ocmnode1' succeeded CRS-2677: Stop of 'ora.cssdmonitor' on 'ocmnode1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'ocmnode1' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'ocmnode1' succeeded CRS-2677: Stop of 'ora.evmd' on 'ocmnode1' succeeded CRS-2677: Stop of 'ora.gipcd' on 'ocmnode1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ocmnode1' has completed CRS-4133: Oracle High Availability Services has been stopped. 2020/08/08 10:51:31 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector. 2020/08/08 10:51:55 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector. 2020/08/08 10:51:55 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node
Delete clusterware configuration from other running nodes:
[root@ocmnode2 ~]# /u01/app/12.1.0/grid/bin/crsctl delete node -n ocmnode1 CRS-4661: Node ocmnode1 successfully deleted.
Verify the Clusterware Status:
[root@ocmnode2 ~]# . oraenv ORACLE_SID = [root] ? +ASM2 The Oracle base has been set to /u01/app/grid [root@ocmnode2 ~]# crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------- ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE ocmnode2 ora.FRA.dg ora....up.type 0/5 0/ ONLINE ONLINE ocmnode2 ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE ocmnode2 ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE ocmnode2 ora....N2.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE ocmnode2 ora....N3.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE ocmnode2 ora.MGMTLSNR ora....nr.type 0/0 0/0 ONLINE ONLINE ocmnode2 ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE ocmnode2 ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE ocmnode2 ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE ocmnode2 ora.mgmtdb ora....db.type 0/2 0/1 ONLINE ONLINE ocmnode2 ora....network ora....rk.type 0/5 0/ ONLINE ONLINE ocmnode2 ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE ocmnode2 ora....de1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE ocmnode2 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE ocmnode2 ora....E2.lsnr application 0/5 0/0 ONLINE ONLINE ocmnode2 ora....de2.ons application 0/3 0/0 ONLINE ONLINE ocmnode2 ora....de2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE ocmnode2 ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE ocmnode2 ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE ocmnode2 ora.scan2.vip ora....ip.type 0/0 0/0 ONLINE ONLINE ocmnode2 ora.scan3.vip ora....ip.type 0/0 0/0 ONLINE ONLINE ocmnode2
[root@ocmnode2 ~]# crsctl check cluster -all ************************************************************** ocmnode2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online **************************************************************
Update Inventory on Grid Home:
[grid@ocmnode1 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@ocmnode1 ~]$ echo $ORACLE_HOME
/u01/app/12.1.0/grid
[grid@ocmnode1 ~]$ cd $ORACLE_HOME
[grid@ocmnode1 grid]$ cd oui/bin
[grid@ocmnode1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={ocmnode1}" CRS=TRUE -local
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4031 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
Deinstall Grid Home:
[grid@ocmnode1 bin]$ cd /u01/app/12.1.0/grid/deinstall [grid@ocmnode1 deinstall]$ ./deinstall -local Checking for required files and bootstrapping ... Please wait ... Location of logs /tmp/deinstall2020-08-08_11-46-53AM/logs/ ############ ORACLE DECONFIG TOOL START ############ ######################### DECONFIG CHECK OPERATION START ######################### ## [START] Install check configuration ## Checking for existence of the Oracle home location /u01/app/12.1.0/grid Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster Oracle Base selected for deinstall is: /u01/app/grid Checking for existence of central inventory location /u01/app/oraInventory Checking for existence of the Oracle Grid Infrastructure home The following nodes are part of this cluster: ocmnode1 Checking for sufficient temp space availability on node(s) : 'ocmnode1' ## [END] Install check configuration ## Traces log file: /tmp/deinstall2020-08-08_11-46-53AM/logs//crsdc_2020-08-08_11-47-10AM.log Network Configuration check config START Network de-configuration trace file location: /tmp/deinstall2020-08-08_11-46-53AM/logs/netdc_check2020-08-08_11-47-10-AM.log Network Configuration check config END Asm Check Configuration START ASM de-configuration trace file location: /tmp/deinstall2020-08-08_11-46-53AM/logs/asmcadc_check2020-08-08_11-47-10-AM.log Database Check Configuration START Database de-configuration trace file location: /tmp/deinstall2020-08-08_11-46-53AM/logs/databasedc_check2020-08-08_11-47-10-AM.log Database Check Configuration END ######################### DECONFIG CHECK OPERATION END ######################### ####################### DECONFIG CHECK OPERATION SUMMARY ####################### Oracle Grid Infrastructure Home is: The following nodes are part of this cluster: ocmnode1 The cluster node(s) on which the Oracle home deinstallation will be performed are:ocmnode1 Oracle Home selected for deinstall is: /u01/app/12.1.0/grid Inventory Location where the Oracle home registered is: /u01/app/oraInventory Option -local will not modify any ASM configuration. Do you want to continue (y - yes, n - no)? [n]: y A log of this session will be written to: '/tmp/deinstall2020-08-08_11-46-53AM/logs/deinstall_deconfig2020-08-08_11-47-09-AM.out' Any error messages from this session will be written to: '/tmp/deinstall2020-08-08_11-46-53AM/logs/deinstall_deconfig2020-08-08_11-47-09-AM.err' ######################## DECONFIG CLEAN OPERATION START ######################## Database de-configuration trace file location: /tmp/deinstall2020-08-08_11-46-53AM/logs/databasedc_clean2020-08-08_11-47-21-AM.log ASM de-configuration trace file location: /tmp/deinstall2020-08-08_11-46-53AM/logs/asmcadc_clean2020-08-08_11-47-21-AM.log ASM Clean Configuration END Network Configuration clean config START Network de-configuration trace file location: /tmp/deinstall2020-08-08_11-46-53AM/logs/netdc_clean2020-08-08_11-47-21-AM.log Network Configuration clean config END ######################### DECONFIG CLEAN OPERATION END ######################### ####################### DECONFIG CLEAN OPERATION SUMMARY ####################### Oracle Clusterware was already stopped and de-configured on node "ocmnode1" Oracle Clusterware is stopped and de-configured successfully. ####################################################################### ############# ORACLE DECONFIG TOOL END ############# Using properties file /tmp/deinstall2020-08-08_11-46-53AM/response/deinstall_2020-08-08_11-47-09-AM.rsp Location of logs /tmp/deinstall2020-08-08_11-46-53AM/logs/ ############ ORACLE DEINSTALL TOOL START ############ ####################### DEINSTALL CHECK OPERATION SUMMARY ####################### A log of this session will be written to: '/tmp/deinstall2020-08-08_11-46-53AM/logs/deinstall_deconfig2020-08-08_11-47-09-AM.out' Any error messages from this session will be written to: '/tmp/deinstall2020-08-08_11-46-53AM/logs/deinstall_deconfig2020-08-08_11-47-09-AM.err' ######################## DEINSTALL CLEAN OPERATION START ######################## ## [START] Preparing for Deinstall ## Setting LOCAL_NODE to ocmnode1 Setting CLUSTER_NODES to ocmnode1 Setting CRS_HOME to false Setting oracle.installer.invPtrLoc to /tmp/deinstall2020-08-08_11-46-53AM/oraInst.loc Setting oracle.installer.local to true ## [END] Preparing for Deinstall ## ..... Oracle Universal Installer cleanup was successful. Oracle Universal Installer clean END ## [START] Oracle install clean ## Clean install operation removing temporary directory '/tmp/deinstall2020-08-08_11-46-53AM' on node 'ocmnode1' ## [END] Oracle install clean ## ######################### DEINSTALL CLEAN OPERATION END ######################### ####################### DEINSTALL CLEAN OPERATION SUMMARY ####################### Successfully detached Oracle home '/u01/app/12.1.0/grid' from the central inventory on the local node. Failed to delete directory '/u01/app/12.1.0/grid' on the local node. Successfully deleted directory '/u01/app/oraInventory' on the local node. Oracle Universal Installer cleanup was successful. Run 'rm -r /etc/oraInst.loc' as root on node(s) 'ocmnode1' at the end of the session. Run 'rm -r /opt/ORCLfmap' as root on node(s) 'ocmnode1' at the end of the session. Run 'rm -r /etc/oratab' as root on node(s) 'ocmnode1' at the end of the session. Oracle deinstall tool successfully cleaned up temporary directories. ####################################################################### ############# ORACLE DEINSTALL TOOL END #############
[root@ocmnode1 ~]# rm -r /etc/oraInst.loc [root@ocmnode1 ~]# rm -rf /opt/ORCLfmap [root@ocmnode1 ~]# rm -rf /etc/oratab
Update Inventory of Oracle Home for all others node:
[grid@ocmnode2 bin]$ pwd
/u01/app/12.1.0/grid/oui/bin
[grid@ocmnode2 bin]$ export ORACLE_HOME=/u01/app/12.1.0/grid/
[grid@ocmnode2 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={ocmnode2}" CRS=TRUE
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4019 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
Validation after deletion of Node.
Check node list on Cluster
[grid@ocmdrnode1 bin]$ olsnodes -n ocmdrnode1 1 ocmdrnode2 2 [grid@ocmdrnode1 bin]$ olsnodes -a -t -s -n ocmdrnode1 1 Active Hub Unpinned [grid@ocmdrnode1 bin]$ olsnodes -t -s -n ocmdrnode1 1 Active Unpinned ocmdrnode2 2 Inactive Unpinned
Need to delete node using below command from workable node.
[root@ocmdrnode1 ~]# crsctl delete node -n ocmdrnode2 CRS-4661: Node ocmdrnode2 successfully deleted. [grid@ocmdrnode1 addnode]$ olsnodes -n ocmdrnode1 1 [grid@ocmdrnode1 addnode]$ olsnodes -t -s -n ocmdrnode1 1 Active Unpinned
[grid@ocmnode2 bin]$ pwd /u01/app/12.1.0/grid/bin [grid@ocmnode2 bin]$ ./cluvfy stage -post nodedel -n ocmnode1 Performing post-checks for node removal Checking CRS integrity... CRS integrity check passed Clusterware version consistency passed. Node removal check passed Post-check for node removal was successful.
