Collaborate 17 Oracle RAC 12cRel 2 Best Practices

Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Oracle Real Application Clusters
Anil Nair
Sr. Principal Product Manager,
Oracle Real Application Clusters (RAC)
Apr 3rd , 2017
@RACMasterPM
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/
12c Release 2 Best Practices

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
3

Oracle RAC 12c Release 2 Best Practices
Cluster configuration Options
Install a Standalone Cluster
Install a Domain Services Cluster (DSC)
Create Member Cluster Configuration file
Install Member Services Cluster (MC)
Frequently asked Questions (FAQ)
1
4
2
3
4
5
6

Cluster Domain
5
Standalone Cluster

• Standalone Cluster hosts Oracle Grid
Infrastructure (including ASM),
Management Repository
• Nodes access shared storage in a
consistent way (NAS or block storage)
• Can be deployed as Hub nodes or Leaf
Nodes
– Read Write instances can be deployed on Hub
Nodes
– Read only instances can be deployed to Leaf
nodes
6
Standalone Cluster deployment

• Cluster domain groups multiple cluster
configurations to help standardize,
centralize and optimize your deployment
• Consists of
– One or more nodes hosting the Domain
Services Cluster (DSC)
– One or more nodes hosting the Database
Member Cluster
– Optionally one or more nodes hosting the
Application Member Cluster
7
Cluster Domain deployment
Domain Services Cluster
ASM
IO
ACFS ASM
Database
Member
Cluster
Uses ASM
Service
Database
Member
Cluster
Uses ASM
IO Service of
DSC
TFAGIMR
Application
Member
Cluster
GI only
Database
Member
Cluster
Uses local
ASM
Shared ASM
OptionalRHP
Private
Network
1 2 3 4
SAN
Storage
ASM
Network

• One or more nodes can be configured as
a Domain Services Cluster (DSC)
• DSC provides
– Centralized Grid Infrastructure Management
Repository (GIMR)Services
– Centralized Trace File Analyzer (TFA) services
– Automatic Storage Management (ASM) via
both direct and indirect mode
– Rapid Home Provisioning (RHP) services
• These centralized services can be utilized
by Member clusters of this Cluster
Domain
8
ASM
IO
Service
ASM
ServiceTFAGIMR
Shared ASM
RHP
Nodes of Domain
Services Cluster

• ASM on DSC service provides centralized
storage management services
• Member Cluster (MC) can access the
shared storage on the DSC using
– a direct physical connection to the shared
storage
– a network path utilizing the ASM IO services
• All nodes of a MC have to use the same
storage access method (direct or indirect)
9
Accessing storage on the Domain Services Cluster
ASM
IO
Service
ASM
ServiceTFAGIMR
Shared ASM
RHP
Nodes of Domain
Services Cluster

Database Member Cluster
with Local Storage
10
Application Member Cluster Database Member Cluster utilizing
the shared storage services
provided by
Types of Member Clusters

• Database Member Cluster consist of one
or more nodes used to host one or many
Databases
• Database Member Cluster
– Uses the DSC for GIMR and TFA services
– Uses ASM with Local storage for Databases
•
Note the over loaded use of term “Local”
– In this case, the member cluster has “local
storage” which is local to members of the
Database Member and is Not shared with
other Member clusters
11
Database Member Cluster with local storage
1
ASM
IO
Service
ASM
ServiceTFAGIMR
Shared ASM
RHP
Local storage is not
shared with other
Member Clusters
1 2
43

• Application Member Cluster consist of
one or more nodes used to host
Applications
• Application Member Cluster
– Uses the DSC for GIMR and TFA services
– May or may not have optional local storage as
required by the Application
• Benefits from the centralized GIMR and
TFA services
Application Member Cluster
ASM
IO
Service
ASM
ServiceTFAGIMR
Shared ASM
RHP
21

Database Member Cluster to utilize the
shared storage services provided by the
DSC without direct connection to Shared
Storage
• ASM instance running on the DSC
provides a new service called ASM I/O
service
• This can be utilized by Member cluster to
access the Shared storage services
provided by the DSC via ASM network
• Also benefits from the centralized GIMR
and TFA services
13
Database Member Cluster utilizing ASM I/O services
ASM
IO
Service
ASM
ServiceTFAGIMR
Shared ASM
RHP
3
1 2
43

Database Member Cluster to utilize the
shared storage services provided by the
DSC with direct connection to Shared
Storage
• Member Clusters utilize the ASM instance
running on the DSC for metadata but
performs Read/Write directly to the
Shared Storage
• Also benefits from the centralized GIMR
and TFA services
14
Database Member Cluster with direct connection to Storage
ASM
IO
Service
ASM
ServiceTFAGIMR
Shared ASM
RHP
4
1 2
43

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 15
Please visit session # 628 for more information

1
16
2
3
4
5
6

Installing a Standalone Cluster

Installing a standalone cluster
GNS and GNS sub-domain
delegation are optional

Cluster Node Information
Virtual IP defaults
to hostname-VIP

Add Cluster Node Information
Add a range of hosts
using regex expressions

Configure nodes as Hub or leaf nodes
1. Optionally configure some
nodes as Leaf Nodes to run
read only instances
2. Note that leaf nodes do not
need Virtual hostname

Specify Network Usage

Configure Storage Options
Disk group containing OCR and
Voting disks needs to be on ASM

Create ASM Disk Group

Slight
digression

Installing a standalone cluster (Extended Cluster)

Back on track

Specify Oracle Base

Root script execution

Prerequisite checks
Cluvfy runs deployment specific
checks for installation

1
32
2
3
4
5
6

Domain Services Cluster Installation steps

Domain Services Cluster Installation steps
GNS is required but GNS sub-
domain delegation is optional

Cluster Node information for DSC
Read only instances cannot be
configured on Leaf nodes in a
Cluster Domain configuration

Network Interface Usage for DSC
Choose separate and possibly
multiple ASM private
networks to ensure adequate
bandwidth is available for
Member clusters accessing
storage on DSC via ASM IO
Services

Grid Infrastructure Management Repository options for DSC
Recommendation is to create
separate Disk group for
Management Database

Configure storage options for DSC
1. For Re-Installs after a failed
install, remember to check for afd
driver using #lsmod |grep afd
2. Executing dd on a device
protected by afd is ignored
silently

Specify ASM password for DSC

Prerequisite Checks for DSC
Cluvfy runs deployment specific
checks for installation

Execute configuration steps for DSC

Execute root.sh to configure DSC

Domain Services Cluster post install tasks

Successful install of DSC

1
46
2
3
4
5
6

Create Member Cluster configuration file
$ssh <DSC_host>
$ cd $GRID_HOME/bin
$./crsctl create member_cluster_configuration -h
$./crsctl create member_cluster_configuration salesmb -file
$HOME/sales_mb .xml
$scp $HOME/sales_mb.xml mc_host:/tmp
Check Usage
crsctl create –h
Create Cluster policyset file using
crsctl create policyset –file <path>
Use the –domain_service attribute to
configure how member cluster accesses
ASM storage
Backup the policyset file as file cannot be
recreated

Check Member Cluster configuration
$ssh <DSC_host>
$ cd $GRID_HOME/bin
$./crsctl query cluster site -all
Site 'rwsxxxxxxxxx5'
identified by GUID '85ce299c88ef4fbdff101a16af6de8b0' in
state 'ENABLED' contains
nodes 'rwsxxxxxx7,rwsxxxxxx8' and no disk.
Check Usage
crsctl query –h
Query Member Cluster configuration using
$crsctl query cluster site -all

1
49
2
3
4
5
6

Member Cluster Installation

Member Cluster Installation
Enter the Cluster configuration file
name created earlier in the DSC

Slight
digression

Note new command to install Grid Infrastructure
$scp linuxx64_12201_grid_home.zip
rwsxxxx9:/u01/app/12.2.0/grid
$ssh rwsxxxx9
$export DISPLAY=rwsxxxx9:1
$unzip linuxx64_12201_grid_home.zip
$./gridSetup.sh
1. Use gridSetup.sh to install Grid
Infrastructure
2. Use gridSetup.sh to add Nodes
3. Use deinstall to delete Nodes
4. Continue to use runInstaller to
install Database Home

A brief overview of ./runInstaller
Step 12.1.0.2
1 Download shiphome zip files
2 Unzip grid1/2.zip to some stage location
(stage_loc)
3 Execute <stage_loc>/runInstaller.sh
4 Bootstraps files to some temp area (1GB)
5 Copies files from <stage_loc> to Grid Home
6 Zip up the Grid Home and store it as image files
7 Unzip image files to OH on all nodes of the
cluster
8 Execute clone.pl on all nodes of cluster
9 Run config.sh from one of the nodes
64

Step 12.2.0.1
1 Download image zip files
2 Unzip image files to OH on one of the nodes of
cluster
3 Run gridSetup.sh from one of the nodes
• Create gold images
– Download the GI home
– Unzip the files
– apply environment specific patches
– Zip it up
– use as Gold Image
65
So what’s new with ./gridSetup.sh?

1
66
2
3
4
5
6

Frequently asked Question # 1
Why does MGMT
DB need so much
space?

Cluster Type Redundancy MGMT DG
(GB)
(2 Node DSC with 4 Member
Clusters of 2 Nodes each)
External 188
Normal 376
High 564
Flex 376
Standalone Cluster
(4 Node Cluster)
External 38
Normal 76
High 114
Flex 76
• Oracle GI 12c Release 2 feature AHF
(Autonomous Health Framework) collects
and stores diagnostics data from OS and
DB in MGMT
• In DSC, one PDB per member cluster is
provisioned to store member cluster
diagnostics data
• The data is used by AHF components like
cluster health advisor to both prevent
and help diagnose issues
68
GIMR space requirements

Machine Learning for Automatic Diagnosis
Faults
Alarms
Incidents
Root Causes
Corrective
Actions
Diagnostic Data collected from OS and the database is analyzed using a Bayesian Belief Network for Cause
and Effect Analysis to automatically discover potential issues or take corrective actions
101011111101011010101010101
010101110110001010101010101
010101010101010101010101101
Diagnostic Data
Analyze

chactl query diagnosis-db
70
$ chactl query diagnosis -db sales -start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“
2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_1) [detected]
2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_2) [detected]
2017-03-31 10:01:13.0 Database sales DB CPU Utilization (sales_2) [detected]
2017-03-31 10:01:33.0 Database salesDB Log File Switch (sales_1) [detected]
Consolidates and displays
information from all instances

chactl query diagnosis -cluster
71
$chactl query diagnosis –start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“
2017-03-31 10:01:25.0 Host rwsxxxxx9 Host Memory Consumption [detected]
Problem: Host Memory Consumption
Description: CHA detected that more memory than expected is consumed on this server. The
memory is not allocated by sessions of this database.
Cause: The Cluster Health Advisor (CHA) detected an increase in memory consumption by other
databases or by applications not connected to a database on this node.
Action: Identify the top memory consumers by using the Cluster Health Monitor (CHM).
Note: This is a 4 node cluster but the memory
consumption issue is reported on 3 of the 4 nodes

chatcl actions & resolutions in text or html format
72

• Stores OS, Database and node
related performance data
– CPU
– Core
– Memory Utilization
– Storage performance
– Network latency and bandwidth
– Clusterware heartbeats
– Database events
• Single framework that utilizes the
data collected to help manage
performance
73
This should clarify MGMT DB space requirements

• opatch automatically patches
MGMT database if required
• clients of MGMT connect using
encrypted credentials
• MGMT listener automatically
maintained by the clusterware
agent
74
But it is still one
more database
for me to manage

Wait, I have more
questions

• Recommendation is to use separate
disk group for MGMT DB
– This disk group will be used for OCR
and Voting disk backups
– Unlike prior releases, there is no OCR
backup to GI home
• Installer prompts during fresh
install
76
Should I use one Disk
Group for OCR, Voting and
MGMT DB?

How do I find out the
cluster type?
$crsctl get node status –all
Node ‘rwsxxxx6’ active role is hub
Node ‘rwsxxxx7’ active role is hub
Node ‘rwsxxxx8’ active role is leaf
Node ‘rwsxxxx9’ active role is leaf
$crsctl get cluster mode status
Cluster is running in “flex” mode
$crsctl get cluster mode class
CRS-41008 : Cluster class is ‘Standalone Cluster’
CRS-41008 : Cluster class is ‘Domain Services Cluster’
CRS-41008 : Cluster class is ‘Database Member Cluster’

How do I continue from
“CLSRSC-400: A system
reboot is required to
continue installing”?
./gridSetup.sh -executeConfigTools
-responseFile
/u01/app/12.2.0/grid/install/response/grid_2017-03-
14_03-19-41PM.rsp
• Use executeConfigTools option of
gridSetup.sh
• Requires responseFile
– Hint: Use the response file auto saved by the
Installer under $GI_HOME/grid/install
• Optionally add passwords to the response
file for a silent install experience
• configToolAllCommands is deprecated

• All 12.2.0 nodes are in Flex Cluster mode
• All 12.2.0 nodes are in Flex ASM mode
• New Oracle 12.2 RAC Node weighting
algorithm considers available ASM
instances during split brain
How to configure cluster
for Flex ASM and Flex
Cluster?

• Node weighting considers
Secondary failure conditions like
the public network and Flex ASM
instances that would survive the
eviction following a split brain
• The idea is to let the majority of the
work survive
80
Node Weighting in Oracle RAC 12c Release 2
1 2
43

Can I convert Read Write
instances to Read Only ?
$crsctl set node role leaf –node rwsxxxxxx1
• Stop the instance
• Set the node role
– crsctl set node role leaf –node <nodename>
• Startup the instance using the spfile with
read only instance specific parameters

Which deployment model
should I use?
• Customers have a choice of deployment
models
• Some factors to consider on the
deployment choices are
– Cluster Domain requires a fresh install
– ASM Database Oriented Storage features on the
Domain Services cluster can be used to rapidly
clone databases
– Storage administrators can consolidate multiple
databases in the DSC by utilizing the new ASM
feature which allows admins to set a quota at the
database level
• Regardless of the deployment choice, the
tools to manage and deploy does change

Please Complete Your
Session Evaluation
Evaluate this session in your COLLABORATE app. Pull up this session
and tap "Session Evaluation"
to complete the survey.
Session ID: 359

Collaborate 17 Oracle RAC 12cRel 2 Best Practices

Collaborate 17 Oracle RAC 12cRel 2 Best Practices

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Collaborate 17 Oracle RAC 12cRel 2 Best Practices

Similar to Collaborate 17 Oracle RAC 12cRel 2 Best Practices (20)

More from Anil Nair

More from Anil Nair (7)

Recently uploaded

Recently uploaded (20)

Collaborate 17 Oracle RAC 12cRel 2 Best Practices