Oracle RAC features on Exadata

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Oracle RAC integration with Exadata, IMDB &
Data Guard
Anil Nair
Sr. Principal Product Manager,
Oracle Real Application Clusters (RAC)
Apr 24th, 2018
@RACMasterPM
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/
20 Years of Continuous Innovation

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
3

• Please remember to provide
feedback
– Session ID Session ID: 1374
– New integration features of Oracle RAC
12c Rel 2 with Exadata, Data Guard and
IMDB
• Reach out to us with follow up
questions, suggestions etc. via
Twitter @RACMasterPM
Linkedin http://www.linkedin.com/in/anil-nair-
01960b6
Feedback is important for successful collaboration

Program Agenda
Oracle RAC, Exadata! Why now?
Oracle RAC optimizations on Exadata, In-Memory, Data
Guard
1
2
5

Program Agenda
Oracle RAC, Exadata! Why now?1
2
6

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 7
Oracle Real Application Clusters (RAC)
0
5000
10000
15000
20000
25000
30000
35000
40000
4 8 32 48 64 80
2035
4010
15520
22416
30016
37040
# of Cores across RAC Nodes
Users
2 Nodes
3 Nodes
4 Nodes
5 Nodes
SAP certified SD Benchmark results
• Scalable Active/Active Instances
• No Application changes Required

Compared to…
8
• One Writer Multiple
Readers
• Connection aware
sessions
• Not Portable
• Lacks popular
Application Support
Source: AWS blog

• CPU/Cores
– RAC features for PDBs ensure smooth
consolidated environments
• NUMA
– (IPC) layer enhanced to consider NUMA
• Flash/NVMe
– Cache Fusion algorithm tweaked for
low latency devices
9
While Oracle RAC continues to innovate

But, Performance improvement is not linear
Source: Intel

Including the latest All Flash Storage Arrays
Flash Storage Array
Servers
10X Bottleneck!
Even with 25x
10Gb/s Links
250 GB/s Flash Bandwidth
25 GB/s
• Flash performance suffers at various
layers like HBA/SAN
– Current Flash Drives are much faster than SANs
and LANs
– One PCI flash drive is 2.5X faster than a 10Gb
net link
• Un-optimized general purpose SSDs
affects performance
– SSD garbage collection typically affects
consistent I/O performance
– Wear levelling and Firmware compatibilities
with underlying OS

Exadata provides best platform for Oracle RAC
1212

Oracle Autonomous Database
Self-Driving - User defines service levels, database makes them happen
Self-Tuning - Continuous adaptive performance tuning
Self-Scaling - Instantly resize compute and storage without downtime
Self-Securing - Protection from both external attacks and internal users
Self-Repairing - Automated protection from all downtime
13
Oracle RAC on Exadata plays critical part

Program Agenda
Oracle RAC optimizations on Exadata, In-Memory, Data
Guard
1
2
14

Optimized Oracle RAC features
15
Fast Node Death Detection
Cache Fusion optimization
ASM optimization
Stretch Cluster
In-Memory
Service Oriented Buffer Cache
Multi Instance Redo Apply
Exadata

Fast Node Death Detection

Node Eviction Basics (Generic Systems)
MyCluster
1 2 3 4
S
p
l
i
t
Sub Cluster A Sub Cluster B
For more details check my blog post on the new Oracle RAC 12c Release 2 Node Weighting features

Node Membership depends on
• NHB (Network Heart Beat)
• One HB per second per interface (UDP and GIPC for Cache fusion)
• PollingThread and SendingThread
• Misscount defaults to 30 sec (60 for upgrades)
18
$crsctl get css miscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services
$oifcfg getif
eth0 XX.XXX.XX.0 global public
eth1 X0.XXX.0.0 global cluster_interconnect
eth2 X0.XXX.0.0 global asm

Network HeartBeats
2018-03-13 17:00:20.023: [
CSSD][4096109472]clssnmSendingThread: sending
status msg to all nodes
2018-03-13 17:00:22.818: [
CSSD][4106599328]clssnmPollingThread: node anair2
(2) at 50% heartbeat fatal, removal in 14.520 seconds
2018-03-13 17:00:22.818: [
CSSD][4106599328]clssnmPollingThread: node anair2
(2) is impending reconfig, flag 132108, misstime
15480
19
Network heartbeats sent to other
nodes.
The message is output once every 5
messages (seconds)
Network heartbeat missing from
node 2 (anair2) for 15 consecutive
seconds (15480 ms).

CSS Logging (Private Network failure)
2018-03-13 17:00:29.833: [ CSSD] [4106599328]
clssnmPollingThread: node anair (2) at 75%
heartbeat fatal, removal in 7.500 seconds
… heartbeat fatal, removal in 2.490 seconds
2018-03-13 17:00:37.337: [ CSSD][4106599328]
clssnmPollingThread: Removal started for node
anair2 (2), flags 0x2040c, state 3, wt4c 0
2018-03-13 17:00:37.340: [ CSSD][4085619616]
clssnmCheckSplit: Node 2, anair2, is alive, DHB
(1281744040, 1396854) more than disk timeout of
27000 after the last NHB (1281744011, 1367154)
20
Network heartbeat missing from
node 2 (anair2)
Node eviction proceedings initiated
for node 2 (anair2)
Note: Node 2 (anair2) is updating
Disk (SplitBrain)

Disk Heartbeats helps to resolve SplitBrain
• Disk Heart Beat aka DHB
#crsctl query css votedisk
## STATE File Universal Id File Name Disk group
1. ONLINE ee0c34dfb13f4f31bfc36d551f919c96 (AFD:DATA1) [DATA]
2. ONLINE ee0334dfedu3213b4s3cc33e9303751 (AFD:RECO1) [RECO]
3. ONLINE ee033dfe58564f32b43g94d683e93f1 (AFD:DATA2) [DATA]
• One write and 1 read per second per Voting file
– Each Node with external redundancy will have 1 writers and 2 readers
• Disktimeout defaults to 200 sec
#crsctl get css disktimeout
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
21

CSS logging during Voting Disk I/O failure(s)
2010-08-13 18:31:19.787: [ SKGFD][4131736480]ERROR: -
9(Error 27072, OS Error (Linux Error: 5: Input/output
error
2010-08-13 18:31:19.787: [
CSSD][4131736480](:CSSNM00060:) clssnmvReadBlocks:
read failed at offset 529 of /dev/XXXX
…. (message repeated)
2010-08-13 18:31:23.782: [ CSSD][150477728]
clssnmvDiskOpen: Opening /dev/sdb8
….
2010-08-13 18:34:37.429: [ CSSD][150477728]
(:CSSNM00060:)clssnmvReadBlocks: read failed at offset
17 of /dev/sdb8
22
Error writing to voting disk (OS error 5)
Error message ….Repeats
18:31:19.787
18:34:37.429
~200 Seconds

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 23
CSS heartbeats on Exadata

Fast Node Death detection (NHB)
• CSS is integrated with the subnet manager
to declare node dead in less than ~2
seconds
– No waiting for long heartbeat timeouts
– Reduces application brownouts
– Configured out of the box
*(Subnet manager runs in the switch)
• GIPC posts the node death failure to CSS
– Resulting in quicker reconfiguration and
Application Failover
24
0.8
30
0
5
10
15
20
25
30
35
Exadata Generic Systems
Node death detection
Seconds

Fast Node death detection Logging (Exadata)
• 2018-02-03 13:33:27.724 : CSSD:866105088:
clssnmeventhndlr: EXADATA NodeDeath detection
from GIPC for node xyzadm1 number 1
• 2018-02-03 13:33:27.724 :GIPCHTHR:988976896:
gipchaWorkerProcessClientDelEndp: workerThread
set nodeDeath req, hendp 0x7f9c04088f70
[0000000000000bca]
{ gipchaEndpoint : port 'gm2_xyzadXX/ba3d-0156-
2b59-0f07', peer ‘xyzadm01:cff5-5bd0-d868-2361',
2018-02-03 13:33:28.457 : CSSD:971667200:
clssnmvDiskEvict: Begin: Kill block write, file
o/192.168.175.192/DBFS_CD_09_xyzadm1
25
CSS receives Node death event from
GIPC
And in less than 1 second writes the
kill block and begins eviction
procedures
13:33:27.724
13:33:28.457
00:00:00.733

• Voting disks on Exadata presented
via iDB protocol
– Does not go through normal OS IO path
– Does not rely on OS` path retries
• Additional diskmon process
– One Master Diskmon and one instance
level diskmon slave processes
– Diskmon will signal CSS if it cannot
reach Storage Cells
– Fast Detection of loss of DHB
2017-09-25 22:37:14.417 [OCSSD(63908)]CRS-1605:
CSSD voting file is online:
o/ZZ.XXX.YYY.9;ZZ.YYY.BB.10/DBFSC3_CD_02_admxxx;
o/ZZ.XXX.YYY.7; ZZ.YYY.BB.8/DBFSC3_CD_02_admxxx;
o/ZZ.XXX.YYY.5; ZZ.YYY.BB.6/DBFSC3_CD_02_admxxx;
ps -ef | egrep "diskmon|dskm" | grep -v grep
oracle 3205 1 0 Feb16 ? 00:01:18
ora_dskm_sales2
oracle 10985 1 0 Feb16 ? 00:32:19
/u01/app/12.2.0/grid/bin/diskmon.bin -d -f
oracle 17542 1 0 Feb16 ? 00:01:17
asm_dskm_+ASM2
oracle 24738 1 0 Feb18 ? 00:00:21
ora_dskm_orcl1
26
Fast Node Death detection (DHB) ** Diskmon

Diskmon cell failure detection
2018-04-10 09:29:29.249 : DISKMON:2218510080: dskm_cell_health_resp1: Encounter a split-brain with
node 1, suicide self....
2018-04-10 09:29:29.255 : DISKMON:2218510080: WARNING: Notifying CSSD to ABORT
2018-04-10 09:29:29.254 : DISKMON:1859278592: dskm_hb_thrd_main11: got status change
2018-04-10 09:29:29.259 : DISKMON:1859278592: INFO: Entering Cell Reconnect: rscnam:
o/192.168.216.217;192.168.216.218 rsc: 0x7f6e4005f7e0 state: UNREACHABLE reconn_attempts: 1
last_reconn_ts: 1523377686
CSSD:3368920832: clssnmPollingThread: Removal started for node XXXXXXadm02 (2), flags 0x20222000,
state 3, wt4c 0
CSSD:3368920832: clssnmMarkNodeForRemoval: node 2, XXXXXXXdm02 marked for removal
CSSD:3368920832: clssnmChangeState: oldstate 3 newstate 5 clssnm1.c 2854
28

Near Zero Downtime Reconfiguration with Recovery Buddy
4x
faster! *
• Recovery Buddy
• Tracks blocks changes on buddy
instance
• Quickly identifies blocks requiring
recovery during reconfiguration
• Allows rapid processing of new
transactions
• * up to 4 times faster with Recovery
Buddy and Optimized (Singleton)
Reconfiguration Time
For more details read my blog post on Recovery Buddy

Faster Reconfiguration
30
Detect
Evict
Elect
Recovery
Read
Redo
Apply
Recovery
Detect
Evict
Elect
Recovery
Read
Redo
Apply
Recovery
Detect
Evict
Elect
Recovery
Read
Redo
Apply
Recovery

Cache Fusion optimization

• Efficient IPC (Inter Process
Communications) is critical for
Cache Fusion
• Lost blocks very expensive
– Failed acks
• Redundant Private network
• Low Latency
32
Cache Fusion IPC requirements
Query
BlockBlock
Network
congestion
Cache Fusion 348 Lost
blocks

• InfiniBand has amazing throughput
with very low latency
– But overheard of calling OS on every
message limits small message rate
• Exafusion along with IPCLW
provides blazing performance
• Database directly calls InfiniBand
hardware
– Bypasses networking software stack,
interrupts, scheduling
33
Exafusion Direct-to-Wire OLTP Protocol
8K OLTP Block Transfers/sec
InfiniBandExafusion
InfiniBand
10GigE
3X Speedup

• New IPCLW communication library
for RAC IPC
• NUMA aware resource assignment
• Incorrect resource placement adds
additional hops over QPI which is
considerably slower.
• Reduced overhead (Sockets,
Memory etc)
• New Extended Reliable Connected
XRC Transport Service
– Provides high performance
communications by passing Kernel
from User space.
– Utilizes multiple Virtual Lanes  Cache
fusion traffic utilizes distinct lane
34
Cache Fusion optimizations
Generic Systems

Alert log: Confirm optimal settings
• Cluster Communication is configured to use IPs
from: GPnP
IP: 169.254.249.167 Subnet: 169.254.0.0
• Relies on HAIP
• Private Interface 'bondib0' configured from GPnP
for use as a private interconnect.
• Relies on Bonding for Private network HA
IPCLW over UDP Oracle RDS/IP

In-Memory Fault Tolerance with Oracle RAC on Exadata
36
• Similar to Storage Mirroring
• Duplicate In-Memory Columns
on other nodes
– Enabled per table/partition
– Application transparent
• Benefit from consistent Query
performance after Node Failure
• Exafusion provides the critical
layer for Fault Tolerance feature
on Engineered Systems*
Duplicate

Accelerate In-Memory with Read Only Instances
• Use Oracle RAC to redirect
– OLTP users to RAC instances running on
Hub Nodes and
– DSS queries to utilize parallel query
slaves with In-Memory option to speed
up analytic queries
• Engineered systems benefit from
fault tolerant In-Memory
compression units (IMCU)Parallel Query
Coordinator
Parallel Query Slaves

Integrate with & utilize the
Exadata platform features

Block waits and Block History
1 2 3 4
Shadow Process LGWR
gc current block busy,
gc buffer busy acquire,
gc buffer busy release

How does it looks in AWR?
Node4

• Outliers in log IO slow down lots of clients
• Outliers from any one copy of mirror slow down
all the foregrounds
– Database wait time goes up by #foregrounds * Stall
time
– Backlog doesn’t clear immediately like an accident
on the freeway and increases “log file sync” waits
• Legacy storage IO cannot differentiate redo log IO
from others
• UPS protected cache in traditional storage helps
but will soon get overwhelmed
• Measure log file latency with full backup or a
data load running
41
Log Writer performance depends on underlying Storage
Log Buffer
client
foreground
client
foreground
client
foreground
Log Writer

• Smart Flash Log uses flash as a parallel write
cache to disk controller cache
• Whichever write completes first wins (disk or
flash)
• Reduces response time and outliers
– “log file parallel write” histogram improves
– Greatly improves “log file sync”
• Uses almost no flash capacity (< 0.1%)
• Network resource management provides priority
for redo log I/Os across the network
• OLTP workloads transparently accelerated and
provide predictable response times
42
Exadata Smart Flash Log
Smart Logging - Off Smart Logging - On
No Outliers

• On Exadata, Oracle does not wait for the
log write notification
– Exadata ensures the log write completes before
changes to block on another instance commit,
guaranteeing durability
– Wait for Log I/O during transfer of hot blocks is
eliminated
– Up to 40% throughput and 33% response time
improvement in some heavily contended OLTP
workloads
• Storage software will ensure
correct ordering of writes
43
Smart Fusion Block Transfer
1. Issue log write
2. Wait for log
write completion
3. Transfer
block
Exadata Avoids I/O Wait confirmation
Storage

Exadata Commit Cache
• In-Memory commit cache per DB
Instance that logs commit time
• DB instance checks against this
cache for committed transactions
– Saves blocks being passed around, no
wait for log flush
• RDMA the commit cache from a
remote instance at one shot
improving batching
• Eliminates up to 60% cache fusion
block traffic for batch
44
Right Growing Indexes particularly benefit by this feature

• OLTP sessions require remote
undo header lookups to find
– If a transaction has committed
– Block cleanouts
• To reduce remote look ups, each instance
maintains a hash table of recent
transactions (active & committed)
• Undo Header Hash table improves
scalability by eliminating remote look ups
Undo Header Hash Table
45
• When a query has to roll a change
out of a block to get a consistent
version, and the change was made
on another instance, an undo block
must be fetched from that instance
• Use RDMA to read UNDO blocks
from other instance
• Reduces latency from 50us to 10us

Scalability and Availability in Consolidated Environments
Multitenant • Consolidate even more PDBs per
server for Scalability and
Availability
– Add RAC instances to scale as load on
PDBs increase
– RAC instances prevents downtime
caused by both planned and unplanned
failures
– RAC “Service Isolation” feature
provides consistent performance on
the consolidated PDB instances

Consolidate up to 4,000 PDBs on Exadata
• Oracle RAC optimized on the Exadata
platform provides the perfect environment
for consolidation
– This combination offers unique end-to-end
resource management and consolidation
capabilities
• IORM and Storage cells aware of PDB
• Algorithm ensures single PDB does not hog
Flash
– Exclusively available on Exadata, SuperCluster and
DB PaaS
47
RetailPricing
Multitenant Container

• Storage server to storage server
offload support for ASM Scoped-
Security
– Applies to ASM re-sync, rebalance,
resilver, rebuild and database high
throughput write operations
• Smart redirect for high throughput
write
– Automatically detects and quarantines
any anomalous operation and uses
alternate IO path
48
ASM integration with Exadata

• Intelligent and flexible rebalance power
setting
– Dynamically change ASM_POWER_LIMIT
• ASM rebalance restores redundancy first
– Drastically reduces secondary failure exposure
window
– Exposed via new REBUILD phase in
v$asm_operation
• Exadata leverages flash cache for
rebalance reads
– Improves performance of redundancy
restoration by up to 30%
• XDMG and XDWK background
processes
– Two additional process run in ASM
instances.
– XDMG monitors cells and performs
automatic actions
• Receives events from Cell on Failures
– XDWK performs async actions like Disk
online, Drop, Add etc
49
Efficient Diskgroup Rebalance and Restore

Data Protection with ASM integration
• If an application update in the database encounters corruption
– Database reads from the ASM mirror
– Repairs the corruption using the good copy
 This repair happens without impacting other database processes and application
• When a network packet in the I/O path between DB server and storage node
is corrupted
– Storage cell prevents the write
– ASM retries by re-sending the packet
 Application never encounters corruptions
• Automatic disk inspection and repair when disks are idle
• Hardware Assisted Resilient Data (HARD) compliant checks
50
Corruption Detection, Prevention & Repair

Multi-Instance Redo Apply
recover managed standby database disconnect using instances 3;

Utilize all RAC nodes on standby to apply Redo
• Parallel, multi-instance recovery
means “the standby DB will keep
up”
– Standby recovery - utilizes CPU and I/O
across all nodes of RAC standby
• Multi-Instance Apply runs on all
MOUNTED instances or all OPEN
Instances
• Exposed in the Broker with the
‘ApplyInstances’ property on
standby
• MRP0: Background Managed Standby Recovery
process started
*** 2018-04-17T21:38:10.974841-04:00
Managed Recovery: Initialization posted.
2018-04-17 21:38:10.989 : [ CSSCLNT][
CSSCLNT]clsssinit:
• Watch out for
– Bug 26327521 - ADG: STANDBY HIT ORA 600
[KRPM_CACHE_CV_2], MIRA DONESN'T
SUPPORT 32K DB BLOCK

Summary
5353
Exadata provides best platform for Oracle RAC

• Please remember to provide
feedback
– Session ID Session ID: 1374
– New integration features of Oracle RAC
12c Rel 2 with Exadata, Data Guard and
IMDB
Reminder for tomorrows session on
Wed Apr 25th at 9:45
Confidential – Oracle Internal/Restricted/Highly Restricted 54
Feedback is important

Oracle RAC features on Exadata

Oracle RAC features on Exadata

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Oracle RAC features on Exadata

Similar to Oracle RAC features on Exadata (20)

More from Anil Nair

More from Anil Nair (7)

Recently uploaded

Recently uploaded (20)

Oracle RAC features on Exadata