SlideShare a Scribd company logo
1 of 59
Download to read offline
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle RAC 12c Release 2 and Clusterware
Internals
Anil Nair
Sr. Principal Product Manager,
Oracle Real Application Clusters (RAC)
Dec 25th, 2018
@RACMasterPM,
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/
20 Years of Continuous Innovation
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Please remember to provide
feedback
– Session ID Session ID: 1369
– Oracle RAC 12 Release 2 and
Clusterware Internals
• Reach out to us with follow up
questions, suggestions etc. via
Twitter @RACMasterPM
Linkedin http://www.linkedin.com/in/anil-nair-
01960b6
Feedback is important for successful collaboration
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Cluster Domain Architecture
RAC New Features
Sample Problems and Resolution
FAQ
1
2
4
3
4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Cluster Domain Architecture1
2
5
3
4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle RAC 12c Rel. 2 Cluster Domain
• Simplified Management
– Fleet Management for installation,
update, patching and maintenance
• Reduced Local Overhead
– Member Clusters benefit from the
consolidation of common services
on the Domain Services Cluster
• Improved IO Performance
– Utilizing consolidated shared storage
6
Centralized Management for Cluster Estates “too big to manage” otherwise
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 7
Cluster Domain
Application
Member Cluster
Uses ASM
Database
Member Cluster
Uses local ASM
Database
Member Cluster
Uses ASM
Service
Database
Member Cluster
Uses IO Service
Domain Services Cluster
(DSC)
Shared ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 8
The DSC – The Heart of the Cluster Domain
Domain Services Cluster (DSC)
IO
Service
ASM
Service
TFA
Service
Management
Service
Shared ASM
ACFS
Service
RHP
Service
• The DSC hosts services that are consumed by Member Clusters, including:
– Management Service for centralized and simplified management
– Trace File Analyzer (TFA) for centralized diagnostics
– Rapid Home Provisioning (RHP) for software fleet management
– Storage Services (ACFS, ASM direct or indirect over IO Service)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Domain Services Cluster (DSC)
IO
Service
ASM
Service
Shared ASM
ACFS
Service
RHP
Service
9
The DSC Management Service
TFA
Service
Management
Service
Applied Machine Learning for Database Diagnostics
• Efficient diagnosis using Machine Learning
• Automatically performs corrective actions to
prevent possible issues
• Provides simple alerts & recommendations for
issues that require manual intervention Subject Matter
ExpertLog
ASH
Metrics
ML
Knowledge
Extraction
Model
Generation
Human
Supervision
Application
Optimized
Models
Feedback
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 10
Rapid Home Provisioning Service
Domain Services Cluster (DSC)
IO
Service
ASM
Service
Shared ASM
ACFS
Service
RHP
Service
TFA
Service
Management
Service
• Provision new pools onto base machines
• DB and GI: provision, scale, patch, upgrade
• Custom workflow framework
• Notification model
• Audit capabilities
RHP
Service
Member
Cluster
Member
Cluster
Member
Cluster
Fleet Management installation, update, patching and maintenance
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 11
Domain Services Cluster (DSC) Availability
• Services provided by the DSC are
unaffected by nodes joining/
leaving the DSC cluster
• DSC can be patched and upgraded
independently without affecting
the services provided by DSC
• Use Member Clusters for user
databases (not the DSC)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 12
Cluster Domain
Application
Member Cluster
Uses ASM
Database
Member Cluster
Uses local ASM
Database
Member Cluster
Uses ASM
Service
Database
Member Cluster
Uses IO Service
Domain Services Cluster
(DSC)
Shared ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• A Member Cluster is a Standalone
Cluster utilizing shared services on
the Domain Services Cluster
• It automatically benefits from the
management, TFA & RHP service.
– ASM services are optional
– and can be utilized as needed
13
Member Clusters = Standalone Cluster + Benefits
Database
Member Cluster
Uses local ASM
Database
Member Cluster
Uses ASM
Service of DSC
Database
Member Cluster
Uses ASM IO
Service
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 14
Same Tools and Commands for all Types of Deployments
[GRID]> crsctl get cluster name
CRS-6724: Current cluster name is 'SolarCluster'
[GRID]> crsctl get cluster class
CRS-41008: Cluster class is 'Standalone Cluster'
[GRID]> crsctl get cluster type
CRS-6539: The cluster type is 'flex'.
[GRID]> crsctl get cluster name
CRS-6724: Current cluster name is 'SalesCluster'
[GRID]> crsctl get cluster class
CRS-41008: Cluster class is ‘Database Member Cluster'
[GRID]> crsctl get cluster type
CRS-6539: The cluster type is 'flex'.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Easy conversion from
Standalone to Member Clusters
• Member Cluster GI version can be
higher or equal to the GI version on
the Domain Services Cluster
15
Convert a Standalone Cluster to Member Cluster
Convert
Database
Member Cluster
Uses local ASM
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
RAC New Features
1
2
16
3
4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 17
All Deployments Benefit from Oracle RAC Features
PDB Isolation Hang
Manager
Choose the “right path”
Zero Impact Patching
Scalable Sequences
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
1010101000010011100101001
1011100101001110010100101
0010010100001111010100101
1010101000010011100101001010010100100111001010010100100
1011100101001110010100101111110100111001010010111101101
0010010100001111010100101001010100100110101001010101000
18
• Using Oracle Multitenant, PDBs can be opened
as singletons (in one database instance only), in
a subset of instances or all in instances at once.
• If certain PDBs are only opened on some
instances, Pluggable Database Isolation
– improves performance by
• Reducing DLM operations for
PDBs not open in all instances.
• Optimizing block operations based
on in-memory block separation.
– improves availability by
• Ensuring that instance failures of instances
only hosting singleton PDBs will not impact
other instances of the same RAC-based CDB.
Optimized Singleton Workload Scaling
Pluggable Database Isolation
NEW IN 12.2PRIOR TO 12.2 Inter Cluster Messages
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Database
Member Cluster
Uses ASM Service
19
Pluggable Database and Service Isolation
Provides higher Isolation in Member Cluster Consolidated Environments
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Hang Manager
• Hang Manager “manages” database
process(es) that are not progressing
– Manage = detection and resolution
– Considers cross-layer hangs
• I.E: Hangs caused by a blocked ASM resource.
• In addition, deadlocks are now
resolved by Hang Manager
• User defined control via PL/SQL
• Early Warning exposed via (V$ view)
20
Resolving cross-layer hangs in the Cluster Domain
Database
Member Cluster
Uses ASM IO
Service
IO
Service
ASM
Service
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 21
Resolving cross-layer hangs in the Cluster Domain
Provides higher Availability in Member Cluster Consolidated Environments
Database
Member Cluster
Uses ASM Service
W W
W
B
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle RAC Always Chooses the “Right Path”
22
• RAC determines the optimal path
to serve blocks – network or disk
• SSDs and NVMe storage technology
continue to drive down latency
• e.g. flash storage may provide better
access times to data than the private
network under high load
• RAC takes those statistics into account
Query
BlockBlock
Network
congestion
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Database
Member Cluster
Uses ASM Service
23
The “Right Path” Ensures Performance on Member Clusters
Effectively reducing the impact of competing workloads overwhelming the I/O on the DSC
B
u
s
y
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Zero Impact Patching
• Zero Impact Patching enables patching of the Oracle Grid Infrastructure without
interrupting database operations.
• Patches are applied out-of-place and in a rolling fashion with one node being patched at
a time while the database instance(s) on that node remain up and running.
• Zero Impact Patching supports Oracle Real Application Clusters (RAC) databases on
clusters with two or more nodes.
24
Never take down a database instance to patch Grid Infrastructure
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Operational Details of Zero Impact Patching
• During patch application, the stack is monitored by a special CSSD running out of place
• This CSSD performs Heartbeat operations
• Once the patch is done, the special CSSD concedes control to the CSSD running out of
the patched home
• The steps are performed on all the nodes until the entire cluster is patched.
• Any interruptions in normal activity during patching will trigger the CSSD to evict the
node.
25
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Sequences cause Index block
contention during heavy OLTP
especially on tables which use
sequence values as keys
• This is not RAC specific but the
problem gets amplified in RAC
environments
• The effects of this issue is mitigated
via Scalable Sequences
26
Scalable Sequences
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• New option in Create/Alter Sequence
SCALE {Extend |NoExtend | NoScale }
• When Scale is specified an internal
algorithm uses the instance id and
other session specific information to
generate an offset
• DBAs need to alter existing sequences
• DBA/User/ALL_sequences will reflect
the additional attributes
27
Scalable Sequences Operational details
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Sample Problems and Resolution
1
2
28
3
4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sample Problems and Resolution
29
Problem 1 – Node Eviction
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Node Eviction – Node 1 ocssd.trc [ Part 1 ]
• 2010-08-13 17:00:22.818: [ CSSD][4106599328]
clssnmPollingThread: node anair2 (2) at 50% heartbeat fatal,
removal in 14.520 seconds
• 2010-08-13 17:00:29.833: [ CSSD][4106599328]
clssnmPollingThread: node anair2 (2) at 75% heartbeat fatal,
removal in 7.500 seconds
• 2010-08-13 17:00:37.337: [ CSSD][4106599328]
clssnmPollingThread: Removal started for node anair2 (2), flags
• 2010-08-13 17:00:37.340: [ CSSD][4085619616]clssnmCheckSplit:
Node 2, anair2, is alive, DHB (1281744040, 1396854) more than disk
timeout of 27000 after the last NHB (1281744011, 1367154)
30
Network heartbeat missing
from node 2 for 15
consecutive seconds
Network heartbeat is still
missing
Finally eviction starts
Node 2 is still updating the
Voting disks
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Node Eviction – Node 1 ocssd.trc [ Part 2 ]
• 2010-08-13 17:00:37.340: [
CSSD][4085619616](:CSSNM00007:) clssnmrEvict: Evicting
node 2, anair2, from the cluster in incarnation 169934272,
node birth incarnation 169934271, death incarnation
169934272, stateflags 0x24000
• 2010-08-13 17:01:07.705: [
CSSD][4043389856]clssgmCMReconfig: reconfiguration
successful, incarnation 169934272 with 1 nodes, local node
number 1, master node number 1
31
Eventually Node 2 eviction
process starts
Concluding with a
reconfiguration
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Node Eviction – Node 2 ocssd.trc [ Part 1 ]
• 2010-08-13 17:00:26.213: [ CSSD][4073040800]
clssnmPollingThread: node anair1 (1) at 50% heartbeat fatal,
removal in 14.540 seconds
• 2010-08-13 17:00:40.702: [ CSSD][4073040800]
clssnmPollingThread: Removal started for node anair1 (1), flags
• 2010-08-13 17:00:40.706: [ CSSD][4052061088]clssnmCheckSplit:
Node 1, anair1, is alive, DHB (1281744036, 1243744) more than disk
timeout of 27000 after the last NHB (1281744007, 1214144)
32
Huh!
So who is right?
It also detects that
Node 1 is still updating
the Voting disks
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Node Eviction – Node 2 ocssd.trc [ Part 2 ]
• 2010-08-13 17:00:40.707: [
CSSD][4052061088](:CSSNM00008:)clssnmCheckDskInfo:
Aborting local node to avoid splitbrain. Cohort of 1 nodes
with leader 2, anair2, is smaller than cohort of 1 nodes led
by node 1, anair1, based on map type 2
• 2010-08-13 17:00:40.707: [ CSSD]
[4052061088]###################################
2010-08-13 17:00:40.707: [ CSSD][4052061088]clssscExit:
CSSD aborting from thread clssnmRcfgMgrThread
2010-08-13 17:00:40.707: [ CSSD]
[4052061088]###################################
33
it correctly detects and
aborts local node to
prevent split brain
And does the right thing
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
So what is the conclusion ?
34
Hmmm….I think it is the network.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What does OS Watcher say?
• netstat does not show any issues
# grep "zzz|udpInOverflows|ipReasmFails"
• OSW data itself is missing possibly due to scheduling issues
• Just prior to the issue Top reports
top - 13:23:52 up 25 days, 21:08, 1 user, load average: 3.43, 3.01, 3.02
Cpu(s): 16.8%us, 23.2%sy, 0.0%ni, 56.5%id, 3.1%wa, 0.1%hi, 0.3%si,
Mem: 74027752k total, 73689744k used, 338008k free, 1516k
Swap: 16771852k total, 9069988k used, 7701864k free, 25836528k
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1049 root 11 -5 0 0 0 R 55.0 0.0 0:57.42 [kswapd0]
35
No issues seen in netstat &
traceroute
Oh no! It is swapping
Why is the OSW data
missing?
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sample Problems and Resolution
36
Problem 2 – Slow I/O
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Start the Database on all instances to
simulate physical reads
• Simulate I/O on shared storage
– Loading data (sqlldr,expdp) OR
– Swingbench
• **behavior may depend on your test setup, HCA, HBA
etc
37
Simulate I/O performance issue
swingbench swingbench
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
chactl query diagnosis-db
38
$ chactl query diagnosis -db sales -start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“
2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_1) [detected]
2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_2) [detected]
2017-03-31 10:01:13.0 Database sales DB CPU Utilization (sales_2) [detected]
2017-03-31 10:01:33.0 Database salesDB Log File Switch (sales_1) [detected]
Consolidates and displays
information from all instances
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What does OS Watcher say?
• iostat confirms I/O performance issue
archive/oswiostat/xxxxxxxx_iostat_17.03.31.1000.dat
• Increase in Reads/writes
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 4.00 0.00 32.00 8.00 0.00 0.25 0.25 0.10
xvdb 0.00 0.00 6.50 2.00 146.00 21.00 19.65 0.00 0.24 0.24 0.20
• ---
• ---
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 144.00 434.00 53.00 12.00 0.00 0.45 1.13 0.10
xvdb 0.00 0.00 6.50 219.00 414.00 67.00 13.65 0.00 0.33 9.44 0.20
39
There is more overall I/O as
seen by the increase in the
number of writes, reads
* values may change
depending on test env.
**Output has been formatted
for presentation
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sample Problems and Resolution
40
Problem 3 – CPU bound
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Ensure Grid Infrastructure is
running for at least an hour
• Run some normal load
• Simulate excessive CPU using a CPU
stressing program on 3 nodes
– stress -- C program
41
Simulate CPU load
stress stress stress
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
chactl query diagnosis -cluster
42
$chactl query diagnosis –start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“
2017-03-31 10:01:25.0 Host rwsxxxxx9 Host Memory Consumption [detected]
2017-03-31 10:01:29.0 Host rwsxxxxx0 Host Memory Consumption [detected]
2017-03-31 10:01:23.0 Host rwsxxxxx1 Host Memory Consumption [detected]
Problem: Host Memory Consumption
Description: CHA detected that more memory than expected is consumed on this server. The
memory is not allocated by sessions of this database.
Cause: The Cluster Health Advisor (CHA) detected an increase in memory consumption by other
databases or by applications not connected to a database on this node.
Action: Identify the top memory consumers by using the Cluster Health Monitor (CHM).
Note: This is a 4 node cluster but the memory
consumption issue is reported on 3 of the 4 nodes
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
chatcl actions & resolutions in text or html format
43
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What does OS Watcher say?
• OSWatcher confirms chactl diagnosis
• mpstat (platform dependent)
zzz ***Fri Mar 31 10:10:29 PST 2017
10:10:29 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
10:10:29 all 74.01 0.00 7.95 12.20 0.00 0.13 0.13 0.00 5.10
10:10:29 0 44.89 0.00 8.79 12.09 0.00 0.00 0.00 0.00 31.23
10:10:29 1 56.00 0.00 4.00 18.00 0.00 0.00 0.00 0.00 23.00
• Top also reports the CPU stress program
Tasks: 454 total, 4 running, 450 sleeping, 0 stopped, 0 zombie
Cpu(s): 94.9%us, 4.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20752 racusr 20 0 2498m 45m 39m R 85.9 0.3 752:32.40 stress
44
System is indeed CPU
starved
* values may change
depending on test env.
**Output has been formatted
for presentation
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sample Problems and Resolution
45
Problem 4 – Why did my Database
Instance move to different node?
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Provides common view of cluster-
wide activities in a co-ordinated
fashion
• Customer readable summary of all
actions in a cluster
• Each activity is a summary of the
issue
– Further details are in the trace files
46
Clusterware Activity Log
On Friday, my instances
were running on Nodes
1 & 2, but today it is
only running on Node 1
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 47
crsctl query calog -- Syntax
$ crsctl query calog -h
Usage:
crsctl query calog [-aftertime <after_timestamp>] [-beforetime <before_timestamp>] [-duration <time_interval> | -
follow] [-filter <filter_expression>] [-fullfmt | -xmlfmt]
Lists the cluster activity log activities matching the specified criteria
Where
after_timestamp Time in format "YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]" OR "YYYY-MM-DD" OR
"HH24:MI:SS[.FF][[+-]HH:MM]"
before_timestamp Time in format "YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]" OR "YYYY-MM-DD" OR
"HH24:MI:SS[.FF][[+-]HH:MM]"
time_interval Time interval in format "DD HH:MM:SS"
filter_expression Refer to product documentation.
-follow Gives continuous output
-fullfmt Detail output format
-xmlfmt XML output format
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 48
crsctl query calog output
$ crsctl query calog -aftertime "2017-03-08 15:09:46.522-07:00"
2017-04-12 20:05:04.668000 : Attempting to start 'ora.anair1.vip' on ‘anair1' :
14920191617156230/1194/11 :
…..
2017-04-12 20:05:06.559000 : Attempting to start 'ora.LISTENER.lsnr' on ‘anair1' :
14920191617156230/1194/16
…
2017-04-12 20:05:32.038000 : Start of 'ora.FRA.dg' on ‘anair1' succeeded : 14920191617156230/1194/27
:
2017-04-12 20:05:32.040000 : Attempting to start 'ora.sales.db' on ‘anair1' :
14920191617156230/1194/28 :
2017-04-12 20:05:59.415000 : Start of 'ora.sales.db' on ‘anair1' succeeded :
14920191617156230/1194/30 :
-- Format of output records is:
DATE & TIME (YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]): Event text: ACTID
Possibly network issues
caused VIP relocation
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 49
crsctl query calog filter output using actid
$ crsctl query calog -filter "actid == 14920191617156230/2449732/1"
2017-08-03 16:30:24.678000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14920191617156230/2449732/1 :
2017-08-03 16:30:24.698000 : Start of 'ora.sscdb.db' on ‘anair1' succeeded : 14920191617156230/2449732/1 :
$ crsctl query calog -filter "actid ~= “14920191617156230”
2017-08-03 16:25:20.658000 : Stop of 'ora.sscdb.test.svc' on anair1' succeeded : 14920191617156230/2449007/2 :
-filter: Use ~= or == on actid to find related actions
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
crsctl query calog filtered output using name
$crsctl query calog -filter "entity_name == ora.sscdb.db"
2017-04-12 18:59:28.330000 : Attempting to start 'ora.sscdb.db' on ‘anair2' : 14919830844562257/9074/1 :
2017-04-12 18:59:28.334000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14919830844562257/9074/2 :
2017-04-12 18:59:28.340000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14919830844562257/9074/3 :
2017-04-12 18:59:28.694000 : Resource 'ora.sscdb.db' has been modified. : 14919830844562257/9074/4 :
2017-04-12 18:59:28.800000 : Resource 'ora.sscdb.db' has been modified. : 14919830844562257/9074/5 :
…
2017-04-12 19:32:39.631000 : Attempting to stop 'ora.sscdb.db' on ‘anair2' : 14919830844562257/9853/2 :
2017-04-12 19:32:42.572000 : Stop of 'ora.sscdb.db' on ‘anair2' succeeded : 14919830844562257/9853/4 :
2017-04-12 19:45:01.175000 : Attempting to stop 'ora.sscdb.db' on ‘anair1' : 14919830844562257/10151/4 :
2017-04-12 19:45:04.176000 : Stop of 'ora.sscdb.db' on ‘anair1' succeeded : 14919830844562257/10151/7 :
50
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Additional examples for querying calog
• crsctl query calog -duration "00 00:02:00“ << last two minutes >>
• crsctl query calog -duration "00 02:00:00“ << last two hours >>
• crsctl query calog –follow << like tail… >>
• crsctl query calog -aftertime "2017-03-08 17:49:12.397" –follow
• crsctl query calog -filter "entity_name == foo“ << activity on foo >>
• crsctl query calog -filter "actid == 145567015239630019/49543/1"
• crsctl query calog –xmlfmt << in xml format >>
51
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
FAQ
1
2
52
3
4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 53
Frequently asked Question # 1
Why does MGMT
DB need so much
space?
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Cluster Type Redundancy MGMT DG
(GB)
Domain Services Cluster
(2 Node DSC with 4 Member
Clusters of 2 Nodes each)
External 188
Normal 376
High 564
Flex 376
Standalone Cluster
(4 Node Cluster)
External 38
Normal 76
High 114
Flex 76
• Oracle GI 12c Release 2 feature AHF
(Autonomous Health Framework)
collects, Co-relates & stores diagnostics
data from OS and DB in MGMT
• In DSC, one PDB per member cluster is
provisioned to store member cluster
diagnostics data
• The data is used by AHF components like
cluster health advisor to both prevent
and help diagnose issues
54
GIMR space requirements
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 55
Frequently asked Question # 2
Should I continue
to use
Oswatcher?
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Should I continue to use OSWatcher?
• Hopefully by now, the value benefits of AHF is clear
• AHF continues to be enhanced to improve
– Diagnosing large number of deployments
– Correlating data to speed diagnosis
– Preventing issues in the first place
– Utilizing latest technologies like Machine learning
• AHF is mandatory (12.2+) but customers can choose to use OSW
along with AHF
56
Learn more about Autonomous Health Framework
https://www.slideshare.net/MarkusMichalewicz/cl
uster-health-advisor-cha-deep-dive-by-mark-
scardina
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• opatch automatically patches
MGMT database if required
• clients of MGMT connect using
encrypted credentials
• MGMT listener automatically
maintained by the clusterware
agent
57
Frequently asked Question # 3
But it is still one
more database
for me to manage
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 58
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 59

More Related Content

What's hot

Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016Anil Nair
 
New Generation Oracle RAC Performance
New Generation Oracle RAC PerformanceNew Generation Oracle RAC Performance
New Generation Oracle RAC PerformanceAnil Nair
 
Using Machine Learning to Debug complex Oracle RAC Issues
Using Machine Learning  to Debug complex Oracle RAC IssuesUsing Machine Learning  to Debug complex Oracle RAC Issues
Using Machine Learning to Debug complex Oracle RAC IssuesAnil Nair
 
Exadata master series_asm_2020
Exadata master series_asm_2020Exadata master series_asm_2020
Exadata master series_asm_2020Anil Nair
 
Collaborate 17 Oracle RAC 12cRel 2 Best Practices
Collaborate 17 Oracle RAC 12cRel 2 Best PracticesCollaborate 17 Oracle RAC 12cRel 2 Best Practices
Collaborate 17 Oracle RAC 12cRel 2 Best PracticesAnil Nair
 
Oracle RAC BP for Upgrade & More by Anil Nair and Markus Michalewicz
Oracle RAC BP for Upgrade & More by Anil Nair and Markus MichalewiczOracle RAC BP for Upgrade & More by Anil Nair and Markus Michalewicz
Oracle RAC BP for Upgrade & More by Anil Nair and Markus MichalewiczMarkus Michalewicz
 
Understanding oracle rac internals part 1 - slides
Understanding oracle rac internals   part 1 - slidesUnderstanding oracle rac internals   part 1 - slides
Understanding oracle rac internals part 1 - slidesMohamed Farouk
 
Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]
Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]
Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]Markus Michalewicz
 
Understanding oracle rac internals part 2 - slides
Understanding oracle rac internals   part 2 - slidesUnderstanding oracle rac internals   part 2 - slides
Understanding oracle rac internals part 2 - slidesMohamed Farouk
 
Oracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - OverviewOracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - OverviewMarkus Michalewicz
 
Oracle RAC 12c Rel. 2 & Cluster Architecture Internals OOW17 by Anil Nair
Oracle RAC 12c Rel. 2 & Cluster Architecture Internals OOW17 by Anil NairOracle RAC 12c Rel. 2 & Cluster Architecture Internals OOW17 by Anil Nair
Oracle RAC 12c Rel. 2 & Cluster Architecture Internals OOW17 by Anil NairMarkus Michalewicz
 
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 VersionOracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 VersionMarkus Michalewicz
 
Oracle RAC 12c Rel. 2 for Continuous Availability
Oracle RAC 12c Rel. 2 for Continuous AvailabilityOracle RAC 12c Rel. 2 for Continuous Availability
Oracle RAC 12c Rel. 2 for Continuous AvailabilityMarkus Michalewicz
 
What's New and Coming in Oracle ASM 12c Rel. 2 - by Jim Williams
What's New and Coming in Oracle ASM 12c Rel. 2 - by Jim WilliamsWhat's New and Coming in Oracle ASM 12c Rel. 2 - by Jim Williams
What's New and Coming in Oracle ASM 12c Rel. 2 - by Jim WilliamsMarkus Michalewicz
 
Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013Markus Michalewicz
 
Expert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RACExpert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RACSolarWinds
 
TFA Collector - what can one do with it
TFA Collector - what can one do with it TFA Collector - what can one do with it
TFA Collector - what can one do with it Sandesh Rao
 
Oracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion EditionOracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion EditionMarkus Michalewicz
 
Oracle RAC and Your Way to the Cloud by Angelo Pruscino
Oracle RAC and Your Way to the Cloud by Angelo PruscinoOracle RAC and Your Way to the Cloud by Angelo Pruscino
Oracle RAC and Your Way to the Cloud by Angelo PruscinoMarkus Michalewicz
 

What's hot (20)

Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016
 
New Generation Oracle RAC Performance
New Generation Oracle RAC PerformanceNew Generation Oracle RAC Performance
New Generation Oracle RAC Performance
 
Using Machine Learning to Debug complex Oracle RAC Issues
Using Machine Learning  to Debug complex Oracle RAC IssuesUsing Machine Learning  to Debug complex Oracle RAC Issues
Using Machine Learning to Debug complex Oracle RAC Issues
 
Exadata master series_asm_2020
Exadata master series_asm_2020Exadata master series_asm_2020
Exadata master series_asm_2020
 
Collaborate 17 Oracle RAC 12cRel 2 Best Practices
Collaborate 17 Oracle RAC 12cRel 2 Best PracticesCollaborate 17 Oracle RAC 12cRel 2 Best Practices
Collaborate 17 Oracle RAC 12cRel 2 Best Practices
 
Oracle RAC BP for Upgrade & More by Anil Nair and Markus Michalewicz
Oracle RAC BP for Upgrade & More by Anil Nair and Markus MichalewiczOracle RAC BP for Upgrade & More by Anil Nair and Markus Michalewicz
Oracle RAC BP for Upgrade & More by Anil Nair and Markus Michalewicz
 
Understanding oracle rac internals part 1 - slides
Understanding oracle rac internals   part 1 - slidesUnderstanding oracle rac internals   part 1 - slides
Understanding oracle rac internals part 1 - slides
 
Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]
Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]
Oracle RAC 12c Practical Performance Management and Tuning OOW13 [CON8825]
 
Understanding oracle rac internals part 2 - slides
Understanding oracle rac internals   part 2 - slidesUnderstanding oracle rac internals   part 2 - slides
Understanding oracle rac internals part 2 - slides
 
Oracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - OverviewOracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - Overview
 
Oracle RAC 12c Rel. 2 & Cluster Architecture Internals OOW17 by Anil Nair
Oracle RAC 12c Rel. 2 & Cluster Architecture Internals OOW17 by Anil NairOracle RAC 12c Rel. 2 & Cluster Architecture Internals OOW17 by Anil Nair
Oracle RAC 12c Rel. 2 & Cluster Architecture Internals OOW17 by Anil Nair
 
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 VersionOracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
 
Oracle RAC 12c Overview
Oracle RAC 12c OverviewOracle RAC 12c Overview
Oracle RAC 12c Overview
 
Oracle RAC 12c Rel. 2 for Continuous Availability
Oracle RAC 12c Rel. 2 for Continuous AvailabilityOracle RAC 12c Rel. 2 for Continuous Availability
Oracle RAC 12c Rel. 2 for Continuous Availability
 
What's New and Coming in Oracle ASM 12c Rel. 2 - by Jim Williams
What's New and Coming in Oracle ASM 12c Rel. 2 - by Jim WilliamsWhat's New and Coming in Oracle ASM 12c Rel. 2 - by Jim Williams
What's New and Coming in Oracle ASM 12c Rel. 2 - by Jim Williams
 
Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013
 
Expert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RACExpert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RAC
 
TFA Collector - what can one do with it
TFA Collector - what can one do with it TFA Collector - what can one do with it
TFA Collector - what can one do with it
 
Oracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion EditionOracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion Edition
 
Oracle RAC and Your Way to the Cloud by Angelo Pruscino
Oracle RAC and Your Way to the Cloud by Angelo PruscinoOracle RAC and Your Way to the Cloud by Angelo Pruscino
Oracle RAC and Your Way to the Cloud by Angelo Pruscino
 

Similar to Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf

AUSOUG - NZOUG-GroundBreakers-Jun 2019 - 19c RAC
AUSOUG - NZOUG-GroundBreakers-Jun 2019 - 19c RACAUSOUG - NZOUG-GroundBreakers-Jun 2019 - 19c RAC
AUSOUG - NZOUG-GroundBreakers-Jun 2019 - 19c RACSandesh Rao
 
AIOUG-GroundBreakers-Jul 2019 - 19c RAC
AIOUG-GroundBreakers-Jul 2019 - 19c RACAIOUG-GroundBreakers-Jul 2019 - 19c RAC
AIOUG-GroundBreakers-Jul 2019 - 19c RACSandesh Rao
 
The Top 5 Reasons to Deploy Your Applications on Oracle RAC
The Top 5 Reasons to Deploy Your Applications on Oracle RACThe Top 5 Reasons to Deploy Your Applications on Oracle RAC
The Top 5 Reasons to Deploy Your Applications on Oracle RACMarkus Michalewicz
 
C6 oracles storage_strategy_from_databases_to_engineered_systems_to_cloud
C6 oracles storage_strategy_from_databases_to_engineered_systems_to_cloudC6 oracles storage_strategy_from_databases_to_engineered_systems_to_cloud
C6 oracles storage_strategy_from_databases_to_engineered_systems_to_cloudDr. Wilfred Lin (Ph.D.)
 
Using Snap Clone with Enterprise Manager 12c
Using Snap Clone with Enterprise Manager 12cUsing Snap Clone with Enterprise Manager 12c
Using Snap Clone with Enterprise Manager 12cPete Sharman
 
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - PresentationThe Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - PresentationMarkus Michalewicz
 
TechEvent 2019: Create a Private Database Cloud in the Public Cloud using the...
TechEvent 2019: Create a Private Database Cloud in the Public Cloud using the...TechEvent 2019: Create a Private Database Cloud in the Public Cloud using the...
TechEvent 2019: Create a Private Database Cloud in the Public Cloud using the...Trivadis
 
20191001 bkk-secret-of inno-db_clusterv1
20191001 bkk-secret-of inno-db_clusterv120191001 bkk-secret-of inno-db_clusterv1
20191001 bkk-secret-of inno-db_clusterv1Ivan Ma
 
Mysql NDB Cluster's Asynchronous Parallel Design for High Performance
Mysql NDB Cluster's Asynchronous Parallel Design for High PerformanceMysql NDB Cluster's Asynchronous Parallel Design for High Performance
Mysql NDB Cluster's Asynchronous Parallel Design for High PerformanceBernd Ocklin
 
Přehled portfolia ODA a praktických případů v regionu EMEA
Přehled portfolia ODA a praktických případů v regionu EMEAPřehled portfolia ODA a praktických případů v regionu EMEA
Přehled portfolia ODA a praktických případů v regionu EMEAMarketingArrowECS_CZ
 
What's New in MySQL 8.0 @ HKOSC 2017
What's New in MySQL 8.0 @ HKOSC 2017What's New in MySQL 8.0 @ HKOSC 2017
What's New in MySQL 8.0 @ HKOSC 2017Ivan Ma
 
Meetup Oracle Database MAD_BCN: 1.1 Servicios de Oracle Database en la nube
 Meetup Oracle Database MAD_BCN: 1.1 Servicios de Oracle Database en la nube Meetup Oracle Database MAD_BCN: 1.1 Servicios de Oracle Database en la nube
Meetup Oracle Database MAD_BCN: 1.1 Servicios de Oracle Database en la nubeavanttic Consultoría Tecnológica
 
Why MySQL High Availability Matters
Why MySQL High Availability MattersWhy MySQL High Availability Matters
Why MySQL High Availability MattersMatt Lord
 
MySQL 8.0 in a nutshell
MySQL 8.0 in a nutshellMySQL 8.0 in a nutshell
MySQL 8.0 in a nutshellOracleMySQL
 
State ofdolphin short
State ofdolphin shortState ofdolphin short
State ofdolphin shortMandy Ang
 
OOW16 - Oracle Enterprise Manager 13c Cloud Control for Managing Oracle E-Bus...
OOW16 - Oracle Enterprise Manager 13c Cloud Control for Managing Oracle E-Bus...OOW16 - Oracle Enterprise Manager 13c Cloud Control for Managing Oracle E-Bus...
OOW16 - Oracle Enterprise Manager 13c Cloud Control for Managing Oracle E-Bus...vasuballa
 
#PCMVision: Oracle Hybrid Cloud Solutions
#PCMVision: Oracle Hybrid Cloud Solutions#PCMVision: Oracle Hybrid Cloud Solutions
#PCMVision: Oracle Hybrid Cloud SolutionsPCM
 
6° Sessione Oracle - CRUI: Oracle Database Appliance: Il potere dell’ingegner...
6° Sessione Oracle - CRUI: Oracle Database Appliance:Il potere dell’ingegner...6° Sessione Oracle - CRUI: Oracle Database Appliance:Il potere dell’ingegner...
6° Sessione Oracle - CRUI: Oracle Database Appliance: Il potere dell’ingegner...Jürgen Ambrosi
 
Oracle Extended Clusters for Oracle RAC
Oracle Extended Clusters for Oracle RACOracle Extended Clusters for Oracle RAC
Oracle Extended Clusters for Oracle RACMarkus Michalewicz
 

Similar to Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf (20)

AUSOUG - NZOUG-GroundBreakers-Jun 2019 - 19c RAC
AUSOUG - NZOUG-GroundBreakers-Jun 2019 - 19c RACAUSOUG - NZOUG-GroundBreakers-Jun 2019 - 19c RAC
AUSOUG - NZOUG-GroundBreakers-Jun 2019 - 19c RAC
 
AIOUG-GroundBreakers-Jul 2019 - 19c RAC
AIOUG-GroundBreakers-Jul 2019 - 19c RACAIOUG-GroundBreakers-Jul 2019 - 19c RAC
AIOUG-GroundBreakers-Jul 2019 - 19c RAC
 
The Top 5 Reasons to Deploy Your Applications on Oracle RAC
The Top 5 Reasons to Deploy Your Applications on Oracle RACThe Top 5 Reasons to Deploy Your Applications on Oracle RAC
The Top 5 Reasons to Deploy Your Applications on Oracle RAC
 
C6 oracles storage_strategy_from_databases_to_engineered_systems_to_cloud
C6 oracles storage_strategy_from_databases_to_engineered_systems_to_cloudC6 oracles storage_strategy_from_databases_to_engineered_systems_to_cloud
C6 oracles storage_strategy_from_databases_to_engineered_systems_to_cloud
 
Using Snap Clone with Enterprise Manager 12c
Using Snap Clone with Enterprise Manager 12cUsing Snap Clone with Enterprise Manager 12c
Using Snap Clone with Enterprise Manager 12c
 
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - PresentationThe Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - Presentation
 
TechEvent 2019: Create a Private Database Cloud in the Public Cloud using the...
TechEvent 2019: Create a Private Database Cloud in the Public Cloud using the...TechEvent 2019: Create a Private Database Cloud in the Public Cloud using the...
TechEvent 2019: Create a Private Database Cloud in the Public Cloud using the...
 
20191001 bkk-secret-of inno-db_clusterv1
20191001 bkk-secret-of inno-db_clusterv120191001 bkk-secret-of inno-db_clusterv1
20191001 bkk-secret-of inno-db_clusterv1
 
Mysql NDB Cluster's Asynchronous Parallel Design for High Performance
Mysql NDB Cluster's Asynchronous Parallel Design for High PerformanceMysql NDB Cluster's Asynchronous Parallel Design for High Performance
Mysql NDB Cluster's Asynchronous Parallel Design for High Performance
 
Přehled portfolia ODA a praktických případů v regionu EMEA
Přehled portfolia ODA a praktických případů v regionu EMEAPřehled portfolia ODA a praktických případů v regionu EMEA
Přehled portfolia ODA a praktických případů v regionu EMEA
 
What's New in MySQL 8.0 @ HKOSC 2017
What's New in MySQL 8.0 @ HKOSC 2017What's New in MySQL 8.0 @ HKOSC 2017
What's New in MySQL 8.0 @ HKOSC 2017
 
Oracle Storage a ochrana dat
Oracle Storage a ochrana datOracle Storage a ochrana dat
Oracle Storage a ochrana dat
 
Meetup Oracle Database MAD_BCN: 1.1 Servicios de Oracle Database en la nube
 Meetup Oracle Database MAD_BCN: 1.1 Servicios de Oracle Database en la nube Meetup Oracle Database MAD_BCN: 1.1 Servicios de Oracle Database en la nube
Meetup Oracle Database MAD_BCN: 1.1 Servicios de Oracle Database en la nube
 
Why MySQL High Availability Matters
Why MySQL High Availability MattersWhy MySQL High Availability Matters
Why MySQL High Availability Matters
 
MySQL 8.0 in a nutshell
MySQL 8.0 in a nutshellMySQL 8.0 in a nutshell
MySQL 8.0 in a nutshell
 
State ofdolphin short
State ofdolphin shortState ofdolphin short
State ofdolphin short
 
OOW16 - Oracle Enterprise Manager 13c Cloud Control for Managing Oracle E-Bus...
OOW16 - Oracle Enterprise Manager 13c Cloud Control for Managing Oracle E-Bus...OOW16 - Oracle Enterprise Manager 13c Cloud Control for Managing Oracle E-Bus...
OOW16 - Oracle Enterprise Manager 13c Cloud Control for Managing Oracle E-Bus...
 
#PCMVision: Oracle Hybrid Cloud Solutions
#PCMVision: Oracle Hybrid Cloud Solutions#PCMVision: Oracle Hybrid Cloud Solutions
#PCMVision: Oracle Hybrid Cloud Solutions
 
6° Sessione Oracle - CRUI: Oracle Database Appliance: Il potere dell’ingegner...
6° Sessione Oracle - CRUI: Oracle Database Appliance:Il potere dell’ingegner...6° Sessione Oracle - CRUI: Oracle Database Appliance:Il potere dell’ingegner...
6° Sessione Oracle - CRUI: Oracle Database Appliance: Il potere dell’ingegner...
 
Oracle Extended Clusters for Oracle RAC
Oracle Extended Clusters for Oracle RACOracle Extended Clusters for Oracle RAC
Oracle Extended Clusters for Oracle RAC
 

Recently uploaded

IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 

Recently uploaded (20)

IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 

Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf

  • 1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Oracle RAC 12c Release 2 and Clusterware Internals Anil Nair Sr. Principal Product Manager, Oracle Real Application Clusters (RAC) Dec 25th, 2018 @RACMasterPM, http://www.linkedin.com/in/anil-nair-01960b6 http://www.slideshare.net/AnilNair27/ 20 Years of Continuous Innovation
  • 2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2
  • 3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Please remember to provide feedback – Session ID Session ID: 1369 – Oracle RAC 12 Release 2 and Clusterware Internals • Reach out to us with follow up questions, suggestions etc. via Twitter @RACMasterPM Linkedin http://www.linkedin.com/in/anil-nair- 01960b6 Feedback is important for successful collaboration
  • 4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda Cluster Domain Architecture RAC New Features Sample Problems and Resolution FAQ 1 2 4 3 4
  • 5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda Cluster Domain Architecture1 2 5 3 4
  • 6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Oracle RAC 12c Rel. 2 Cluster Domain • Simplified Management – Fleet Management for installation, update, patching and maintenance • Reduced Local Overhead – Member Clusters benefit from the consolidation of common services on the Domain Services Cluster • Improved IO Performance – Utilizing consolidated shared storage 6 Centralized Management for Cluster Estates “too big to manage” otherwise
  • 7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 7 Cluster Domain Application Member Cluster Uses ASM Database Member Cluster Uses local ASM Database Member Cluster Uses ASM Service Database Member Cluster Uses IO Service Domain Services Cluster (DSC) Shared ASM IO Service ACFS Service ASM Service TFA Service Management Service RHP Service
  • 8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 8 The DSC – The Heart of the Cluster Domain Domain Services Cluster (DSC) IO Service ASM Service TFA Service Management Service Shared ASM ACFS Service RHP Service • The DSC hosts services that are consumed by Member Clusters, including: – Management Service for centralized and simplified management – Trace File Analyzer (TFA) for centralized diagnostics – Rapid Home Provisioning (RHP) for software fleet management – Storage Services (ACFS, ASM direct or indirect over IO Service)
  • 9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Domain Services Cluster (DSC) IO Service ASM Service Shared ASM ACFS Service RHP Service 9 The DSC Management Service TFA Service Management Service Applied Machine Learning for Database Diagnostics • Efficient diagnosis using Machine Learning • Automatically performs corrective actions to prevent possible issues • Provides simple alerts & recommendations for issues that require manual intervention Subject Matter ExpertLog ASH Metrics ML Knowledge Extraction Model Generation Human Supervision Application Optimized Models Feedback
  • 10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 10 Rapid Home Provisioning Service Domain Services Cluster (DSC) IO Service ASM Service Shared ASM ACFS Service RHP Service TFA Service Management Service • Provision new pools onto base machines • DB and GI: provision, scale, patch, upgrade • Custom workflow framework • Notification model • Audit capabilities RHP Service Member Cluster Member Cluster Member Cluster Fleet Management installation, update, patching and maintenance
  • 11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 11 Domain Services Cluster (DSC) Availability • Services provided by the DSC are unaffected by nodes joining/ leaving the DSC cluster • DSC can be patched and upgraded independently without affecting the services provided by DSC • Use Member Clusters for user databases (not the DSC)
  • 12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 12 Cluster Domain Application Member Cluster Uses ASM Database Member Cluster Uses local ASM Database Member Cluster Uses ASM Service Database Member Cluster Uses IO Service Domain Services Cluster (DSC) Shared ASM IO Service ACFS Service ASM Service TFA Service Management Service RHP Service
  • 13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • A Member Cluster is a Standalone Cluster utilizing shared services on the Domain Services Cluster • It automatically benefits from the management, TFA & RHP service. – ASM services are optional – and can be utilized as needed 13 Member Clusters = Standalone Cluster + Benefits Database Member Cluster Uses local ASM Database Member Cluster Uses ASM Service of DSC Database Member Cluster Uses ASM IO Service
  • 14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 14 Same Tools and Commands for all Types of Deployments [GRID]> crsctl get cluster name CRS-6724: Current cluster name is 'SolarCluster' [GRID]> crsctl get cluster class CRS-41008: Cluster class is 'Standalone Cluster' [GRID]> crsctl get cluster type CRS-6539: The cluster type is 'flex'. [GRID]> crsctl get cluster name CRS-6724: Current cluster name is 'SalesCluster' [GRID]> crsctl get cluster class CRS-41008: Cluster class is ‘Database Member Cluster' [GRID]> crsctl get cluster type CRS-6539: The cluster type is 'flex'.
  • 15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Easy conversion from Standalone to Member Clusters • Member Cluster GI version can be higher or equal to the GI version on the Domain Services Cluster 15 Convert a Standalone Cluster to Member Cluster Convert Database Member Cluster Uses local ASM
  • 16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda RAC New Features 1 2 16 3 4
  • 17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 17 All Deployments Benefit from Oracle RAC Features PDB Isolation Hang Manager Choose the “right path” Zero Impact Patching Scalable Sequences
  • 18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 1010101000010011100101001 1011100101001110010100101 0010010100001111010100101 1010101000010011100101001010010100100111001010010100100 1011100101001110010100101111110100111001010010111101101 0010010100001111010100101001010100100110101001010101000 18 • Using Oracle Multitenant, PDBs can be opened as singletons (in one database instance only), in a subset of instances or all in instances at once. • If certain PDBs are only opened on some instances, Pluggable Database Isolation – improves performance by • Reducing DLM operations for PDBs not open in all instances. • Optimizing block operations based on in-memory block separation. – improves availability by • Ensuring that instance failures of instances only hosting singleton PDBs will not impact other instances of the same RAC-based CDB. Optimized Singleton Workload Scaling Pluggable Database Isolation NEW IN 12.2PRIOR TO 12.2 Inter Cluster Messages
  • 19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Database Member Cluster Uses ASM Service 19 Pluggable Database and Service Isolation Provides higher Isolation in Member Cluster Consolidated Environments
  • 20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Hang Manager • Hang Manager “manages” database process(es) that are not progressing – Manage = detection and resolution – Considers cross-layer hangs • I.E: Hangs caused by a blocked ASM resource. • In addition, deadlocks are now resolved by Hang Manager • User defined control via PL/SQL • Early Warning exposed via (V$ view) 20 Resolving cross-layer hangs in the Cluster Domain Database Member Cluster Uses ASM IO Service IO Service ASM Service
  • 21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 21 Resolving cross-layer hangs in the Cluster Domain Provides higher Availability in Member Cluster Consolidated Environments Database Member Cluster Uses ASM Service W W W B
  • 22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Oracle RAC Always Chooses the “Right Path” 22 • RAC determines the optimal path to serve blocks – network or disk • SSDs and NVMe storage technology continue to drive down latency • e.g. flash storage may provide better access times to data than the private network under high load • RAC takes those statistics into account Query BlockBlock Network congestion
  • 23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Database Member Cluster Uses ASM Service 23 The “Right Path” Ensures Performance on Member Clusters Effectively reducing the impact of competing workloads overwhelming the I/O on the DSC B u s y
  • 24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Zero Impact Patching • Zero Impact Patching enables patching of the Oracle Grid Infrastructure without interrupting database operations. • Patches are applied out-of-place and in a rolling fashion with one node being patched at a time while the database instance(s) on that node remain up and running. • Zero Impact Patching supports Oracle Real Application Clusters (RAC) databases on clusters with two or more nodes. 24 Never take down a database instance to patch Grid Infrastructure
  • 25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Operational Details of Zero Impact Patching • During patch application, the stack is monitored by a special CSSD running out of place • This CSSD performs Heartbeat operations • Once the patch is done, the special CSSD concedes control to the CSSD running out of the patched home • The steps are performed on all the nodes until the entire cluster is patched. • Any interruptions in normal activity during patching will trigger the CSSD to evict the node. 25
  • 26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Sequences cause Index block contention during heavy OLTP especially on tables which use sequence values as keys • This is not RAC specific but the problem gets amplified in RAC environments • The effects of this issue is mitigated via Scalable Sequences 26 Scalable Sequences
  • 27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • New option in Create/Alter Sequence SCALE {Extend |NoExtend | NoScale } • When Scale is specified an internal algorithm uses the instance id and other session specific information to generate an offset • DBAs need to alter existing sequences • DBA/User/ALL_sequences will reflect the additional attributes 27 Scalable Sequences Operational details
  • 28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda Sample Problems and Resolution 1 2 28 3 4
  • 29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Sample Problems and Resolution 29 Problem 1 – Node Eviction
  • 30. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Node Eviction – Node 1 ocssd.trc [ Part 1 ] • 2010-08-13 17:00:22.818: [ CSSD][4106599328] clssnmPollingThread: node anair2 (2) at 50% heartbeat fatal, removal in 14.520 seconds • 2010-08-13 17:00:29.833: [ CSSD][4106599328] clssnmPollingThread: node anair2 (2) at 75% heartbeat fatal, removal in 7.500 seconds • 2010-08-13 17:00:37.337: [ CSSD][4106599328] clssnmPollingThread: Removal started for node anair2 (2), flags • 2010-08-13 17:00:37.340: [ CSSD][4085619616]clssnmCheckSplit: Node 2, anair2, is alive, DHB (1281744040, 1396854) more than disk timeout of 27000 after the last NHB (1281744011, 1367154) 30 Network heartbeat missing from node 2 for 15 consecutive seconds Network heartbeat is still missing Finally eviction starts Node 2 is still updating the Voting disks
  • 31. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Node Eviction – Node 1 ocssd.trc [ Part 2 ] • 2010-08-13 17:00:37.340: [ CSSD][4085619616](:CSSNM00007:) clssnmrEvict: Evicting node 2, anair2, from the cluster in incarnation 169934272, node birth incarnation 169934271, death incarnation 169934272, stateflags 0x24000 • 2010-08-13 17:01:07.705: [ CSSD][4043389856]clssgmCMReconfig: reconfiguration successful, incarnation 169934272 with 1 nodes, local node number 1, master node number 1 31 Eventually Node 2 eviction process starts Concluding with a reconfiguration
  • 32. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Node Eviction – Node 2 ocssd.trc [ Part 1 ] • 2010-08-13 17:00:26.213: [ CSSD][4073040800] clssnmPollingThread: node anair1 (1) at 50% heartbeat fatal, removal in 14.540 seconds • 2010-08-13 17:00:40.702: [ CSSD][4073040800] clssnmPollingThread: Removal started for node anair1 (1), flags • 2010-08-13 17:00:40.706: [ CSSD][4052061088]clssnmCheckSplit: Node 1, anair1, is alive, DHB (1281744036, 1243744) more than disk timeout of 27000 after the last NHB (1281744007, 1214144) 32 Huh! So who is right? It also detects that Node 1 is still updating the Voting disks
  • 33. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Node Eviction – Node 2 ocssd.trc [ Part 2 ] • 2010-08-13 17:00:40.707: [ CSSD][4052061088](:CSSNM00008:)clssnmCheckDskInfo: Aborting local node to avoid splitbrain. Cohort of 1 nodes with leader 2, anair2, is smaller than cohort of 1 nodes led by node 1, anair1, based on map type 2 • 2010-08-13 17:00:40.707: [ CSSD] [4052061088]################################### 2010-08-13 17:00:40.707: [ CSSD][4052061088]clssscExit: CSSD aborting from thread clssnmRcfgMgrThread 2010-08-13 17:00:40.707: [ CSSD] [4052061088]################################### 33 it correctly detects and aborts local node to prevent split brain And does the right thing
  • 34. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | So what is the conclusion ? 34 Hmmm….I think it is the network.
  • 35. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | What does OS Watcher say? • netstat does not show any issues # grep "zzz|udpInOverflows|ipReasmFails" • OSW data itself is missing possibly due to scheduling issues • Just prior to the issue Top reports top - 13:23:52 up 25 days, 21:08, 1 user, load average: 3.43, 3.01, 3.02 Cpu(s): 16.8%us, 23.2%sy, 0.0%ni, 56.5%id, 3.1%wa, 0.1%hi, 0.3%si, Mem: 74027752k total, 73689744k used, 338008k free, 1516k Swap: 16771852k total, 9069988k used, 7701864k free, 25836528k PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1049 root 11 -5 0 0 0 R 55.0 0.0 0:57.42 [kswapd0] 35 No issues seen in netstat & traceroute Oh no! It is swapping Why is the OSW data missing?
  • 36. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Sample Problems and Resolution 36 Problem 2 – Slow I/O
  • 37. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Start the Database on all instances to simulate physical reads • Simulate I/O on shared storage – Loading data (sqlldr,expdp) OR – Swingbench • **behavior may depend on your test setup, HCA, HBA etc 37 Simulate I/O performance issue swingbench swingbench
  • 38. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | chactl query diagnosis-db 38 $ chactl query diagnosis -db sales -start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“ 2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_1) [detected] 2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_2) [detected] 2017-03-31 10:01:13.0 Database sales DB CPU Utilization (sales_2) [detected] 2017-03-31 10:01:33.0 Database salesDB Log File Switch (sales_1) [detected] Consolidates and displays information from all instances
  • 39. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | What does OS Watcher say? • iostat confirms I/O performance issue archive/oswiostat/xxxxxxxx_iostat_17.03.31.1000.dat • Increase in Reads/writes Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda 0.00 0.00 0.00 4.00 0.00 32.00 8.00 0.00 0.25 0.25 0.10 xvdb 0.00 0.00 6.50 2.00 146.00 21.00 19.65 0.00 0.24 0.24 0.20 • --- • --- Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda 0.00 0.00 0.00 144.00 434.00 53.00 12.00 0.00 0.45 1.13 0.10 xvdb 0.00 0.00 6.50 219.00 414.00 67.00 13.65 0.00 0.33 9.44 0.20 39 There is more overall I/O as seen by the increase in the number of writes, reads * values may change depending on test env. **Output has been formatted for presentation
  • 40. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Sample Problems and Resolution 40 Problem 3 – CPU bound
  • 41. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Ensure Grid Infrastructure is running for at least an hour • Run some normal load • Simulate excessive CPU using a CPU stressing program on 3 nodes – stress -- C program 41 Simulate CPU load stress stress stress
  • 42. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | chactl query diagnosis -cluster 42 $chactl query diagnosis –start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“ 2017-03-31 10:01:25.0 Host rwsxxxxx9 Host Memory Consumption [detected] 2017-03-31 10:01:29.0 Host rwsxxxxx0 Host Memory Consumption [detected] 2017-03-31 10:01:23.0 Host rwsxxxxx1 Host Memory Consumption [detected] Problem: Host Memory Consumption Description: CHA detected that more memory than expected is consumed on this server. The memory is not allocated by sessions of this database. Cause: The Cluster Health Advisor (CHA) detected an increase in memory consumption by other databases or by applications not connected to a database on this node. Action: Identify the top memory consumers by using the Cluster Health Monitor (CHM). Note: This is a 4 node cluster but the memory consumption issue is reported on 3 of the 4 nodes
  • 43. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | chatcl actions & resolutions in text or html format 43
  • 44. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | What does OS Watcher say? • OSWatcher confirms chactl diagnosis • mpstat (platform dependent) zzz ***Fri Mar 31 10:10:29 PST 2017 10:10:29 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:10:29 all 74.01 0.00 7.95 12.20 0.00 0.13 0.13 0.00 5.10 10:10:29 0 44.89 0.00 8.79 12.09 0.00 0.00 0.00 0.00 31.23 10:10:29 1 56.00 0.00 4.00 18.00 0.00 0.00 0.00 0.00 23.00 • Top also reports the CPU stress program Tasks: 454 total, 4 running, 450 sleeping, 0 stopped, 0 zombie Cpu(s): 94.9%us, 4.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20752 racusr 20 0 2498m 45m 39m R 85.9 0.3 752:32.40 stress 44 System is indeed CPU starved * values may change depending on test env. **Output has been formatted for presentation
  • 45. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Sample Problems and Resolution 45 Problem 4 – Why did my Database Instance move to different node?
  • 46. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Provides common view of cluster- wide activities in a co-ordinated fashion • Customer readable summary of all actions in a cluster • Each activity is a summary of the issue – Further details are in the trace files 46 Clusterware Activity Log On Friday, my instances were running on Nodes 1 & 2, but today it is only running on Node 1
  • 47. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 47 crsctl query calog -- Syntax $ crsctl query calog -h Usage: crsctl query calog [-aftertime <after_timestamp>] [-beforetime <before_timestamp>] [-duration <time_interval> | - follow] [-filter <filter_expression>] [-fullfmt | -xmlfmt] Lists the cluster activity log activities matching the specified criteria Where after_timestamp Time in format "YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]" OR "YYYY-MM-DD" OR "HH24:MI:SS[.FF][[+-]HH:MM]" before_timestamp Time in format "YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]" OR "YYYY-MM-DD" OR "HH24:MI:SS[.FF][[+-]HH:MM]" time_interval Time interval in format "DD HH:MM:SS" filter_expression Refer to product documentation. -follow Gives continuous output -fullfmt Detail output format -xmlfmt XML output format
  • 48. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 48 crsctl query calog output $ crsctl query calog -aftertime "2017-03-08 15:09:46.522-07:00" 2017-04-12 20:05:04.668000 : Attempting to start 'ora.anair1.vip' on ‘anair1' : 14920191617156230/1194/11 : ….. 2017-04-12 20:05:06.559000 : Attempting to start 'ora.LISTENER.lsnr' on ‘anair1' : 14920191617156230/1194/16 … 2017-04-12 20:05:32.038000 : Start of 'ora.FRA.dg' on ‘anair1' succeeded : 14920191617156230/1194/27 : 2017-04-12 20:05:32.040000 : Attempting to start 'ora.sales.db' on ‘anair1' : 14920191617156230/1194/28 : 2017-04-12 20:05:59.415000 : Start of 'ora.sales.db' on ‘anair1' succeeded : 14920191617156230/1194/30 : -- Format of output records is: DATE & TIME (YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]): Event text: ACTID Possibly network issues caused VIP relocation
  • 49. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 49 crsctl query calog filter output using actid $ crsctl query calog -filter "actid == 14920191617156230/2449732/1" 2017-08-03 16:30:24.678000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14920191617156230/2449732/1 : 2017-08-03 16:30:24.698000 : Start of 'ora.sscdb.db' on ‘anair1' succeeded : 14920191617156230/2449732/1 : $ crsctl query calog -filter "actid ~= “14920191617156230” 2017-08-03 16:25:20.658000 : Stop of 'ora.sscdb.test.svc' on anair1' succeeded : 14920191617156230/2449007/2 : -filter: Use ~= or == on actid to find related actions
  • 50. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | crsctl query calog filtered output using name $crsctl query calog -filter "entity_name == ora.sscdb.db" 2017-04-12 18:59:28.330000 : Attempting to start 'ora.sscdb.db' on ‘anair2' : 14919830844562257/9074/1 : 2017-04-12 18:59:28.334000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14919830844562257/9074/2 : 2017-04-12 18:59:28.340000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14919830844562257/9074/3 : 2017-04-12 18:59:28.694000 : Resource 'ora.sscdb.db' has been modified. : 14919830844562257/9074/4 : 2017-04-12 18:59:28.800000 : Resource 'ora.sscdb.db' has been modified. : 14919830844562257/9074/5 : … 2017-04-12 19:32:39.631000 : Attempting to stop 'ora.sscdb.db' on ‘anair2' : 14919830844562257/9853/2 : 2017-04-12 19:32:42.572000 : Stop of 'ora.sscdb.db' on ‘anair2' succeeded : 14919830844562257/9853/4 : 2017-04-12 19:45:01.175000 : Attempting to stop 'ora.sscdb.db' on ‘anair1' : 14919830844562257/10151/4 : 2017-04-12 19:45:04.176000 : Stop of 'ora.sscdb.db' on ‘anair1' succeeded : 14919830844562257/10151/7 : 50
  • 51. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Additional examples for querying calog • crsctl query calog -duration "00 00:02:00“ << last two minutes >> • crsctl query calog -duration "00 02:00:00“ << last two hours >> • crsctl query calog –follow << like tail… >> • crsctl query calog -aftertime "2017-03-08 17:49:12.397" –follow • crsctl query calog -filter "entity_name == foo“ << activity on foo >> • crsctl query calog -filter "actid == 145567015239630019/49543/1" • crsctl query calog –xmlfmt << in xml format >> 51
  • 52. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda FAQ 1 2 52 3 4
  • 53. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 53 Frequently asked Question # 1 Why does MGMT DB need so much space?
  • 54. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Cluster Type Redundancy MGMT DG (GB) Domain Services Cluster (2 Node DSC with 4 Member Clusters of 2 Nodes each) External 188 Normal 376 High 564 Flex 376 Standalone Cluster (4 Node Cluster) External 38 Normal 76 High 114 Flex 76 • Oracle GI 12c Release 2 feature AHF (Autonomous Health Framework) collects, Co-relates & stores diagnostics data from OS and DB in MGMT • In DSC, one PDB per member cluster is provisioned to store member cluster diagnostics data • The data is used by AHF components like cluster health advisor to both prevent and help diagnose issues 54 GIMR space requirements
  • 55. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 55 Frequently asked Question # 2 Should I continue to use Oswatcher?
  • 56. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Should I continue to use OSWatcher? • Hopefully by now, the value benefits of AHF is clear • AHF continues to be enhanced to improve – Diagnosing large number of deployments – Correlating data to speed diagnosis – Preventing issues in the first place – Utilizing latest technologies like Machine learning • AHF is mandatory (12.2+) but customers can choose to use OSW along with AHF 56 Learn more about Autonomous Health Framework https://www.slideshare.net/MarkusMichalewicz/cl uster-health-advisor-cha-deep-dive-by-mark- scardina
  • 57. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • opatch automatically patches MGMT database if required • clients of MGMT connect using encrypted credentials • MGMT listener automatically maintained by the clusterware agent 57 Frequently asked Question # 3 But it is still one more database for me to manage
  • 58. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 58
  • 59. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 59