Using Machine Learning to Debug complex Oracle RAC Issues

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Using Machine Learning to Debug complex
Oracle RAC Issues
Klaus Thielen – Architect
Anil Nair -- Sr. Principal Product Manager,
Oracle Real Application Clusters (RAC)
Jun 3rd , 2019
@RACMasterPM, @OracleRACpm
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2

Program Agenda
Introduction
How do we diagnose issues?
What’s new?
Walk through some common scenarios
FAQ
1
2
3
4
5
3

Program Agenda
Introduction1
2
3
4
5
4

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 5
Scalability without Application code change(s)
0
500 0
100 00
150 00
200 00
250 00
300 00
350 00
400 00
4 8 32 48 64 80
2035
4010
15520
22416
30016
37040
# of Cores across RAC Nodes
Users
2 Nodes
3 Nodes
4 Nodes
5 Nodes
SAP certified SD Benchmark results
Active Active Instances scales Writes, Reads and Hybrid workloads

Oracle Real Application Cluster Family of Solutions
• Integrated set of tools that work
cohesively to provide High Availability
and Scalability
• The functionality provided by Oracle RAC
Family of Solutions can be used by
licensed Oracle RAC, Oracle RAC One
Node and Single Instance customers
without any additional charge

Program Agenda
How do we Diagnose issues?
1
2
3
4
5
7

How do we Diagnose
Issues?

Lets walk through a
sample problem
resolution

Lets walk through a
sample problem
resolution
There is a quiz in the end so pay attention!!!

Lets walk through a
sample problem
resolution
There is a quiz in the end so pay attention!!!
https://www.slideshare.net/AnilNair27/using-machine-learning-to-debug-oracle-rac-issues
Case Studies covered last year

• Reactive
• Detection and resolution requires
diagnostic data from all the nodes
along with
• Different types of Issues such as
Performance slowdown, Hangs,
Deadlocks
12
How do we diagnose Issues?
Detect
ReactFix

How do we proactively
detect and attempt to
prevent issues?

• Utilize TFA notification features
• $tfactl set smtp
• $tfactl set notificationAddress=“..”
– Prepend the owner for specific homes
• $tfactl set notificationAddress=“salesora:
John.Doe@oracle.com”
• Events include
– Oracle Errors (ORA-)
– Cluster Health Monitor events
• Diagnostic Data is moved to DSC for
further analysis by TFA Service
Automatic Notification and Diagnostic Collection using TFA
DBA 1
DBA 2
TFA
Agent
TFA
Agent

• Utilize TFA notification features
• $tfactl set smtp
• $tfactl set notificationAddress=“..”
– Prepend the owner for specific homes
• $tfactl set notificationAddress=“salesora:
John.Doe@oracle.com”
• Events include
– Oracle Errors (ORA-)
– Cluster Health Monitor events
• Diagnostic Data is moved to DSC for
further analysis by TFA Service
Automatic Notification and Diagnostic Collection using TFA
DBA 1
DBA 2
TFA
Agent
TFA
Agent
Enable CHA Auto-collections
tfactl set chaautocollect=ON
Enable Email Notifications in TFA
tfactl set smtp
Enable Notifications for CHA events
tfactl set chanotification=ON
Specify Email Address for Notifications (optional)
tfactl set notificationAddress=chatfa:john.doe@oracle.com

Cluster Health Advisor TFA SMTP Notifications
From: <tfa@myserver1.example.com>
Date: Thursday, June 7, 2018 at 9:55 AM
To: <john.doe@example.com>
Subject: TFA CHA Event ASM Disk Utilization Cleared on myserver
At Thu Jun 07 09:55:35 PDT 2018 Oracle Cluster Health Analyzer detected ASM Cluster-wide Disk Utilization
on host myserver1
Cluster: mycluster
CHA Problem Description : CHA detected that the ASM disk IO request rate is higher than expected.
CHA Problem Cause : The Cluster Health Advisor (CHA) detected slower than expected disk performance because
the high disk I/O demand from the other servers increased the utilization of the shared disks.
CHA Problem Action : Review the CHA findings and corrective actions from the other servers and database instances
in the cluster for IO issues. Add disks to the database disk groups.
Example Notification Email

• OraChk can help compare and
contrast different configurations
– Different I/O scheduler?
– Centralized location makes it easier to
identify diffs
• Daemon Mode
– $orachk –d start/stop/status
• Configure Schedule
– $orachk AUTORUN_SCHEDULE
• What’s difference between OraChk
and ExaChk?
Configure OraChk to run in daemon mode
11204
11204DBA 1
DBA 2
Oracle RAC Assessment Report
System Health Score is 89 out of 100
Oracle RAC Assessment Report
System Health Score is 98 out of 100

• OraChk can help compare and
contrast different configurations
– Different I/O scheduler?
– Centralized location makes it easier to
identify diffs
• Daemon Mode
– $orachk –d start/stop/status
• Configure Schedule
– $orachk AUTORUN_SCHEDULE
• What’s difference between OraChk
and ExaChk?
Configure OraChk to run in daemon mode
11204
11204DBA 1
DBA 2

Lets walk through a
sample problem
resolution

Node Eviction due to memory pressure
20
Memory pressure as
reported by the OS
In the past, this would
result in Node eviction

Misconfiguration!!!
Application
Server
Application
Server
Application
Server

Memory Guard will proactively step in to prevent Node
Eviction
• Monitors memory allocated and detects
conditions that may result in swapping
• Issues alert should any server be at risk
• Stops services transactionally on the
server
• This prevents any new applications
connections
• Re-starts the services once the excess
memory allocation issue is cleared
Sales_svc Sales_svc

Oracle Memory Guard
M em ory stress detected by M em ory G uard
DB Alert Log Audit Reports

Oracle Memory Guard
M em ory G uard stops services for incom ing connections

Oracle Memory Guard
Services for new connections are restarted on the node

So what is the root cause?
26
Hmmm….Who used all my memory?

CHM provides aggregated data by Process Type
Use Historical Data to fix root cause

Program Agenda
What’s new?
1
2
3
4
5
28

Oracle Autonomous Health Framework (AHF)
• Integrated next generation tools
running as components - 24/7
• Discovers Potential Issues and takes
Corrective Actions
• Speeds up Issue Diagnosis and
Resolution
• Maintains Database Performance
and Availability
29

Applied Machine Learning Pipeline Diagnostics
• Generic ML-extracted Data Clusters
are insufficient for diagnostics
• Operational data correlation does
not determine root cause
• Trusted root cause determination
critical to swift corrective actions
• Algorithms selected and models
built require domain expertise
• Models refined via field feedback
Subject Matter
ExpertLog
ASH
Metrics
ML
Knowledge
Extraction
Model
Generation
Human
Supervision
Application
Optimized
Models
Feedback
ScrubData

Data Pipeline Overview
Autonomous Health – Database Performance

Program Agenda
Walk through some common scenarios
1
2
3
4
5
32

Scenario 1 – Remember our Service Shutdown
33

• CHAG (Cluster Health Advisor
Graphical) Doc ID 2340062.1
• CHAG is the GUI to utilize the
benefits of AHF
• Can be run on the Cluster node
– Not Recommended*
– Set ORACLE_HOME= GI HOME
– Connects to the GIMR using wallets
– but Install locally on a separate Linux
host
$export ORACLE_HOME=/u01/app/12.2.0/grid
$ ./chag
CHAG logging to log/chagout_20873.log
Initializing DB reader
Connect via
'jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDR
ESS=…..
CHAG MDB feed open. Timings: Load JDBC driver:
147.50 ms, Connect to MDB: 1398.85 ms
1st Query with 60 minutes of data (15:00:56..16:00:56):
16546 ms
1st CLOB : (59290 lines, 1.484 MB) parsing time: 644
ms, 10.49 mics/line
34
You could have reached same conclusion using CHA

anair1

Use CHAG on a Remote system
• Remote Mode requires the GIMR Data to be exported
– Execute the following on the cluster node
– $chactl export repository -format mdb -start ‘’ -end '‘
–
• Copy the MDB to the local node and execute
– chag –f <mdb_file>
• MDB file includes all the data for post mortem analysis from all the nodes
36
$chactl export repository -format mdb -start '2017-12-15 00:00:00' -end '2017-12-20 00:00:00'
successfully dumped the CHA statistics to location
"/u01/app/gridbase/crsdata/anair/trace/chad/cha_dump_20171215_000000_20171215_010000.mdb“

Sample Problems and Resolution
37
Problem 2 – Why was my Session killed?

Hang Manager interventions reported via ORA-32701
Dump file …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
Oracle Database 12c Enterprise Edition Release 12.2.0.0.0 - 64bit Beta
With the Partitioning, Real Application Clusters, OLAP, Advanced Analytics
and Real Application Testing options
Build label: RDBMS_MAIN_LINUX.X64_151013
ORACLE_HOME: …/3775268204/oracle
System name: Linux
Node name: slc05kyr
Release: 2.6.39-400.211.1.el6uek.x86_64
Version: #1 SMP Fri Nov 15 13:39:16 PST 2013
Machine: x86_64
VM name: Xen Version: 3.4 (PVM)
Instance name: hm62
Redo thread mounted by this instance: 2
Oracle process number: 19
Unix process pid: 12656, image: oracle@slc05kyr (DIA0)
*** 2015-10-13T16:47:59.541509+17:00
*** SESSION ID:(96.41299) 2015-10-13T16:47:59.541519+17:00
*** CLIENT ID:() 2015-10-13T16:47:59.541529+17:00
*** SERVICE NAME:(SYS$BACKGROUND) 2015-10-13T16:47:59.541538+17:00
*** MODULE NAME:() 2015-10-13T16:47:59.541547+17:00
*** ACTION NAME:() 2015-10-13T16:47:59.541556+17:00
*** CLIENT DRIVER:() 2015-10-13T16:47:59.541565+17:00
2015-10-13T16:47:59.435039+17:00
Errors in file /oracle/log/diag/rdbms/hm6/hm6/trace/hm6_dia0_12433.trc (incident=7353):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm6/incident/incdir_7353/hm6_dia0_12433_i7353.trc
2015-10-13T16:47:59.506775+17:00
DIA0 requesting termination of session sid:40 with serial # 43179 (ospid:13031) on instance 2
due to a GLOBAL, HIGH confidence hang with ID=1.
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=1.
2015-10-13T16:47:59.538673+17:00
Errors in file …/diag/rdbms/hm6/hm62/trace/hm62_dia0_12656.trc (incident=5753):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
2015-10-13T16:48:04.222661+17:00
DIA0 terminating blocker (ospid: 13031 sid: 40 ser#: 43179) of hang with ID = 1
requested by master DIA0 process on instance 1
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
by terminating session sid:40 with serial # 43179 (ospid:13031)
ORA-32701: Possible hangs up to hang ID=1
detected
Incident details in:
…/../hm62_dia0_12656_i5753.trc
DIA0 terminating blocker
(ospid: 13031 sid: 40 ser#: 43179)
requested by master DIA0 process on
instance 1
Hang Resolution Reason: Automatic hang
resolution was performed to free a
significant umber of affected sessions.
Alert log

39
Problem 3 – How long did the
reconfiguration take?

Reconfiguration Diagnosability
**************** BEGIN DLM RCFG HA STATS ****************
Total dlm rcfg time (inc 6): 3.586 secs (394926177, 394929763)
Begin step .........: 0.005 secs (394926177, 394926182)
Freeze step ........: 0.019 secs (394926182, 394926201)
Sync 1 step ........: 0.002 secs (394926264, 394926266)
Sync 2 step ........: 0.024 secs (394926266, 394926290)
Enqueue cleanup step: 0.002 secs (394926290, 394926292)
Sync pcm1 step .....: 0.004 secs (394926293, 394926297)
……
….
Enqueue dubious step: 0.004 secs (394926432, 394926436)
Sync 5 step ........: 0.000 secs (394926436, 394926436)
Enqueue grant step .: 0.001 secs (394926436, 394926437)
Sync 6 step ........: 0.012 secs (394926437, 394926449)
Fixwrt replay step .: 0.885 secs (394928837, 394929722)
Sync 8 step ........: 0.040 secs (394929722, 394929762)
End step ...........: 0.001 secs (394929762, 394929763)
Number of replayed enqueues sent / received .......: 2246 / 893
Number of replayed fusion locks sent / received ...: 124027 / 0
Number of enqueues mastered before / after rcfg ...: 2058 / 1384
**************** END DLM RCFG HA STATS *****************
Detailed timing
breakdown available
in LMON trace file

41
Problem 5 – Is Dynamic Resource
Management (DRM) helping my workload ?

DRM Diagnosability
Dynamic Remastering Statistics DB/Inst: SALES/sales1 Snaps: 393-452
-> Affinity objects - Affinity objects mastered at the begin/end snapshot
-> Read-mostly objects - Read-mostly objects mastered at the begin/end snapshot
per Begin End
Name Total Remaster Op Snap Snap
-------------------------------- ------------ ------------- -------- --------
remaster ops 24 1.00
remastered objects 24 1.00
remaster time (s) 7.4 0.31
freeze time (s) 1.5 0.06
cleanup time (s) 2.4 0.10
replay time (s) 0.3 0.01
fixwrite time (s) 2.4 0.10
sync time (s) 0.1 0.01
affinity objects N/A 3 27
read-mostly objects N/A 0 0
read-mostly objects (persistent) N/A 0 0
Detailed timing
breakdown available
in AWR Report

Program Agenda
FAQ
1
2
3
4
5
43

Frequently asked Question # 1
Why does GIMR
(mgmtdb) need
so much space?

Cluster Type Redundancy MGMT DG
(GB)
Domain Services Cluster
(2 Node DSC with 4 Member
Clusters of 2 Nodes each)
External 345
Normal 690
High/Flex 1035
Standalone Cluster
(4 Node Cluster)
External 30
Normal 60
High/Flex 90
• Oracle GI 18c feature AHF (Autonomous
Health Framework) collects, correlate &
stores diagnostics data from OS and DB in
MGMT
• In DSC, one PDB per member cluster is
provisioned to store member cluster
diagnostics data
• DSC also stores central TFA Repository
• Diagnose issues in real time, accurate
and quick post-mortem analysis
and provide historical data for
workload characterization over time ,
performance and capacity review
45
GIMR space requirements

Why is the Grid
Infrastructure
Repository
mandatory?

Is the Grid Infrastructure Repository (GIMR) mandatory?
• Starting with Version 19 the GIMR is going to
be optional!
– You can choose to install it during the installation of
the Grid Infrastructure
– If you decide to go without the GIMR we are going to
write into the filesystem
– As soon as we find an running Repository (GIMR) we
start writing into the repository
• Databases also are going to be monitored
automatically
– Pre version 19: The cluster nodes were monitored
automatically
47

Will we get the same functionality without the GIMR?
• No! There is no support for
– Cluster Heath Advisor GUI (CHAG)
• Online and offline mode
– CHACTL
• Calibrate
• Export repository -format
• query diagnosis
– Global view over the cluster
– Good and Bad Neighbor analysis
• Trace File Analyzer will provide a limited graphical view of Root causes
– When chaautocollect and chanotification set = ON
48

GIMR is optional but recommended!!!
If you install it, you
will get the
advantage of the full
functionality of CHA

Should I continue
to use
Oswatcher?

Should I continue to use OSWatcher?
• Hopefully by now, the value benefits of AHF is clear
• AHF collects much of the data which OSWatcher also collects
• AHF continues to be enhanced to improve
– Diagnosing large number of deployments
– Correlating data to speed diagnosis
– Preventing issues in the first place
– Utilizing latest technologies like Machine learning
• However, Customers can choose to use both OSW and AHF
51

• opatch automatically patches GIMR
if required
• clients of GIMR connect using
encrypted credentials
• MGMT listener is automatically
maintained.
52
But it is still one
more database
for me to manage

For Further Information
• Oracle 19c Autonomous Health Framework User’s Guide
• Oracle 19c Clusterware Adminstration and Deployment Guide
• Oracle Autonomus Health Framework on OTN
• Oracle QoS Management on OTN
• Oracle 19c ORAchk
• Oracle 19c Trace File Analyzer
• MDBUtil: GI Management Repository configuration tool (MOS Doc ID
2065175.1)
• Cluster Health Advisor Graphical User Interface - CHAG (Doc ID 2340062.1)

Using Machine Learning to Debug complex Oracle RAC Issues

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Using Machine Learning to Debug complex Oracle RAC Issues

Similar to Using Machine Learning to Debug complex Oracle RAC Issues (20)

Recently uploaded

Recently uploaded (20)

Using Machine Learning to Debug complex Oracle RAC Issues