The deck cover upgrade best practices along with new features in Oracle 19c RAC including Cache Fusion

1. 1
Oracle RAC 19c:
Upgrade Tips and Tricks
Copyright © 2019 Oracle and/or its affiliates.
Anil Nair
Sr Principal Product Manager,
Oracle Real Application Clusters (RAC)
@RACMasterPM
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/

2. The preceding is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September
2019 and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor
Copyright © 2019 Oracle and/or its affiliates.

3. Agenda
Why Upgrade?
Best Practices to Upgrade to Oracle 19c
Oracle 19c Grid Infrastructure (GI) New Features
What’s new with Cache Fusion?
1
2
3
3
4

4. Agenda
Why Upgrade?1
2
4
3
4

5. Why should I upgrade?

6. Lifetime Support Policy
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
Oracle 18
(12.2.0.2)
EXTENDED
EXTENDED
EXTENDED
Waived EXTENDEDOracle 11.2
Oracle 12.1
Oracle 12.2.0.1
Oracle 19
(12.2.0.3)
Paid Extended SupportPremier Support Waived Extended Support
MARKET
DRIVEN
Market Driven Support
▼

7. Oracle 12.2 Release Family?
• Includes:
• Oracle 12.2.0.1, Oracle 18c (12.2.0.2), Oracle 19c (12.2.0.3)
• MOS Note:742060.1 - The Single Source of Truth
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
Oracle 18
(12.2.0.2)
EXTENDED
12.2.0.1
Oracle 19
(12.2.0.3)

8. Lifetime Support Policy | Future Projection
Copyright © 2019 Oracle and/or its affiliates.
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
Oracle 18c
EXTENDED
EXTENDED
EXTENDED
Waived EXTENDEDOracle 11.2
Oracle 12.1
Oracle 12.2.0.1
Oracle 19c
Paid Extended SupportPremier Support Waived Extended Support
MARKET
DRIVEN
Market Driven Support
▼
Oracle 20c
Oracle 21c
Oracle 22c
Release Family
Release Family
Interim
Interim
Long Term
Neither future release dates nor
future release names are set yet!
This is a projection only!

9. Agenda
Best Practices to Upgrade to Oracle 19c
1
2
9
3
4

10. • Execution of ./gridSetup.sh on old
OS releases may fail
• Failure is reported as a perl error
message
• perl has hard dependency on
glibc
• Similar message reported by DB
installer
• Additional details in URL below
https://www.linkedin.com/pulse/high-level-
steps-upgrade-oracle-19c-rac-anil-nair/
Oracle 19c Upgrade requires Linux 7

11. Upgrade to Linux 7 with least downtime
Drain
delNode
Linux7
addNode
For each Node perform the following
until last node
Linux6 Linux6 Linux6 Linux6Linux7
Services
$srvctl relocate service
–drain_timeout
$./delNode ….
Upgrade or
Reinstall
$./addNode ….
*Inline Upgrade depends on initial configuration

12. Alternatively Upgrade OS and
Drain
Stop CRS
Linux7
Update
Files
Start CRS
For each Node perform the following
until last node
Linux6 Linux6 Linux6 Linux6Linux7
Services
$srvctl relocate service
–drain_timeout
$ crsctl stop crs
Inline Upgradeupdateosfiles
#rootcrs -updateosfiles
$ crsctl start crs

13. • Download latest orachk and benefit
from the latest checks
• No need to individually
download autoupgrade.jar or
cluvfy
• Single report with results from
autoupgrade.jar, orachk and cluvfy
checks
• *orachk also includes other
components like Application
Continuity and Security related
checks
ORAchk=ORAchk+cluvfy+Autoupgrade.jar*
Cluvfy PreUpgrade
ORAchk
ORAchk autoupgrade includes autoupgrade.jar
checks and cluvfy pre-upgrade checks

14. One command for all Autoupgrade checks
Report includes results from all components with appropriate options
• orachk -preupgrade -targetversion 19.3.0.0.0 –showpass
• cluvfy –stage pre/post
• orachk includes preupgrade.jar and cluvfy pre-upgrade checks

15. • Grid Infrastructure Management
Repository (GIMR) aka mgmtDB is
NO longer mandatory with starting
with Oracle 19c
• Limited AHF functionality by
utilizing filesystem without GIMR
• No support for CHA GUI chactl
• Trace File Analyzer (TFA) will
provide limited graphical view
State of the GIMR

16. Choose to install GIMR: Thanks to Your
Feedback
• Upgrades depend on Initial GIMR configuration
• Choose to Install Grid Infrastructure Management Repository (GIMR)
• Eventual goal is to move GIMR into its own new separate home
17
Only for New Installations

17. -dryRunForUpgrade: Thanks to Your
Feedback
18
gridSetup now can be used with the –dryRunForUpgrade option for dry run
testing of Oracle Grid Infrastructure upgrades
$./gridSetup.sh –dryRunForUpgrade

18. Read Only Oracle HOME (ROOH)
• ROOH enabled Oracle Database homes store configuration
files outside of the Oracle Home
• Faster cloning of Oracle software home with ROOH as
environment specific configuration files are stored outside of
Oracle home
• Improves security as running processes cannot create new
files under Oracle Home
• Oracle RAC DB Home is ROOH
• $roohctl can be used in versions 18c and 19c for manual
conversion
• Only pertinent to Oracle Database Home (Not GI home)
• Plan to remove configuration files used by application before
converting to ROOH such as
• tnsnames.ora

19. • Apply patch to a new grid
home while stack continues to
run from current home
• Reduces downtime as stack is
up and running during the
copy process
• Reduces errors caused by
common issues such as “Out
of space”
• Easy fallback in case of issues
Patch faster with -SwitchHome
/u01/app/19.0/grid
/u01/app/19.3/grid

20. Summary of Best Practices for Upgrade
Always download the latest version of orachk/exachk from
• https://support.oracle.com/epmos/faces/DocContentDisplay?id=1268927.2
• https://support.oracle.com/epmos/faces/DocContentDisplay?id=1070954.1
Consider storage requirements of GIMR
Apply latest OS patches
• orachk includes the DBSAT (Oracle Database security assessment tool)
Add user defined checks to benefit from a single report
Find environment specific files in ORACLE_HOME such as
password, tnsnames.ora, pfile) and other files that may affect using
ROOH

21. Agenda
Oracle 19c Grid Infrastructure (GI) New Features
1
2
23
3
4

22. • Optimize management of
resources such as database
instance, listener on nodes
• Include the ability to startup, stop,
prioritize, relocate resources
• Define certain PDBs as critical and
therefore start consider reordering
startup
New GI Resource Modeling for PDBs
Linux6 Linux6 Linux6 Linux6Linux7
Services

23. • Utilizes Service(s) to drive
workload placement
• Services implicitly opens PDB
Instance(s)
• Order of PDB open based on
service definition
• Defined using Preferred,
Available attributes
• Default modeling after
upgrades
Resource Modeling Today
Services trigger PDB open
Services

24. • Define PDB Cardinality, Rank
• Higher Ranked PDBs started
before Lower Ranked PDB
• Cardinality defines the number
of nodes where PDB is started
• Considers
• host runtime CPU load before
PDB open
• availability of nodes before PDB
open
• all nodes may not be equal in
terms of # of CPUs or speed
New Resource Modeling Scheme
PDB open triggers Service startup

25. • Environment changes,
incorrect permissions of
binaries can prevent stack
startup
• Oracle 19c Clusterware stack
attempts to auto-diagnose
unsuccessful startup issues
• Provides detailed logging in
case of failures
CRS-41053: checking Oracle Grid Infrastructure for file permission
issues
PRVG-2031 : Owner of file ”…gipcd.bin" did not match the
[Expected="grid(54320)" Found="oracle(54325)"]
Oracle Clusterware start Diagnosticsinit
cssdAgent
cssdmonitor
ohasd
oraAgent
cssd
crsd
ctssd
HAIP
ACFS
mdnsd
gipcd
evmd
ASM
*NOT all daemons are shown in illustration above
oraRootAgent

26. Clusterware runtime Diagnostics
• Oracle 19c Clusterware processes maintains histograms, statistics
such as trace file rotation frequency, time taken for rotation
• Preserves critical information on very busy systems
• Severity tagging provides human readable criticality of messages
• 2019-08-20 08:36:13.142 : CSSD:1871161088: [ ERROR]
clssgmclienteventhndlr: (SENDCOMPLETE) No proc found for ClientID
• 2019-08-20 08:36:13.188 : CSSD:1871161088: [ INFO] clssgmDeadProc:
Removing clientID 2:43454:0 (0x7fda802df820), with GIPC
• New diagnostics monitor thread ensures in-memory logs (UTS) are
periodically written to ensure diagnostics are available in case of
process crash

27. Private Network Interface Check
• Oracle 19c recommends using bonding mode 0 Balance-RR, 1-
Active Backup when HAIP is not used for network redundancy
• Recommendation is based on greater tolerance to network
jitters with different combination of interface, switch, OS
• Check also ensures every node of the cluster has same
bonding mode
• The check is a warning
• It is possible but not recommended to use other modes

28. • Clusterware processes
communicate with each
other using gIPC
• gIPC today utilizes TLS
(Transport Layer
Security)
• Easy configuration of
any future secure
communication
protocol
Oracle Clusterware Ciphers
$crsctl get cluster security tls
ON
$crsctl get cluster tlsciphersuite
enabled

29. Summary of Clusterware features
Plan to utilize the New Resource modelling capabilities
Clusterware start failure(s) will trigger environment checks with
detailed logging to help pinpoint probable mis-configurations
• Permission
• Network/Storage
Human readable messages in trace files during runtime failures
for faster issue resolution
Additional checks to ensure cluster interconnect is configured
correctly

30. Agenda
What’s new with Cache Fusion?
1
2
32
3
4

31. Cache Fusion: A long Journey
BufferB
B
B
Private
Network
Private
Network B
Private
Network
B
Before Cache Fusion
Cache Fusion
• Multiple LMSs
• Higher Priority
• Auto Tune # of LMS
• BOC Synchronization
• Integration with DRF
• Dynamic GRD resizing
Oracle 8i Oracle 19c
B
2
3
1
2
B
1

32. • During Startup
• Resources are distributed across nodes
• GRD maintains information on these
resources
• Resource Master may or may not be on
same node as the resource
• Steady State
• DRM (Dynamic Re-Mastering) helps
moves Resource Master to same node
as Resource
34
Optimize Resource Master placement
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
Goal is to reduce 3- way communication providing performance equal to Single Instance
Global Resource Directory
B Buffer M MasterB M

33. During planned shutdown
• Distribute Resource Masters before
Instance shutdown
• Distribution before shutdown does not
require any recovery on the surviving
instances
• Effectively reduces the time spent on
reconfiguration during planned outages
to zero
• Rolling Windows to reduce
impact of reconfiguration
• During runtime
• Smart DRM
35
Zero Downtime for Planned outages
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
Global Resource Directory
B Buffer M MasterB M

34. Oracle RAC Performance Automation
• Note 1619155.1 Best Practices and Recommendations for RAC
databases with SGA size over 100GB
• Automatic configuration in 19c
• Dynamic CR slaves to deal with changing workload
• LMS CR slaves (_max_cr_rollbacks deprecated) (1630755.1)
• Reduce “LMS process busy” event in AWR report
• LMS CR Slaves
• Dynamic DLM ticket adjustment to prevent hangs
• remove _lm_tickets

35. • Subnet Manager for Fast
Node Death detection
• Network (Subnet Manager)
• Disk (Diskmon)
• Utilize low latency RDMA
• Read/Write to remote memory
without CPU
• More details available at
• https://www.slideshare.net/AnilNair2
7/oracle-rac-features-on-exadata
Oracle RAC Exadata optimizations
0.8
30
0
10
20
30
40
Exadata Generic Systems
Fast Node Death Detection
Exafusion
Smart Fusion
Block Transfer

36. Cache Fusion Optimizations on Exadata
Utilize RDMA for
• propagating BoC (Broadcast on commit)
• Replace messages based protocol with RDMA
• Reduce CPU usage on LMS
• Current read blocks
• Very common access pattern in various workloads
• Commit Cache messaging
• https://www.slideshare.net/AnilNair27/oracle-rac-features-on-exadata (Slide#43)
• Reduces load on LMS from remote node
• Direct read than sending an 8k block
• Undo blocks
• https://www.slideshare.net/AnilNair27/oracle-rac-features-on-exadata (Slide#44)
38

37. 39
All this is good
but who uses
Oracle RAC

38. #1 Proven Choice for Scalability & Availability

39. 150+
Oracle RAC Clusters
5M+
Execs/Sec
25% Y-o-Y
DB Storage Growth
50+ PB
Total DB Storage
18M+
Logical Reads/Sec
4 x Oracle X7-8
(768 Cores, 24 TB memory)
200k
Execs/Sec
300k
GC/GE messages/Sec
Extremely Busy OLTP RAC Cluster
2M+
IC packets sent/received
< 0.250 µs
Avg GC message latency
< 4ms
Avg SQL call latency
75K
Execs/Sec on a table
Oracle RAC at PayPal
Across Databases

40. Summary
• Oracle RAC is the proven choice for Scalability and Availability without any application changes

41. 43
Oracle RAC Choice of Platform, On-Prem & Cloud
Autonomous
Database
Exadata Cloud
Service
Oracle RAC on Generic
systems in Oracle Cloud
Oracle
Exadata
Generic
System
AutomaticManual
Oracle RAC on Generic
systems in Oracle Cloud

42. I don’t have test
environment

43. Solution: Oracle Cloud

44. oracle.com/cloud/free
Oracle Cloud
Free Tier
Always Free
Oracle Cloud Infrastructure
Services you can use for unlimited time
$300 free credits for 30 days
Free credits you can use for more services
+
$500 OpenWorld Special

45. Thank You
Anil Nair
Sr Principal Product Manager,
Oracle Real Application Clusters (RAC)
@RACMasterPM
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/