PayPal workloads scale efficiently with Oracle Real Application Clusters (RAC). Oracle RAC provides high availability, scalability, and performance for PayPal's workloads. Oracle RAC utilizes features like cache fusion, dynamic resource management, and fine-grained processes to optimize performance on modern CPU architectures. Oracle RAC also provides near-zero downtime for reconfiguration and patching through optimizations like recovery buddies. PayPal leverages over 250 Oracle RAC clusters with over 90 petabytes of database storage to handle its transaction processing and analytics workloads at scale with high performance.

1. Scaling PayPal Workloads with Oracle RAC
New Generation Oracle Real Application Clusters (RAC)
Anil Nair (Anil.Nair@Oracle.com)
Sr. Principal Product Manager
Oracle Real Application Clusters (RAC) & ASM
@RACMasterPM
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/

2. About Pramod

3. We are a trusted part of people’s financial lives and a partner to merchants in
200+ markets around the world
Two decades ago, our founders invented payment technology to make buying and selling
faster, secure, and easier; and put economic power where it belongs: In the hands of people
About PayPal
Our 390+ Million consumers can accept payments in > 100
currencies and interact with 20M+ Merchants across 19K+
corridors
Almost 8000 PayPal team members provide
support to our customers in over 20 languages

4. Click to add image
“If you have not looked into the
New Generation Oracle RAC 19c
yet, you have not experienced what
Oracle RAC can do for you.”
https://www.oracle.com/technetwork/database/options/cluste
ring/overview/new-generation-oracle-rac-5975370.pdf
https://www.slideshare.net/AnilNair27/oracle-rac-new-generation
Private
Network
Failover
Private
Network
Copyright © 2021, Oracle and/or its affiliates |

5. Oracle Real Applications Clusters (RAC)
Constantly improving scalability and HA to protect your data automatically and faster
• Oracle Grid Infrastructure
• Oracle Clusterware
• Oracle ASM
• Oracle ACFS
• UCP and OCI Connection Load Balancing
• Run time load balancing
• Database Reliability Framework
• Scalable Sequences
• Continuous Application Availability
• Oracle RAC Sharding
• Cluster Health Advisor (CHA)
• Service Oriented Buffer Cache Access
• Transparent Application Continuity
• Commit Cache Messaging
• Automatic Hang Resolution
• Application Continuity (AC)
• Recovery Buddy
• LMS CR Server Threads
• Node Weighting
• Higher Priority for LMS
• PDB Service Isolation
11.2 12c
18c
• Colocation Tag for Client Routing
• Dynamic Services Fallback Option
• RAC resource runtime management
• Threaded Cache Fusion Processes
• Right-Growing Index
Performance improvements
• In-Memory Transaction Table
• Undo Block RDMA Reads
21c
• Optimized Broadcast of Commits
• PDB resource management
• Cache Fusion Hardening
• Zero downtime Grid Infrastructure
Patching
• RDMA for current read blocks
• Zero downtime for Planned
Outages
19c
* Documented features list is selective

6. New Generation Oracle RAC Scalability and High Availability
New Apps based on Converged DB’s using IoT, Kafka, Blockchain & Traditional packaged Apps
Private
Network
Failover
Private
Network
New Apps Traditional Apps
＆
Copyright © 2021, Oracle and/or its affiliates |

7. New Generation Oracle RAC scales IoT workloads
New Apps
https://www.oracle.com/a/tech/docs/wp-bp-for-iot-with-12c-042017-3679918.pdf
IoT Workload
Copyright © 2021, Oracle and/or its affiliates |

8. PayPal Database Requirements
Scalability Performance
Availability
Oracle RAC
• Scale within an
RAC Instance
• Scale by adding
RAC Instances
• Rolling Upgrades
with
Zero Downtime

9. 250+
Oracle RAC Clusters
Oracle RAC – By the numbers---
15M+
Execs/Sec
33% Y-o-Y
DB Storage Growth
90+ PB
Total DB Storage
Across Databases
18M+
Logical Reads/Sec
(768 Cores, 4 x Oracle X7-8
12 TB memory)
350k
User calls/Sec
300k
GC/GE messages/Sec
High Throughput OLTP RAC Database
2M+
IC packets sent/received
< 0.250ms
Avg GC message latency
< 4ms
Avg SQL call latency
> 100K
Executions/Sec on a table
Oracle RAC Deployment at PayPal

10. How does Oracle RAC
scale ALL kinds of Workloads?
Copyright © 2020, Oracle and/or its affiliates
10

11. Innovative and Unique : Oracle RAC Cache Fusion
Buffer
B
B
B
Private
Network
Private
Network B
Private
Network
B
Before Cache Fusion With Cache Fusion
• Multiple LMSs
• Higher Priority
• Auto Tune # of LMS
• BoC Synchronization
• Integration with DRF
• Dynamic GRD resizing
Oracle 8i Oracle 21c
B
2
3
1
2
B
1
Copyright © 2021, Oracle and/or its affiliates |

12. Out of the box better Application Performance
Right-Growing Index Optimizations
• Index split optimizations (12c onwards)
• Scalable Sequences (18c)
• In-memory Commit Cache (18c)
• Delayed ping (19c)
Interconnect Optimizations
• Light Weight IPC on Generic systems
• Improved handling of OS resources like
sockets, file descriptors
• RDMA on Exadata
• Cache Fusion speed improves to <10us,
even under load
Oracle RAC Cache Fusion Optimizations
3. Data Block Transfer
Instance 2
Instance 3
1. Request
Instance 1
2. Forward
LMS
FG
LMS
3. Direct Read
Instance 2
Instance 3
1. Request
Instance 1
FG
SGA LMS
2. Grant
Copyright © 2021, Oracle and/or its affiliates |

13. Utilize RDMA for
• Propagating BoC (Broadcast on Commit)
• Reduce messages
• Reduce CPU usage on LMS
• Current read blocks
• Very common access pattern in various workloads
• Commit Cache messaging
• https://www.slideshare.net/AnilNair27/oracle-rac-features-on-exadata
(Slide#43)
• Results in reduced load on LMS from remote node
• Undo block
• https://www.slideshare.net/AnilNair27/oracle-rac-features-on-exadata
(Slide#44)
Cache Fusion Optimizations on Exadata
Copyright © 2021, Oracle and/or its affiliates |

14. Efficient Operating System Resource
Management
Adaptive Oracle Database Resource
Management
Efficient and Adaptive Resource Management
CPU Network
Disks RAM Global Resource Directory
Copyright © 2021, Oracle and/or its affiliates |

15. Adapt to Modern CPU Architectures
DRAM
Process
Process
Core Core
Core Core
Process
Process
L2 Cache
Process Process Process Process
Thread Thread
Thread Thread Thread Thread
Thread
Thread
L1 Cache L1 Cache L1 Cache L1 Cache
Copyright © 2021, Oracle and/or its affiliates |
Modern CPUs have multiple
cores designed to run
multiple processing threads
at the same time

16. Leaner Threaded Processes benefit from current CPU architectures
• Threaded LMS process
dynamically started based on
need
• Process runs in higher
Priority (RR) while Threads
run in Normal Priority TS
• Monitored and changed via
DRF (Database Reliability
Framework)
• Continuously adapts based
on incoming workload
Oracle RAC Optimized Priority for Critical Processes
$ps -eLo 'pid tid cls cmd ' |grep '_lm' |grep sales |sort -k4
6970 6970 TS ora_lmon_rws17c
6972 6972 TS ora_lmd0_rws17c
6974 6977 RR ora_lms0_rws17c
6979 6979 RR ora_lmhb_rws1
6974 6981 TS ora_lms1_rws17c
6974 6983 TS ora_lms2_rws17c
LMS runs in RR LMS threads run in TS
Copyright © 2021, Oracle and/or its affiliates |

17. More fine-grained process threads
introduced over the years
• Dia0 – Hang Manager
• Gcr0- Global Conflict Resolution
thread
• Insv – DG broker
communications
• IPC0- Basic messaging
• Ping – Interconnect
measurement
• ACMS – CF Writes
Oracle RAC Fine Grained Process Model
$ps -ef |grep 'sales' |egrep "gcr|lmd" |sort -k8
racusr 15871 1 0 Jun25 ? 01:40:56 ora_gcr0_sales2
racusr 16925 1 0 Jun25 ? 00:26:18 ora_gcr0_sales2
racusr 16931 1 0 Jun25 ? 00:25:47 ora_gcr1_sales2
racusr 10828 1 0 Jun25 ? 00:26:13 ora_gcr2_sales2
racusr 11587 1 0 Jun25 ? 00:25:59 ora_gcr3_sales2
racusr 15593 1 0 Jun25 ? 00:41:17 ora_lmd0_sales2
racusr 15603 1 0 Jun25 ? 00:32:55 ora_lmd1_sales2
Copyright © 2021, Oracle and/or its affiliates |
LMD threads
GCR threads

18. Significantly contributing to Oracle RAC 19c Availability
18c
19c
11g
12c
4x
faster
7x
faster
Oracle RAC reconfiguration times reduced by
• 4x between Oracle RAC 11g and 12c
• 7x between Oracle RAC 11g and 19c
Selection of contributing features:
• Remastering servers (12c)
• Recovery Buddy (12c)
• Support for 100 LMS’s (12c)
• LMS’s run as OS threads (18c)
• More Reconfig Optimizations (19c)
Copyright © 2021, Oracle and/or its affiliates |

19. ORACLE RAC – Zero Downtime
Out of Box <10 sec
Optimized DRM
Zero Downtime with RAC Rolling Patches Reconfiguration Brownout times

20. ORACLE RAC - Reconfiguration Customizations
• Alter system set db_cache_size =
• Reconfiguration of < 5 sec
• 1TB cache shrink ~ 20 min
50% improvement

21. Optimize usage of OS Resources. – Memory, Huge Pages
Available system pagesizes:
4K, 2048K
Supported system pagesize(s):
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2048K 128000 66562 66562 NONE
Initial number of CPU is 72
Number of processor cores in the system is 36
Number of processor sockets in the system is 2
NUMA system with 2 nodes detected
Copyright © 2021, Oracle and/or its affiliates |

22. Optimize usage of OS Resources. -- Network
Cluster Communication is configured to use IPs from: GPnP
IP: 169.254.12.142 Subnet: 169.254.0.0
KSIPC Loopback IP addresses(OSD):
127.0.0.1
KSIPC Available Transports for cluster communication: UDP:TCP
KSIPC: Client: KCL Transport: UDP
KSIPC: Client: DLM Transport: UDP
KSIPC CAPABILITIES :MGA:IPCLW:GRPAM:TOPO:DLL:STATSFW:CRTRK
KSXP: ksxpsg_ipclwtrans: 2 UDP
cluster interconnect IPC version: [IPCLW over UDP(mode 3) ]
IPC Vendor 1 proto 2
Copyright © 2021, Oracle and/or its affiliates |

23. Database Reliability Framework – Adapt to changing workloads
Dynamic DRM
Consider Block congestion statistic for Cache
Fusion
Copyright © 2021, Oracle and/or its affiliates |

24. Near Zero Reconfiguration time with Recovery Buddies
Cache Fusion
provides complete
isolation between
PDBs
Continuous improvements in every release
Oracle RAC High Availability Optimizations
Smart Fencing ensures majority of work survives
Hang Manager
automatically detects
and resolves hangs and
deadlocks
Copyright © 2021, Oracle and/or its affiliates |

25. Requires no changes to the Application
• Incoming connection requests
are load balanced across all
instances
• Public network and instance
health is constantly monitored
by Grid Infrastructure
• On failure, GI will relocate the
VIP
• Existing connection requests
will receive error as GI will send
a GARP
• ONS consumers like SCAN
listeners, Connection Pools get
notified of failure
Database Clients – Load Balancing and Failure Handling
ODP.Net, OCI, Connection Pool
Single Client Access Network
New Connection request
Scan Listener routes to
DB listener on first node
Instance or Public
Network failure
GI relocates VIP to
another instance
Connection attempted
on VIP
Instant Error instead of
waiting on TCP timeout
Retry Connection
ONS Event Notification
Gratuitous ARP
Arp Table update
Copyright © 2020, Oracle and/or its affiliates |

26. Oracle RAC Deployment Pattern at PayPal
USERs
MICRO SERVICES • REDO 300 MBytesPs
• Interconnect 200 MBytesPs
• Write ~ 50K IOPs/ 5 GBPs
• Read ~ 50K IOPs/ 5 GBPs
• Execs/s ~100k per instance
• Gets/Block changes ~ 5m
Read Write Service
Read only Service
Sync Sync Sync

27. Oracle RAC Scalability Optimizations
Smart Cache Fusion
monitors and
dynamically chooses
optimal path
SSD
Service Oriented Buffer Cache
Cache Fusion monitors
and maintains Services
and Buffer Cache affinity
Pluggable
Database
Service
optimization
Optimized Singleton
Workload Scaling
provides consistent
performance after planned
service failover
Copyright © 2021, Oracle and/or its affiliates |
Continuous improvements in every release

28. 15M+
Execs/Sec
Extreme Availability and Performance
Oracle RAC - Performance Characteristics
< 0.250ms
Avg GC message latency
300k
GC/GE messages/Sec
< 4ms
Avg SQL call latency

29. • Older versions leverages DBMS_CACHEUTIL
to grab affinity
• Reduce unplanned performance impact with
traffic shifts
• Instance stop/start reconfiguration impact
• Mileage will vary
ORACLE RAC – Performance Customizations
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
B B
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
Global Resource Directory
B Buffer M Admin
B A

30. Summary
PayPal workloads scale efficiently with Oracle RAC

31. Preview of what’s coming up in
Oracle RAC 21c
Copyright © 2020, Oracle and/or its affiliates
31

32. Zero Downtime Grid Infrastructure Patching
Zero-Downtime Oracle Grid Infrastructure (GI) Patching
• Patch Oracle GI without interrupting database
operations
• Patches are applied in a rolling fashion (one node at
a time) while
the database instance(s) on that node remain up
and running
• Limited availability with version 19.7 and higher
• General availability with Oracle 21c + RU & RUR
support
https://docs.oracle.com/en/database/oracle/oracle-
database/19/cwlin/about-zero-downtime-gi-patching.html#GUID-
253CF841-DAAD-4B13-A1F2-6F83EF3FDE47
Copyright © 2021, Oracle and/or its affiliates |

33. Oracle RAC Cache Fusion Hardening
• This feature attempts to recover from discrepancies
received from the sending instance
• In the past, any such discrepancy between Cache
Fusion processes may cause instance to crash with
(ORA-481)
• Cache Fusion will try and attempt to fix the
discrepancy by retrying the buffer request from the
sending instance
Copyright © 2021, Oracle and/or its affiliates |

34. Zero Downtime for Planned outages
During planned shutdown
• Distribute Resource Admins before Instance
shutdown
• Distribution before shutdown does not require
any recovery on the surviving instances
• Effectively reduces the time spent on
reconfiguration during planned outages to zero
M M
M M
M M
M M
Copyright © 2021, Oracle and/or its affiliates |
A
A
A
A
A
A
A
A A
A A A A
A
A
A
A
A
A
A
A A A
A A
A
A
A
A
A
A
A Admin

35. Both traditional packaged Applications along with new apps based on IoT, Kafka, Blockchain
Benefits of Deploying on Oracle RAC
Highly Available
Scale within a Database Instance
Scale by adding Instances
Requires No Changes to Application
Client Application Failover
Connection Load Balancing
Scale In-Memory (Analytics)
Scale Multitenant (Microservices)
Complete Isolation in Converged Envs
Security
New Generation Oracle RAC provides HA and Seamlessly Scales
Copyright © 2021, Oracle and/or its affiliates |

36. Thank you
Oracle RAC Product Development Team
&
PayPal Engineering Team
Copyright © 2021, Oracle and/or its affiliates |

37. Questions?