Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf

Oracle RAC Operational Best Practices as presented at Sangam 2017

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf

  1. 1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Oracle RAC 12c Release 2 and Clusterware Internals Anil Nair Sr. Principal Product Manager, Oracle Real Application Clusters (RAC) Dec 25th, 2018 @RACMasterPM, http://www.linkedin.com/in/anil-nair-01960b6 http://www.slideshare.net/AnilNair27/ 20 Years of Continuous Innovation
  2. 2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2
  3. 3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Please remember to provide feedback – Session ID Session ID: 1369 – Oracle RAC 12 Release 2 and Clusterware Internals • Reach out to us with follow up questions, suggestions etc. via Twitter @RACMasterPM Linkedin http://www.linkedin.com/in/anil-nair- 01960b6 Feedback is important for successful collaboration
  4. 4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda Cluster Domain Architecture RAC New Features Sample Problems and Resolution FAQ 1 2 4 3 4
  5. 5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda Cluster Domain Architecture1 2 5 3 4
  6. 6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Oracle RAC 12c Rel. 2 Cluster Domain • Simplified Management – Fleet Management for installation, update, patching and maintenance • Reduced Local Overhead – Member Clusters benefit from the consolidation of common services on the Domain Services Cluster • Improved IO Performance – Utilizing consolidated shared storage 6 Centralized Management for Cluster Estates “too big to manage” otherwise
  7. 7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 7 Cluster Domain Application Member Cluster Uses ASM Database Member Cluster Uses local ASM Database Member Cluster Uses ASM Service Database Member Cluster Uses IO Service Domain Services Cluster (DSC) Shared ASM IO Service ACFS Service ASM Service TFA Service Management Service RHP Service
  8. 8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 8 The DSC – The Heart of the Cluster Domain Domain Services Cluster (DSC) IO Service ASM Service TFA Service Management Service Shared ASM ACFS Service RHP Service • The DSC hosts services that are consumed by Member Clusters, including: – Management Service for centralized and simplified management – Trace File Analyzer (TFA) for centralized diagnostics – Rapid Home Provisioning (RHP) for software fleet management – Storage Services (ACFS, ASM direct or indirect over IO Service)
  9. 9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Domain Services Cluster (DSC) IO Service ASM Service Shared ASM ACFS Service RHP Service 9 The DSC Management Service TFA Service Management Service Applied Machine Learning for Database Diagnostics • Efficient diagnosis using Machine Learning • Automatically performs corrective actions to prevent possible issues • Provides simple alerts & recommendations for issues that require manual intervention Subject Matter ExpertLog ASH Metrics ML Knowledge Extraction Model Generation Human Supervision Application Optimized Models Feedback
  10. 10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 10 Rapid Home Provisioning Service Domain Services Cluster (DSC) IO Service ASM Service Shared ASM ACFS Service RHP Service TFA Service Management Service • Provision new pools onto base machines • DB and GI: provision, scale, patch, upgrade • Custom workflow framework • Notification model • Audit capabilities RHP Service Member Cluster Member Cluster Member Cluster Fleet Management installation, update, patching and maintenance
  11. 11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 11 Domain Services Cluster (DSC) Availability • Services provided by the DSC are unaffected by nodes joining/ leaving the DSC cluster • DSC can be patched and upgraded independently without affecting the services provided by DSC • Use Member Clusters for user databases (not the DSC)
  12. 12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 12 Cluster Domain Application Member Cluster Uses ASM Database Member Cluster Uses local ASM Database Member Cluster Uses ASM Service Database Member Cluster Uses IO Service Domain Services Cluster (DSC) Shared ASM IO Service ACFS Service ASM Service TFA Service Management Service RHP Service
  13. 13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • A Member Cluster is a Standalone Cluster utilizing shared services on the Domain Services Cluster • It automatically benefits from the management, TFA & RHP service. – ASM services are optional – and can be utilized as needed 13 Member Clusters = Standalone Cluster + Benefits Database Member Cluster Uses local ASM Database Member Cluster Uses ASM Service of DSC Database Member Cluster Uses ASM IO Service
  14. 14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 14 Same Tools and Commands for all Types of Deployments [GRID]> crsctl get cluster name CRS-6724: Current cluster name is 'SolarCluster' [GRID]> crsctl get cluster class CRS-41008: Cluster class is 'Standalone Cluster' [GRID]> crsctl get cluster type CRS-6539: The cluster type is 'flex'. [GRID]> crsctl get cluster name CRS-6724: Current cluster name is 'SalesCluster' [GRID]> crsctl get cluster class CRS-41008: Cluster class is ‘Database Member Cluster' [GRID]> crsctl get cluster type CRS-6539: The cluster type is 'flex'.
  15. 15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Easy conversion from Standalone to Member Clusters • Member Cluster GI version can be higher or equal to the GI version on the Domain Services Cluster 15 Convert a Standalone Cluster to Member Cluster Convert Database Member Cluster Uses local ASM
  16. 16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda RAC New Features 1 2 16 3 4
  17. 17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 17 All Deployments Benefit from Oracle RAC Features PDB Isolation Hang Manager Choose the “right path” Zero Impact Patching Scalable Sequences
  18. 18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 1010101000010011100101001 1011100101001110010100101 0010010100001111010100101 1010101000010011100101001010010100100111001010010100100 1011100101001110010100101111110100111001010010111101101 0010010100001111010100101001010100100110101001010101000 18 • Using Oracle Multitenant, PDBs can be opened as singletons (in one database instance only), in a subset of instances or all in instances at once. • If certain PDBs are only opened on some instances, Pluggable Database Isolation – improves performance by • Reducing DLM operations for PDBs not open in all instances. • Optimizing block operations based on in-memory block separation. – improves availability by • Ensuring that instance failures of instances only hosting singleton PDBs will not impact other instances of the same RAC-based CDB. Optimized Singleton Workload Scaling Pluggable Database Isolation NEW IN 12.2PRIOR TO 12.2 Inter Cluster Messages
  19. 19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Database Member Cluster Uses ASM Service 19 Pluggable Database and Service Isolation Provides higher Isolation in Member Cluster Consolidated Environments
  20. 20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Hang Manager • Hang Manager “manages” database process(es) that are not progressing – Manage = detection and resolution – Considers cross-layer hangs • I.E: Hangs caused by a blocked ASM resource. • In addition, deadlocks are now resolved by Hang Manager • User defined control via PL/SQL • Early Warning exposed via (V$ view) 20 Resolving cross-layer hangs in the Cluster Domain Database Member Cluster Uses ASM IO Service IO Service ASM Service
  21. 21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 21 Resolving cross-layer hangs in the Cluster Domain Provides higher Availability in Member Cluster Consolidated Environments Database Member Cluster Uses ASM Service W W W B
  22. 22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Oracle RAC Always Chooses the “Right Path” 22 • RAC determines the optimal path to serve blocks – network or disk • SSDs and NVMe storage technology continue to drive down latency • e.g. flash storage may provide better access times to data than the private network under high load • RAC takes those statistics into account Query BlockBlock Network congestion
  23. 23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Database Member Cluster Uses ASM Service 23 The “Right Path” Ensures Performance on Member Clusters Effectively reducing the impact of competing workloads overwhelming the I/O on the DSC B u s y
  24. 24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Zero Impact Patching • Zero Impact Patching enables patching of the Oracle Grid Infrastructure without interrupting database operations. • Patches are applied out-of-place and in a rolling fashion with one node being patched at a time while the database instance(s) on that node remain up and running. • Zero Impact Patching supports Oracle Real Application Clusters (RAC) databases on clusters with two or more nodes. 24 Never take down a database instance to patch Grid Infrastructure
  25. 25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Operational Details of Zero Impact Patching • During patch application, the stack is monitored by a special CSSD running out of place • This CSSD performs Heartbeat operations • Once the patch is done, the special CSSD concedes control to the CSSD running out of the patched home • The steps are performed on all the nodes until the entire cluster is patched. • Any interruptions in normal activity during patching will trigger the CSSD to evict the node. 25
  26. 26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Sequences cause Index block contention during heavy OLTP especially on tables which use sequence values as keys • This is not RAC specific but the problem gets amplified in RAC environments • The effects of this issue is mitigated via Scalable Sequences 26 Scalable Sequences
  27. 27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • New option in Create/Alter Sequence SCALE {Extend |NoExtend | NoScale } • When Scale is specified an internal algorithm uses the instance id and other session specific information to generate an offset • DBAs need to alter existing sequences • DBA/User/ALL_sequences will reflect the additional attributes 27 Scalable Sequences Operational details
  28. 28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda Sample Problems and Resolution 1 2 28 3 4
  29. 29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Sample Problems and Resolution 29 Problem 1 – Node Eviction
  30. 30. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Node Eviction – Node 1 ocssd.trc [ Part 1 ] • 2010-08-13 17:00:22.818: [ CSSD][4106599328] clssnmPollingThread: node anair2 (2) at 50% heartbeat fatal, removal in 14.520 seconds • 2010-08-13 17:00:29.833: [ CSSD][4106599328] clssnmPollingThread: node anair2 (2) at 75% heartbeat fatal, removal in 7.500 seconds • 2010-08-13 17:00:37.337: [ CSSD][4106599328] clssnmPollingThread: Removal started for node anair2 (2), flags • 2010-08-13 17:00:37.340: [ CSSD][4085619616]clssnmCheckSplit: Node 2, anair2, is alive, DHB (1281744040, 1396854) more than disk timeout of 27000 after the last NHB (1281744011, 1367154) 30 Network heartbeat missing from node 2 for 15 consecutive seconds Network heartbeat is still missing Finally eviction starts Node 2 is still updating the Voting disks
  31. 31. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Node Eviction – Node 1 ocssd.trc [ Part 2 ] • 2010-08-13 17:00:37.340: [ CSSD][4085619616](:CSSNM00007:) clssnmrEvict: Evicting node 2, anair2, from the cluster in incarnation 169934272, node birth incarnation 169934271, death incarnation 169934272, stateflags 0x24000 • 2010-08-13 17:01:07.705: [ CSSD][4043389856]clssgmCMReconfig: reconfiguration successful, incarnation 169934272 with 1 nodes, local node number 1, master node number 1 31 Eventually Node 2 eviction process starts Concluding with a reconfiguration
  32. 32. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Node Eviction – Node 2 ocssd.trc [ Part 1 ] • 2010-08-13 17:00:26.213: [ CSSD][4073040800] clssnmPollingThread: node anair1 (1) at 50% heartbeat fatal, removal in 14.540 seconds • 2010-08-13 17:00:40.702: [ CSSD][4073040800] clssnmPollingThread: Removal started for node anair1 (1), flags • 2010-08-13 17:00:40.706: [ CSSD][4052061088]clssnmCheckSplit: Node 1, anair1, is alive, DHB (1281744036, 1243744) more than disk timeout of 27000 after the last NHB (1281744007, 1214144) 32 Huh! So who is right? It also detects that Node 1 is still updating the Voting disks
  33. 33. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Node Eviction – Node 2 ocssd.trc [ Part 2 ] • 2010-08-13 17:00:40.707: [ CSSD][4052061088](:CSSNM00008:)clssnmCheckDskInfo: Aborting local node to avoid splitbrain. Cohort of 1 nodes with leader 2, anair2, is smaller than cohort of 1 nodes led by node 1, anair1, based on map type 2 • 2010-08-13 17:00:40.707: [ CSSD] [4052061088]################################### 2010-08-13 17:00:40.707: [ CSSD][4052061088]clssscExit: CSSD aborting from thread clssnmRcfgMgrThread 2010-08-13 17:00:40.707: [ CSSD] [4052061088]################################### 33 it correctly detects and aborts local node to prevent split brain And does the right thing
  34. 34. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | So what is the conclusion ? 34 Hmmm….I think it is the network.
  35. 35. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | What does OS Watcher say? • netstat does not show any issues # grep "zzz|udpInOverflows|ipReasmFails" • OSW data itself is missing possibly due to scheduling issues • Just prior to the issue Top reports top - 13:23:52 up 25 days, 21:08, 1 user, load average: 3.43, 3.01, 3.02 Cpu(s): 16.8%us, 23.2%sy, 0.0%ni, 56.5%id, 3.1%wa, 0.1%hi, 0.3%si, Mem: 74027752k total, 73689744k used, 338008k free, 1516k Swap: 16771852k total, 9069988k used, 7701864k free, 25836528k PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1049 root 11 -5 0 0 0 R 55.0 0.0 0:57.42 [kswapd0] 35 No issues seen in netstat & traceroute Oh no! It is swapping Why is the OSW data missing?
  36. 36. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Sample Problems and Resolution 36 Problem 2 – Slow I/O
  37. 37. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Start the Database on all instances to simulate physical reads • Simulate I/O on shared storage – Loading data (sqlldr,expdp) OR – Swingbench • **behavior may depend on your test setup, HCA, HBA etc 37 Simulate I/O performance issue swingbench swingbench
  38. 38. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | chactl query diagnosis-db 38 $ chactl query diagnosis -db sales -start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“ 2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_1) [detected] 2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_2) [detected] 2017-03-31 10:01:13.0 Database sales DB CPU Utilization (sales_2) [detected] 2017-03-31 10:01:33.0 Database salesDB Log File Switch (sales_1) [detected] Consolidates and displays information from all instances
  39. 39. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | What does OS Watcher say? • iostat confirms I/O performance issue archive/oswiostat/xxxxxxxx_iostat_17.03.31.1000.dat • Increase in Reads/writes Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda 0.00 0.00 0.00 4.00 0.00 32.00 8.00 0.00 0.25 0.25 0.10 xvdb 0.00 0.00 6.50 2.00 146.00 21.00 19.65 0.00 0.24 0.24 0.20 • --- • --- Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda 0.00 0.00 0.00 144.00 434.00 53.00 12.00 0.00 0.45 1.13 0.10 xvdb 0.00 0.00 6.50 219.00 414.00 67.00 13.65 0.00 0.33 9.44 0.20 39 There is more overall I/O as seen by the increase in the number of writes, reads * values may change depending on test env. **Output has been formatted for presentation
  40. 40. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Sample Problems and Resolution 40 Problem 3 – CPU bound
  41. 41. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Ensure Grid Infrastructure is running for at least an hour • Run some normal load • Simulate excessive CPU using a CPU stressing program on 3 nodes – stress -- C program 41 Simulate CPU load stress stress stress
  42. 42. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | chactl query diagnosis -cluster 42 $chactl query diagnosis –start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“ 2017-03-31 10:01:25.0 Host rwsxxxxx9 Host Memory Consumption [detected] 2017-03-31 10:01:29.0 Host rwsxxxxx0 Host Memory Consumption [detected] 2017-03-31 10:01:23.0 Host rwsxxxxx1 Host Memory Consumption [detected] Problem: Host Memory Consumption Description: CHA detected that more memory than expected is consumed on this server. The memory is not allocated by sessions of this database. Cause: The Cluster Health Advisor (CHA) detected an increase in memory consumption by other databases or by applications not connected to a database on this node. Action: Identify the top memory consumers by using the Cluster Health Monitor (CHM). Note: This is a 4 node cluster but the memory consumption issue is reported on 3 of the 4 nodes
  43. 43. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | chatcl actions & resolutions in text or html format 43
  44. 44. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | What does OS Watcher say? • OSWatcher confirms chactl diagnosis • mpstat (platform dependent) zzz ***Fri Mar 31 10:10:29 PST 2017 10:10:29 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:10:29 all 74.01 0.00 7.95 12.20 0.00 0.13 0.13 0.00 5.10 10:10:29 0 44.89 0.00 8.79 12.09 0.00 0.00 0.00 0.00 31.23 10:10:29 1 56.00 0.00 4.00 18.00 0.00 0.00 0.00 0.00 23.00 • Top also reports the CPU stress program Tasks: 454 total, 4 running, 450 sleeping, 0 stopped, 0 zombie Cpu(s): 94.9%us, 4.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20752 racusr 20 0 2498m 45m 39m R 85.9 0.3 752:32.40 stress 44 System is indeed CPU starved * values may change depending on test env. **Output has been formatted for presentation
  45. 45. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Sample Problems and Resolution 45 Problem 4 – Why did my Database Instance move to different node?
  46. 46. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • Provides common view of cluster- wide activities in a co-ordinated fashion • Customer readable summary of all actions in a cluster • Each activity is a summary of the issue – Further details are in the trace files 46 Clusterware Activity Log On Friday, my instances were running on Nodes 1 & 2, but today it is only running on Node 1
  47. 47. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 47 crsctl query calog -- Syntax $ crsctl query calog -h Usage: crsctl query calog [-aftertime <after_timestamp>] [-beforetime <before_timestamp>] [-duration <time_interval> | - follow] [-filter <filter_expression>] [-fullfmt | -xmlfmt] Lists the cluster activity log activities matching the specified criteria Where after_timestamp Time in format "YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]" OR "YYYY-MM-DD" OR "HH24:MI:SS[.FF][[+-]HH:MM]" before_timestamp Time in format "YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]" OR "YYYY-MM-DD" OR "HH24:MI:SS[.FF][[+-]HH:MM]" time_interval Time interval in format "DD HH:MM:SS" filter_expression Refer to product documentation. -follow Gives continuous output -fullfmt Detail output format -xmlfmt XML output format
  48. 48. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 48 crsctl query calog output $ crsctl query calog -aftertime "2017-03-08 15:09:46.522-07:00" 2017-04-12 20:05:04.668000 : Attempting to start 'ora.anair1.vip' on ‘anair1' : 14920191617156230/1194/11 : ….. 2017-04-12 20:05:06.559000 : Attempting to start 'ora.LISTENER.lsnr' on ‘anair1' : 14920191617156230/1194/16 … 2017-04-12 20:05:32.038000 : Start of 'ora.FRA.dg' on ‘anair1' succeeded : 14920191617156230/1194/27 : 2017-04-12 20:05:32.040000 : Attempting to start 'ora.sales.db' on ‘anair1' : 14920191617156230/1194/28 : 2017-04-12 20:05:59.415000 : Start of 'ora.sales.db' on ‘anair1' succeeded : 14920191617156230/1194/30 : -- Format of output records is: DATE & TIME (YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]): Event text: ACTID Possibly network issues caused VIP relocation
  49. 49. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 49 crsctl query calog filter output using actid $ crsctl query calog -filter "actid == 14920191617156230/2449732/1" 2017-08-03 16:30:24.678000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14920191617156230/2449732/1 : 2017-08-03 16:30:24.698000 : Start of 'ora.sscdb.db' on ‘anair1' succeeded : 14920191617156230/2449732/1 : $ crsctl query calog -filter "actid ~= “14920191617156230” 2017-08-03 16:25:20.658000 : Stop of 'ora.sscdb.test.svc' on anair1' succeeded : 14920191617156230/2449007/2 : -filter: Use ~= or == on actid to find related actions
  50. 50. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | crsctl query calog filtered output using name $crsctl query calog -filter "entity_name == ora.sscdb.db" 2017-04-12 18:59:28.330000 : Attempting to start 'ora.sscdb.db' on ‘anair2' : 14919830844562257/9074/1 : 2017-04-12 18:59:28.334000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14919830844562257/9074/2 : 2017-04-12 18:59:28.340000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14919830844562257/9074/3 : 2017-04-12 18:59:28.694000 : Resource 'ora.sscdb.db' has been modified. : 14919830844562257/9074/4 : 2017-04-12 18:59:28.800000 : Resource 'ora.sscdb.db' has been modified. : 14919830844562257/9074/5 : … 2017-04-12 19:32:39.631000 : Attempting to stop 'ora.sscdb.db' on ‘anair2' : 14919830844562257/9853/2 : 2017-04-12 19:32:42.572000 : Stop of 'ora.sscdb.db' on ‘anair2' succeeded : 14919830844562257/9853/4 : 2017-04-12 19:45:01.175000 : Attempting to stop 'ora.sscdb.db' on ‘anair1' : 14919830844562257/10151/4 : 2017-04-12 19:45:04.176000 : Stop of 'ora.sscdb.db' on ‘anair1' succeeded : 14919830844562257/10151/7 : 50
  51. 51. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Additional examples for querying calog • crsctl query calog -duration "00 00:02:00“ << last two minutes >> • crsctl query calog -duration "00 02:00:00“ << last two hours >> • crsctl query calog –follow << like tail… >> • crsctl query calog -aftertime "2017-03-08 17:49:12.397" –follow • crsctl query calog -filter "entity_name == foo“ << activity on foo >> • crsctl query calog -filter "actid == 145567015239630019/49543/1" • crsctl query calog –xmlfmt << in xml format >> 51
  52. 52. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda FAQ 1 2 52 3 4
  53. 53. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 53 Frequently asked Question # 1 Why does MGMT DB need so much space?
  54. 54. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Cluster Type Redundancy MGMT DG (GB) Domain Services Cluster (2 Node DSC with 4 Member Clusters of 2 Nodes each) External 188 Normal 376 High 564 Flex 376 Standalone Cluster (4 Node Cluster) External 38 Normal 76 High 114 Flex 76 • Oracle GI 12c Release 2 feature AHF (Autonomous Health Framework) collects, Co-relates & stores diagnostics data from OS and DB in MGMT • In DSC, one PDB per member cluster is provisioned to store member cluster diagnostics data • The data is used by AHF components like cluster health advisor to both prevent and help diagnose issues 54 GIMR space requirements
  55. 55. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 55 Frequently asked Question # 2 Should I continue to use Oswatcher?
  56. 56. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Should I continue to use OSWatcher? • Hopefully by now, the value benefits of AHF is clear • AHF continues to be enhanced to improve – Diagnosing large number of deployments – Correlating data to speed diagnosis – Preventing issues in the first place – Utilizing latest technologies like Machine learning • AHF is mandatory (12.2+) but customers can choose to use OSW along with AHF 56 Learn more about Autonomous Health Framework https://www.slideshare.net/MarkusMichalewicz/cl uster-health-advisor-cha-deep-dive-by-mark- scardina
  57. 57. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • opatch automatically patches MGMT database if required • clients of MGMT connect using encrypted credentials • MGMT listener automatically maintained by the clusterware agent 57 Frequently asked Question # 3 But it is still one more database for me to manage
  58. 58. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 58
  59. 59. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 59

    Be the first to comment

    Login to see the comments

  • SeanScott5

    Apr. 10, 2019
  • LucreciaPaolini

    Jun. 27, 2019
  • AnilNair27

    Jan. 24, 2020
  • ssuser5a05c7

    Apr. 2, 2020

Oracle RAC Operational Best Practices as presented at Sangam 2017

Views

Total views

669

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

98

Shares

0

Comments

0

Likes

4

×