Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Monitoring 改造計畫:流程觀點

Monitoring: a process perspective
Video: http://www.ithome.com.tw/video/108317

【Monitoring 改造計畫:流程觀點】

組織會變化,業務會變化,流程會變化,軟體系統會變化,同樣的,在背後綜觀全域的監控系統,也會變化。如果對監控系統也有版本編號的話,葉秉哲在 2015 年發表的「Whoscall 的 Realtime Monitoring 經驗分享」,是我們 1.0 版的經驗;這一年來,我們已經進化到 3.0 版。

這場演講,葉秉哲會從流程觀點,分享對監控系統持續改善的原因及作法。

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Monitoring 改造計畫:流程觀點

  1. 1. William Yeh DevOps Summit 2016 (2016-07-06) Monitoring 
 Monitoring: a Process Perspective
  2. 2. 
 
 
 
 
 CRT (Current Reality Tree) #6 #10 ( ) best practice #3 #5 ( ) ( )( ) AND AND AND AND #1#2 #4 #7 #8 #9 #11 Murphy exists AND AND × AND DevOps AND AND AND AND AND AND AND AND Op AND http://www.slideshare.net/williamyeh/devops-63711710
  3. 3. #3 #1#2 #7 #8 #9 #11 Murphy exists AND AND AND AND AND AND
  4. 4. #1#2 #4 #8 #9 Murphy exists AND AND × AND AND AND
  5. 5. #7 
 
 #9 #1 #2 
 … #11 … #8 #4
  6. 6. Risk management • Threats • avoid • transfer • mitigate 7 • Opportunities • exploit • enhance • share 👍👎 http://www.slideshare.net/williamyeh/whoscall-realtime-monitoring
  7. 7. William Yeh DevOps Summit 2016 (2016-07-06) Monitoring 
 Monitoring: a Process Perspective
  8. 8. Process Monitoring Monitoring
  9. 9. Process ? ? ? ? Monitoring Monitoring
  10. 10. ? ? Process Monitoring
  11. 11. #5 
 
 
 
 
 
 
 Part 2
  12. 12. 
 Efrat Goldratt-Ashlag
  13. 13. 
 Efrat Goldratt-Ashlag
  14. 14. What to change To What to change How to cause the change 

  15. 15. 
 
 
 
 
 CRT (Current Reality Tree)
  16. 16. 
 

  17. 17. 
 
 DevOps
  18. 18. 
 DevOps 
 
 leverage
 
 ✘ TOC CCPM 
 FRT (Future Reality Tree)
  19. 19. 
 DevOps 
 
 leverage
 
 ✘ TOC CCPM 
 FRT (Future Reality Tree)
  20. 20. TOC CCPM
  21. 21. Stephen R. Covey
  22. 22.
  23. 23. What get measured, get done. 
 Peter Drucker
  24. 24. Policy What get measured, get done.
  25. 25. Policy 
 
 

  26. 26. 
 
 Policy
  27. 27. Policy Buy-in Policy
  28. 28. What to change To What to change How to cause the change 

  29. 29. Adrian Cockcroft
  30. 30. CloudFront ELB API servers MongoDB Cloud Manager CloudWatch log in S3 StatsD BigQuery
  31. 31. CloudFront ELB API servers MongoDB Cloud Manager CloudWatch log in S3 StatsD BigQuery
  32. 32. CloudFront ELB API servers MongoDB Cloud Manager CloudWatch log in S3 BigQuery
  33. 33. CloudFront ELB API servers MongoDB Cloud Manager CloudWatch log in S3 BigQuery
  34. 34. 
 

  35. 35. http://school.soft-arch.net/blog/125009/change-viewpoint-on-lord-of-rings Lean Change Canvas
  36. 36. Lean Change Canvas Commitment Wins/Benefits Urgency
 
 Target State Success Criteria Vision 
 Communication 
 Action Change Recipients FYI: http://kojenchieh.pixnet.net/blog/post/442550432-firstthing_of_agile_promotion
 FYI: http://leankit.com/blog/2015/02/lean-change-method/ Monitoring Q1 (brainstorming) 2016-Jan-06 Iteration #1 
 
 
 TO DO LIST details Augmented
  37. 37. Lean Change Canvas gency
 
 Target State Success Criteria Vision 
 Communication 
 Action Monitoring Q1 (brainstorming) 2016-Jan-06 Iteration #1
  38. 38. What to change To What to change How to cause the change 

  39. 39. Lean Change Canvas gency
 
 Target State Success Criteria Vision 
 Communication 
 Action Monitoring Q1 (brainstorming) 2016-Jan-06 Iteration #1 Flow Tech Monitoring
  40. 40. Buy-in Flow Buy-in Policy
  41. 41. Flow TOC Lean Thinking CCPM
  42. 42. TOC
  43. 43. Lean Thinking Value Value stream Flow Pull Perfection http://school.soft-arch.net/blog/115652/devops-a-lean-perspective
  44. 44. “The Three Ways” Create fast flow of work from Dev into IT Ops. Shorten and amplify feedback loops. Create a culture that simultaneously fosters 2 things: 1. continual experimentation, learning from failure. 2. repetition and practice is the prerequisite to mastery. Create fast flow of work from Dev into IT Ops.
 Shorten and amplify feedback loops.
  45. 45. CCPM Critical Chain
 Project Management
  46. 46. Flow TOC Lean Thinking CCPM
  47. 47. VPC CloudFront ELB API servers DB Simplified version
  48. 48. CloudFront ELB API servers DB ELB API servers DB Microservices Simplified version
  49. 49. Flow
  50. 50. Flow
  51. 51. Flow
  52. 52. Flow
  53. 53. Overview
  54. 54. Incoming
 requests
  55. 55. API
 servers
  56. 56. DB servers
  57. 57. DB servers API
 servers Incoming
 requests Overview Flow
  58. 58. Lean Change Canvas gency
 
 Target State Success Criteria Vision 
 Communication 
 Action Monitoring Q1 (brainstorming) 2016-Jan-06 Iteration #1 Flow
  59. 59. TOC Flow TOC
  60. 60. Flow Buy-in Policy Tech Flow
  61. 61. Lean Change Canvas gency
 
 Target State Success Criteria Vision 
 Communication 
 Action Monitoring Q1 (brainstorming) 2016-Jan-06 Iteration #1 Tech
  62. 62. Personal Preferences • Golang • Microservices • Composability • OSS ecosystem of server technologies
  63. 63. Personal Preferences • Golang • Microservices • Composability • OSS ecosystem Runtime dependency william Ansible
  64. 64. Personal Preferences • Golang • Microservices • Composability • OSS ecosystem Scalability Overhead
  65. 65. Personal Preferences • Golang • Microservices • Composability • OSS ecosystem Node/system metrics exporter AWS CloudWatch exporter Blackbox exporter Collectd exporter Consul exporter Graphite exporter HAProxy exporter InfluxDB exporter JMX exporter Memcached exporter Mesos task exporter MySQL server exporter SNMP exporter StatsD exporter cAdvisor Doorman Etcd Kubernetes-Mesos Kubernetes RobustIRC SkyDNS Weave Flux Aerospike exporter Apache exporter BIG-IP exporter BIND exporter Ceph exporter CouchDB exporter Django exporter Google's mtail log data extractor Heka dashboard exporter Heka exporter IoT Edison exporter Jenkins exporter knxd exporter Meteor JS web framework exporter Minecraft exporter module Mirth Connect exporter MongoDB exporter Munin exporter New Relic exporter Nginx metric library NSQ exporter OpenWeatherMap exporter Passenger exporter PgBouncer exporter PostgreSQL exporter PowerDNS exporter RabbitMQ exporter RabbitMQ Management Plugin exporter Rancher exporter Redis exporter RethinkDB exporter rTorrent exporter scollector exporter SMTP/Maildir MDA blackbox prober Speedtest.net exporter SQL query result set metrics exporter Ubiquiti UniFi exporter Varnish exporter Zookeeper exporter
  66. 66. CloudFront ELB API servers MongoDB Cloud Manager CloudWatch log in S3 StatsD BigQuery
  67. 67. ELB API servers MongoDB Cloud Manager CloudWatch
  68. 68. Prometheus vs Graphite/StatsD
  69. 69. abs() absent() bottomk() ceil() changes() clamp_max() clamp_min() count_scalar() delta() deriv() drop_common_labels() exp() floor() histogram_quantile() holt_winters() increase() irate() label_replace() ln() log2() log10() predict_linear() rate() resets() round() scalar() sort() sort_desc() sqrt() time() topk() vector() <aggregation>_over_time()
  70. 70. node_cpu time number
  71. 71. node_cpu time number {mode="idle"} mode node_cpu {mode="irq"} node_cpu {instance="10.0.37.12"} {service="web"} {zone="ap-northest-1a"}
  72. 72. sum( irate( node_netstat_TcpExt_TCPTimeWaitOverflow[1m] ) ) by (ec2tag_Service) counter gauge aggregate TCP Timeout node_netstat_TcpExt_TCPTimeWaitOverflow[1m] irate( node_netstat_TcpExt_TCPTimeWaitOverflow[1m] ) grouping
  73. 73. gauge aggregate Memory Used 1 - node_memory_MemFree/node_memory_MemTotal grouping avg( 1 - node_memory_MemFree/node_memory_MemTotal ) by (ec2tag_Service)
  74. 74. avg by (ec2tag_Service) ( irate( node_cpu{job="node", mode="idle"}[1m] ) ) counter gauge aggregate CPU Utilization 100 - ( * 100)
  75. 75. avg( request_time_summary ) by (ec2tag_Service, quantile) summary aggregate Latency grouping Customized metrics
 with Fluentd plugin for Prometheus
  76. 76. Conclusion
  77. 77. #7 
 
 #9 #1 #2 
 … #11 … #8 #4
  78. 78. Policy Buy-in Flow Tech
  79. 79. Policy Buy-in Flow Tech ??? Issue tracking
  80. 80. William Yeh DevOps Summit 2016 (2016-07-06) Monitoring 
 Monitoring: a Process Perspective

×