Monitoring: a process perspective Video: http://www.ithome.com.tw/video/108317 【Monitoring 改造計畫：流程觀點】 組織會變化，業務會變化，流程會變化，軟體系統會變化，同樣的，在背後綜觀全域的監控系統，也會變化。如果對監控系統也有版本編號的話，葉秉哲在 2015 年發表的「Whoscall 的 Realtime Monitoring 經驗分享」，是我們 1.0 版的經驗；這一年來，我們已經進化到 3.0 版。 這場演講，葉秉哲會從流程觀點，分享對監控系統持續改善的原因及作法。

1. William Yeh
DevOps Summit 2016 (2016-07-06)
Monitoring
Monitoring: a Process Perspective

3.
CRT (Current Reality Tree)
#6
#10
( )
best practice
#3 #5
( ) ( )( )
AND
AND
AND
AND
#1#2
#4
#7
#8
#9
#11
Murphy exists
AND
AND
×
AND
DevOps
AND
AND
AND
AND
AND
AND
AND
AND
Op
AND
http://www.slideshare.net/williamyeh/devops-63711710

4. #3
#1#2
#7
#8
#9
#11
Murphy exists
AND
AND
AND
AND
AND
AND

5. #1#2
#4
#8
#9
Murphy exists
AND
AND
×
AND
AND
AND

6. #7
#9 #1
#2
…
#11
…
#8 #4

7. Risk management
• Threats
• avoid
• transfer
• mitigate
7
• Opportunities
• exploit
• enhance
• share
👍👎
http://www.slideshare.net/williamyeh/whoscall-realtime-monitoring

8. William Yeh
DevOps Summit 2016 (2016-07-06)
Monitoring
Monitoring: a Process Perspective

9. Process Monitoring
Monitoring

10. Process
?
?
?
?
Monitoring
Monitoring

11. ?
?
Process
Monitoring

12. #5
Part 2

13.
Efrat Goldratt-Ashlag

17.
Efrat Goldratt-Ashlag

18. What to change
To What to change
How to cause the change

19.
CRT (Current Reality Tree)

21.
DevOps

22.
DevOps
leverage
✘
TOC
CCPM
FRT (Future Reality Tree)

23.
DevOps
leverage
✘
TOC
CCPM
FRT (Future Reality Tree)

24. TOC
CCPM

25. Stephen R. Covey

28. What get measured, get done.
Peter Drucker

29. Policy
What get measured, get done.

30. Policy

31.
Policy

32. Policy
Buy-in
Policy

35. What to change
To What to change
How to cause the change

36. Adrian Cockcroft

39. CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
StatsD
BigQuery

40. CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
StatsD
BigQuery

41. CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
BigQuery

42. CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
BigQuery

44. http://school.soft-arch.net/blog/125009/change-viewpoint-on-lord-of-rings
Lean Change Canvas

45. Lean Change Canvas
Commitment Wins/Benefits
Urgency
Target State
Success Criteria
Vision
Communication
Action
Change Recipients
FYI: http://kojenchieh.pixnet.net/blog/post/442550432-firstthing_of_agile_promotion
FYI: http://leankit.com/blog/2015/02/lean-change-method/
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1
TO DO LIST
details
Augmented

46. Lean Change Canvas
gency
Target State
Success Criteria
Vision
Communication
Action
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1

47. What to change
To What to change
How to cause the change

48. Lean Change Canvas
gency
Target State
Success Criteria
Vision
Communication
Action
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1
Flow
Tech
Monitoring

49. Buy-in
Flow
Buy-in
Policy

50. Flow
TOC Lean Thinking
CCPM

51. TOC

52. Lean Thinking
Value
Value stream
Flow
Pull
Perfection
http://school.soft-arch.net/blog/115652/devops-a-lean-perspective

53. “The Three Ways”
Create fast flow of work from Dev into IT Ops.
Shorten and amplify feedback loops.
Create a culture that simultaneously fosters 2
things:
1. continual experimentation, learning from
failure.
2. repetition and practice is the prerequisite
to mastery.
Create fast flow of work
from Dev into IT Ops.
Shorten and amplify
feedback loops.

54. CCPM
Critical Chain
Project Management

55. Flow
TOC Lean Thinking
CCPM

56. VPC
CloudFront ELB API servers DB
Simplified version

57. CloudFront ELB API servers DB
ELB API servers DB
Microservices
Simplified version

58. Flow

62. Flow

63. Flow

65. Flow

67. Overview

68. Incoming
requests

69. API
servers

70. DB servers

71. DB servers
API
servers
Incoming
requests
Overview
Flow

72. Lean Change Canvas
gency
Target State
Success Criteria
Vision
Communication
Action
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1
Flow

73. TOC
Flow
TOC

74. Flow
Buy-in
Policy
Tech
Flow

75. Lean Change Canvas
gency
Target State
Success Criteria
Vision
Communication
Action
Monitoring Q1 (brainstorming)
2016-Jan-06
Iteration #1
Tech

76. Personal Preferences
• Golang
• Microservices
• Composability
• OSS ecosystem
of server technologies

77. Personal Preferences
• Golang
• Microservices
• Composability
• OSS ecosystem
Runtime dependency
william Ansible

78. Personal Preferences
• Golang
• Microservices
• Composability
• OSS ecosystem
Scalability
Overhead

85. Personal Preferences
• Golang
• Microservices
• Composability
• OSS ecosystem
Node/system metrics exporter
AWS CloudWatch exporter
Blackbox exporter
Collectd exporter
Consul exporter
Graphite exporter
HAProxy exporter
InfluxDB exporter
JMX exporter
Memcached exporter
Mesos task exporter
MySQL server exporter
SNMP exporter
StatsD exporter
cAdvisor
Doorman
Etcd
Kubernetes-Mesos
Kubernetes
RobustIRC
SkyDNS
Weave Flux
Aerospike exporter
Apache exporter
BIG-IP exporter
BIND exporter
Ceph exporter
CouchDB exporter
Django exporter
Google's mtail log data extractor
Heka dashboard exporter
Heka exporter
IoT Edison exporter
Jenkins exporter
knxd exporter
Meteor JS web framework exporter
Minecraft exporter module
Mirth Connect exporter
MongoDB exporter
Munin exporter
New Relic exporter
Nginx metric library
NSQ exporter
OpenWeatherMap exporter
Passenger exporter
PgBouncer exporter
PostgreSQL exporter
PowerDNS exporter
RabbitMQ exporter
RabbitMQ Management Plugin exporter
Rancher exporter
Redis exporter
RethinkDB exporter
rTorrent exporter
scollector exporter
SMTP/Maildir MDA blackbox prober
Speedtest.net exporter
SQL query result set metrics exporter
Ubiquiti UniFi exporter
Varnish exporter
Zookeeper exporter

86. CloudFront ELB API servers MongoDB
Cloud Manager
CloudWatch
log in
S3
StatsD
BigQuery

87. ELB API servers MongoDB
Cloud Manager
CloudWatch

88. Prometheus vs Graphite/StatsD

89. abs()
absent()
bottomk()
ceil()
changes()
clamp_max()
clamp_min()
count_scalar()
delta()
deriv()
drop_common_labels()
exp()
floor()
histogram_quantile()
holt_winters()
increase()
irate()
label_replace()
ln()
log2()
log10()
predict_linear()
rate()
resets()
round()
scalar()
sort()
sort_desc()
sqrt()
time()
topk()
vector()
<aggregation>_over_time()

90. node_cpu
time
number

91. node_cpu
time
number
{mode="idle"}
mode
node_cpu {mode="irq"}
node_cpu {instance="10.0.37.12"}
{service="web"}
{zone="ap-northest-1a"}

92. sum(
irate(
node_netstat_TcpExt_TCPTimeWaitOverflow[1m]
)
) by (ec2tag_Service)
counter
gauge
aggregate
TCP Timeout
node_netstat_TcpExt_TCPTimeWaitOverflow[1m]
irate(
node_netstat_TcpExt_TCPTimeWaitOverflow[1m]
)
grouping

93. gauge
aggregate
Memory Used
1 - node_memory_MemFree/node_memory_MemTotal
grouping
avg(
1 - node_memory_MemFree/node_memory_MemTotal
) by (ec2tag_Service)

94. avg by (ec2tag_Service) (
irate(
node_cpu{job="node", mode="idle"}[1m]
)
)
counter
gauge
aggregate
CPU Utilization
100 - (
* 100)

95. avg(
request_time_summary
) by (ec2tag_Service, quantile)
summary
aggregate
Latency
grouping
Customized metrics
with Fluentd plugin for Prometheus

96. Conclusion

97. #7
#9 #1
#2
…
#11
…
#8 #4

98. Policy
Buy-in
Flow
Tech

99. Policy
Buy-in
Flow
Tech
???
Issue tracking

100. William Yeh
DevOps Summit 2016 (2016-07-06)
Monitoring
Monitoring: a Process Perspective