Today, AdRoll runs its infrastructure by instrumentation: constantly asking empirical questions, analyzing data for answers, and designing new features with instrumentation in mind to understand how functionality will work upon release. AdRoll’s development methodology did not start out this way, however. It took a cultural shift and many new tools and processes to adopt this approach. In this session, AdRoll and Datadog will discuss how to evolve your organization from a state of “flying blind” to a culture focused on monitoring and data-based decisions.  Session sponsored by Datadog.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DVO205
The AdRoll Monitoring Evolution:
From Flying Blind to Flying by Instrument
Brian Troutwine, AdRoll
Ilan Rabinovitch, Datadog
October 2015

2. Today’s speakers
Ilan Rabinovitch
Dir. Technical Community
Datadog
Brian Troutwine
Sr. Software Engineer
AdRoll

3. Quick Overview of
Datadog
• Monitoring for modern applications
• Dynamic Infrastructure
• Microservices
• Time series storage of metrics and events
• 100s of built in integrations
• Eg. EC2, ELB, ECS and more.

4. CAMS
Culture
Automation
Metrics
Sharing
Damon Edwards and John Willis

5. CAMS
Culture
Automation
METRICS
SHARING

6. You’re in the cloud and it's everything you dreamed of!

7. You’re in the cloud and it's everything you dreamed of!
Autoscaling
Container
orchestration
Infinite storage

8. In cloud we trust.
But how do we verify health?

9. If it moves, monitor it.

10. How does your current monitoring fit in?

12. • Host-centric
How does our current monitoring fit in?

13. • Host-centric
• Static configurations tracking dynamic infrastructure
How does our current monitoring fit in?

14. • Host-centric
• Static configurations tracking
dynamic infrastructure
• Focused on resources, rather than
work
How does our current monitoring fit in?

15. • Host-centric
• Static configurations tracking dynamic infrastructure
• Focused on resources, rather than work
• Difficult to pull together and compare data from
multiple sources
How does our current monitoring fit in?

16. So what to monitor?
More at: http://goo.gl/t1Rgcg

17. How to use that data?
More at: http://goo.gl/t1Rgcg

18. Recurse until you find root cause

19. Query-based monitoring
• Aggregates matter because the underlying infrastructure is dynamic
• Express our monitors or alerts as queries on predicates:
• “avg response time for requests to hosts running nginx > 500
ms”
• “min # of hosts running nginx < 3”
• Mash up data sources for a 360-degree view of a problem

20. Query-based monitoring
“Show me iowait across nginx hosts, grouped by
availability zone”

23. Real-time
bidding

24. The problem domain
• Low latency (< 100 ms per transaction)
• Firm real-time system
• Highly concurrent ( ~2 million transactions
per second, peak)
• Global, 24/7 operation

25. In the early days of the
AdRoll real-time bidding
(RTB) project, we could
use our intuition.

26. • The system was simple.
• The number of total
requests was small.
• The impact of mistakes
was minor.

27. We could be reasonably
confident that our mental
model of the system’s
behavior was accurate.

28. The trouble with a
complex system is that its
behavior in practice gets
away from you pretty fast.

29. Our first approach was
to batch process logs
generated by
individual bidders.
Batch processing

30. Pros:
• We were already doing this.
• It’s simple to implement.
• It’s straightforward to
conceive.
Batch processing

31. Cons:
• High update latency
• Catastrophic errors lose logs
• Denies impulse
experimentation
Batch processing

32. Our second approach
was to generate coarse
real-time metrics and
analyze those.
Coarse real-time metrics

33. Pros:
• Iterative step up from
batch processing
• Proves out the concept
• Simple to implement
Coarse real-time metrics

34. Cons:
• Still relied on intuition
• Bidder implementation was
sub-optimal
• Dashboards were one-size-
fits-all approach
Coarse real-time metrics

35. By this point, the complexity of the
system and our ambitions were
growing.
• Two engineers were added to the
team.
• Tens more in the department.
• RTB became a central project.

36. We were making
decisions in a
knowledge void.

37. At this point, we have
AWS CloudWatch.

38. CloudWatch
reports the basic
health of your
system.

39. CloudWatch provides
the total view of the
AWS services you’re
using.

40. What we don’t have at
this point is a detailed
view of our system.

41. What we don’t have is
the ability to explore the
information we have,
especially in high-
stress situations.

42. Exometer solves our
Erlang-side problem.
Detailed application-
level instrumentation is
cheap and easy.

43. Datadog solves
our aggregation,
visualization, and
alerting problems.

44. Datadog integrates with
CloudWatch. Our system-
specific metrics can be
correlated with the basic
health of the system.

45. This can be done in real time.
Correlation of system information

46. This can be done in
high-stress situations.
Correlation of system information

47. This can be done by
other departments of
the business.
Correlation of system information

48. A bid “times out”
when we don’t reply
back to the exchange
in 100 ms.
Timeout spikes

49. Timeout spikes

50. We didn’t realize this
was happening. It’s an
early win of our
sophisticated monitoring.
Timeout spikes

51. Timeout spikes

52. System load is normal.
There’s not a periodic
spike in bid request traffic.
Timeout spikes

53. Timeout spikes

54. There is a correlated
jump in network
traffic, however.
Timeout spikes

55. Timeout spikes

56. There were also
correlated spikes in
the Erlang VM’s
process run queue.
Timeout spikes

57. • VM scheduler threads are locked to CPU
• CPU-intensive background process kicks
on every 20 minutes
• No CPU shield on the server
• VM scheduler thread gets kicked from its
assigned CPU, processes back up
Timeout spikes

58. Failure of bid-
request traffic is an
all-hands problem.
Traffic crash

59. Without traffic, the
bidders can do nothing.
Traffic crash

60. Traffic crash

61. That’s a healthy couple
of days' worth of traffic.
It dips in the night, and
climbs in the day.

62. Traffic crash

63. This is a weekend’s
worth of traffic lost.
Traffic crash

64. • Confirmed with CloudWatch that
networking to the machines was fine
• No changes had been made to the
production system (it was a looser
time)
• All detail metrics from the Erlang VM
are acceptable
Traffic crash

65. Traffic crash

66. The exchange
confirmed a drop in
traffic from their system.
Traffic crash

67. Turns out, we hit an
implicit exchange
limitation.
Traffic crash

68. We also became more
conscientious about
alerting effectively.
Traffic crash

69. At high scale, it’s very
easy to have to be
over-provisioned for
the system’s load.
Sophisticated autoscaling

70. Worse, it’s very easy to
be under provisioned
for system load.
Sophisticated autoscaling

71. All CloudWatch alarms
on EC2 instances can
be pressed into service
for autoscaling.
Sophisticated autoscaling

72. Our first autoscaling
approach used
remaining idle CPU.
Sophisticated autoscaling

73. Sophisticated autoscaling

74. Sophisticated autoscaling
As traffic drops off at
the end of the day,
we need less CPU
time to process it.

75. Sophisticated autoscaling

76. This was great! We
immediately saved
loads of money.
Sophisticated autoscaling

77. Problem was, it’s an
indirect measurement.
There’s always some
nuance you’ll miss.
Sophisticated autoscaling

78. Co-resident subsystems
eat into the CPU time,
giving an inaccurate
impression.
Sophisticated autoscaling

79. CPU consumption
carries no information
about aberrant
system issues.
Sophisticated autoscaling

80. What can be done?
Sophisticated autoscaling

81. Distill the performance
capability of your
system into a single
signal.
Sophisticated autoscaling

82. Sophisticated autoscaling

83. Sophisticated autoscaling
The "metadata index"
tracks the load on the
bidders. It’s a weighted
sum of key metrics.

84. Sophisticated autoscaling
As traffic drops, the
metadata index
drops. Indirectly, idle
CPU increases.

85. Sophisticated autoscaling
We emit this
metadata index into
CloudWatch as a
custom metric.

86. Sophisticated autoscaling
As soon as it hits
CloudWatch, you
can autoscale on it.

87. Sophisticated autoscaling

88. Sophisticated autoscaling

89. Sophisticated autoscaling

90. Sophisticated autoscaling
This is twice as efficient
as the CPU idle scaling
signal. One-half the
number of machines.

91. Sophisticated autoscaling

92. There’s a lot of fraud in
the online advertising
industry.
Anti-fraud CookieBouncer

93. A certain kind of “hot
cookie” fraud
caused a tolerable
fault in the bidders.
Anti-fraud CookieBouncer

94. Anti-fraud CookieBouncer

95. CookieBouncer
blocks bidding on
fraudulent, “hot,”
cookies in real time.
Anti-fraud CookieBouncer

96. Our concern was
blocking too much traffic,
in turn blocking
legitimate bids through
over-aggressive tuning.
Anti-fraud CookieBouncer

97. We built a new
CookieBouncer
dashboard and introduced
the ability to tune it in real
time on every bidder.
Anti-fraud CookieBouncer

98. We rolled CookieBouncer
out with conservative
settings and started
adjusting, keeping tabs
on the key indicators.
Anti-fraud CookieBouncer

99. Anti-fraud CookieBouncer

100. We adjusted and were
very surprised at the total
number of blocked
cookies and the
percentage of total traffic.
Anti-fraud CookieBouncer

101. Anti-fraud CookieBouncer

102. Anti-fraud CookieBouncer

103. The instrumentation
speaks for itself.
Anti-fraud CookieBouncer

104. Anti-fraud CookieBouncer

105. Learn more at….
DVO204 - Monitoring Strategies: Finding Signal in the Noise
Thursday, Oct 8, 11:00 AM - 12:00 PM
OR
http://bit.ly/1Qo4Zmy

106. Thank you!

107. Remember to complete
your evaluations!