This presentation provides practical guidance using external agent-based measurements and real user monitoring techniques. We review common content delivery network (CDN) architectures and how they relate to performance measurement. Finally, we walk through real-world CDN performance monitoring implementations used by MapBox, Amazon.com, and Amazon CloudFront.

1. #reinvent

2. Why Measure?
“For e-commerce sites, a one
second delay in web page
response time decreased
conversions by 7%”
Source: Aberdeen Group
Source: Compuware Gomez

5. What are they downloading?
Where are they?
How does that influence my
measurement?

6. Identify Customer Vital Signs
Customer Experience Metrics Diagnostic Metrics

7. Identify Customer Experience Metrics
Availability
Download Latency

8. Availability
# of concurrent requests
Customers want to
reach your content

9. Download Latency
Avg. US Latency 1.8 KB Object
Customers want fast 420
410
400
downloads 390
380
370
360
350
340
330
320

11. Service-Side
External
Real User

12. Customer Ease of Diagnostic
Experience Setup Data
Service-Side ? ? ?
External ? ? ?
Real User ? ? ?

13. Service-Side
External
Real User

14. Service-Side Measurement
• First-byte latency
• Server utilization
• Edge-cache hits
• W3C access logs

15. Service-Side Measurement
Customer Ease of Diagnostic
Experience Setup Data
Service-Side D B A
External ? ? ?
Real User ? ? ?

16. Service-Side
External
Real User

17. External Measurement
GET /test_object.png
Amazon
CloudFront

18. External Measurement

19. Path of a typical web request
Amazon
CloudFront
Web Client DNS Server CDN Cache Web Content

20. First Request
GET /cat_video.mp4Miss: 100ms
Lookup video.amazon.com Miss: 1sec
Amazon
CloudFront
Web Client DNS Server CDN Cache Web Content

21. Second Request: Cache is Primed
GET /cat_video.mp4
Amazon
CloudFront
Web Client DNS Server CDN Cache Web Content

22. Amazon
CloudFront
Web Client DNS Server CDN Cache Web Content

23. Amazon
CloudFront
Web Client DNS Server Network CDN Cache Web Content

24. 23 ms
300 ms
42
ms

25. Object
Popularity
Viewer Network
Location Performance
Viewer
Experience

26. Internet Backbone
2 ms
Regional
ISP
Amazon
CloudFront
40 ms Metro
ISP

27. External Measurement
Customer Ease of Diagnostic
Experience Setup Data
Service-Side D B A
External C B B
Real User ? ? ?

28. Service-Side
External
Real User

29. Real-User Measurement
Download Time

30. Real User Measurement
Telemetry Data
Web Browser Monitoring Aggregator

32. var startTime = new Date().getTime();
var testObjectUrl = ”/test_object.png”;
//Measure round trip to the edge
<script /> $.ajax({
url: testObjectUrl,
success: function(data) {
var endTime = new Date().getTime();
publishMeasurement(endTime - startTime);
}
});

33. var startTime = new Date().getTime();
var testObjectUrl = ”/test_object.png”;
Request Test
Object //Measure round trip to the edge
$.ajax({
url: testObjectUrl,
success: function(data) {
var endTime = new Date().getTime();
publishMeasurement(endTime - startTime);
}
});

34. var startTime = new Date().getTime();
var testObjectUrl = ”/test_object.png”;
Wrapped in
JavaScript //Measure round trip to the edge
Timers*
Timers $.ajax({
url: testObjectUrl,
success: function(data) {
var endTime = new Date().getTime();
publishMeasurement(endTime - startTime);
}
});
*More Info on JavaScript timers: http://ejohn.org/blog/accuracy-of-javascript-time/

35. Real User Measurement
Customer Ease of Diagnostic
Experience Setup Data
Service-Side D B A
External C B B
Real User A C C+

36. Real User Measurement 2.0:

37. Real User Measurement 1.0
Customer Ease of Diagnostic
Experience Setup Data
Service-Side D B A
External C B B
Real User A C C+

38. Real User Measurement 2.0
Customer Ease of Diagnostic
Experience Setup Data
Service-Side D B A
External C B B
Real User A C A

39. Measurement Summary
Customer Ease of Diagnostic
Experience Setup Data
Service-Side D B A
External C B B
Real User A C A
Get your customer experience metrics here

41. Cache Busting
Object Size

42. • Edge selection: connect time
• Cache server performance: FBL of hits
• Network performance: LBL of hits >100 KB
• Origin fetch: FBL of misses

43. • Edge selection: connect time
• Cache server performance: FBL of hits
• Network performance: LBL of hits >100 KB
• Origin fetch: FBL of misses

44. • Edge selection: connect time
• Cache server performance: FBL of hits
• Network performance: LBL of hits >100 KB
• Origin fetch: FBL of misses

45. • Edge selection: connect time
• Cache server performance: FBL of hits
• Network performance: LBL of hits >100 KB
• Origin fetch: FBL of misses

46. ~1 KB 100 KB+
2 to 3 Round Trips 7 to 8 Round Trips
0-1 DNS Lookup 0-1 DNS Lookup
1 TCP Connect 1 TCP Connect
1 Object Download 6 Object Download
Tests Client to Edge Latency Tests Client to Edge Latency & Network Quality

47. DNS Lookup TCP Connect Edge Cache Client Cache
Viewer-Driven Bust Prime Bust
DNS rotation example: Two-requests example: Query string example:
test1.example.com GET /test_100k.png GET /test_100k.png?ver=1
test2.example.com GET /test_100k.png GET /test_100k.png?ver=2
Prime Viewer-Driven
Two-requests example: One-request example:
test.example.com GET /test_100k.png
test.example.com
Bust
One-request w/ CDN
Config example:
GET /test_100k.png?ver=1

48. http://rum1.internetkitties.com/test_100k.png?cache_bust=16349583&rid=345-04324-034533
1. Don’t Prime Edge Cache

49. http://rum1.internetkitties.com/test_100k.png?cache_bust=16349583&rid=345-04324-034533
1. Don’t Prime Edge Cache
2. Fresh TCP Connection

50. http://rum1.internetkitties.com/test_100k.png?cache_bust=16349583&rid=345-04324-034533
1. Don’t Prime Edge Cache
2. Fresh TCP Connection
3.
100KB Object

51. http://rum1.internetkitties.com/test_100k.png?cache_bust=16349583&rid=345-04324-034533
1. Don’t Prime Edge Cache
2. Fresh TCP Connection
3.
100KB Object
4. Bust Client Cache

52. http://rum1.internetkitties.com/test_100k.png?cache_bust=16349583&rid=345-04324-034533
1. Don’t Prime Edge Cache
2. Fresh TCP Connection
3.
100KB Object
4. Bust Client Cache
5. Diagnostics via Nav Timing API

53. http://rum1.internetkitties.com/test_100k.png?cache_bust=16349583&rid=345-04324-034533
1. Don’t Prime Edge Cache
2. Fresh TCP Connection
3.
100KB Object
4. Bust Client Cache
5. Diagnostics via Nav Timing API
6. **Use a Request ID**

55. Visible
Actionable

56. Geographic Regions
Clients/Devices
CDNs
Viewer Networks

57. Geographic Regions
Clients/Devices
CDNs
Viewer Networks

58. # of requests - GLOBAL
time
Regional Partition Uncovers Hidden Insights
# of requests – PER REGION
Europe
US
Asia
time

59. Geographic Regions
Clients/Devices
CDNs
Viewer Networks

60. Geographic Regions Amazon
CloudFront
Clients/Devices
CDN A
CDNs
CDN B
Viewer Networks

61. Geographic Regions
Clients/Devices
CDNs
Viewer Networks

62. Segmentation Schemes
Geographic Regions
Clients/Devices
CDNs
Viewer Networks
Image source: The Opte Project

63. Viewer Network Segmentation
23 ms
27 ms
42
ms

64. Viewer Network Segmentation
23 ms
300 ms
42
ms

65. Metric Sorted Lists
Viewer Networks
Object

66. Metric Sorted Lists
Viewer Networks
Object
Image source: The Opte Project

67. Isolate Outliers With Sorted Lists
Network ID Last Byte Latency
Viewer Networks
Object

68. Isolate Outliers With Sorted Lists
Viewer Networks
Object

73. MapBox: Design maps in the cloud, publish in
minutes

84. 256x256 (png/jpg)

85. Maps as HTTP
Content-Type: image/jpg
Content-Length: 33049
Connection: keep-alive
Cache-Control: max-age=7200
Date: Fri, 19 Oct 2012 13:43:11 GMT
ETag: "6ff3ca335cb686dae60905b20...”
Last-Modified: Tue, 18 Sep 2012 02:20:00
GMT
Age: 4
X-Cache: Hit from cloudfront

86. “Scale” You Say?
• A popular map can do 50+ million tiles a day
• 50e6 / 864e2 = ~580 req/sec
• I have tens of thousands of maps

87. Dimensions of Scale

88. Dimensions of Scale

89. Maps as Utility
• Foursquare
• Weather (WDT, USA Today)
• Airplane pilot maps (foreflight)

90. Maps as Utility
• Our customers maps are critical elements of
their service
• No maps isn’t an option for them

91. On AWS from the Start
• Benchmarking the whole time
• For example: `c1.xlarge` is our sweet spot
• We tried everything…
• …with every configuration we could think of

92. On AWS from the Start
• Use Amazon Cloudfront as our CDN
• Got business pressure to switch...
• So, we benchmarked (Jan/Feb 2012)

93. CDN Evaluation
What do we care about?

94. CDN Evaluation
• Speed
• Per-GB cost, setup fees & minimums
• Log delivery
• Cache purging
• Support quality

95. CDN Evaluation
• Akamai
• Amazon CloudFront
• EdgeCast
• Limelight
• HighWinds

97. CDN Benchmarking
• Objects are almost exclusively images
• Very difficult to measure existing users

98. CDN Benchmarking
• External monitoring
• Page loads & js evaluation
• 64 monitoring stations
• Global distribution

99. CDN Benchmarking
• Two identical stacks
• Simultaneously
• Same map, same max-age, etc….
• 24 hours

100. CDN Benchmarking
• Spot tested with real people across the US,
Europe, and South America

101. CDN Benchmarking

102. CDN Evaluation
• Speed: Faster
• Cost: Lower
• Logs: Not FTP, timely delivery
• Cache purges: check
• Support: check

103. Ok, CDN is fast, great. All done. Right?
• No
• Do less
• Get closer

104. Do Less

105. Do Less
• If-Modified-Since
• CDN has an expired item, it checks back with
the origin
• Origin says "oh, you can keep using that“ –
Just headers

106. Do Less
• Ability to independently quickly query metadata
for resource components
• Know validity before loading full resource
• Sometimes this is dismissed as impractical,
more often worth the work

107. Do Less

108. Do Less

109. Do Less

110. Get Closer
http://www.flickr.com/photos/lightsniper

111. Get Closer
• Even electrons take time to cross an ocean
• CDN doesn't have to go as far on misses

114. Get Closer
• Two independent clusters: Ireland & Virginia
• Master: Master replication
• DNS: any-cast / latency-based / geo-based
routing

115. Get Closer
• AWS CloudFormation
• Config management (Puppet)
• Automated deploys

116. Traffic: US vs. EU

117. Get Closer

118. Did That Help?
• IMS - 10x
• Proximity - 6x

119. tl;dr
• Know what matters to your service
• Measure all the things
• Be a good upstream to your CDN

120. http://www.stevesouders.com/blog/2012/11/14/comparing-rum-synthetic-
page-load-times/

121. www.aws.amazon.com/cloudfront
www.aws.amazon.com/cloudfront
www.aws.amazon.com/cloudfront
www.aws.amazon.com/cloudfront/dynamic-content

122. #reinvent
We are sincerely eager to
hear your feedback on this
presentation and on re:Invent.
Please fill out an evaluation
form when you have a
chance.