SlideShare a Scribd company logo
1 of 44
Download to read offline
Simple Log Analysis and Trending

                     Mike Brittain
                     DIRECTOR OF ENGINEERING, ETSY
                     @mikebrittain
                     EXAMPLES: bit.ly/velocity-etsy
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_n
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.c
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no




                                                                                     A
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heart
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_quer
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_po
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
PAGE GENERATION TIME (MS)
600




450




300
                B
150




 0
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_n
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.c
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no




                                                                                     A
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heart
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_quer
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_po
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
PAGE GENERATION TIME (MS)
600




450




300
                B
150




 0
A                                                                                          B
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_state
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Cen
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5I
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_s
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.fa
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorit
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gat
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/lis
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listi
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Moz
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_l
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_communit
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
$ tail -f access.log
Plotting log data for ad hoc analysis and
           real-time trending

           EXAMPLES: bit.ly/velocity-etsy
Performance data belongs in your app logs
            (e.g. web server access log)
Access log format
Time, remote address, http method, request uri, referrer, user-agent,
response size, response code, execution time, memory consumed,
plus custom fields...

•   Signed-in/out (user_id vs. “-”)
•   display mode (“desktop” vs. “mobile”)
•   i10n/i18n (“en-US”)
•   A/B experiment buckets
apache_note(“custom_field”, $whatever);
LogFormat %l %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i" %{custom_field}n ...
apache_note(“custom_field”, $whatever);
LogFormat %l %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i" %{custom_field}n ...

LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”


web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
We have logs. Cool.
Ad hoc analysis using the newest,
hottest, bleeding-edge plotting tool.
gnuplot!
1986
# access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
# access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876

$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}'
# access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876

$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}' 
| sed "s/[//"
# access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876

$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}' 
| sed "s/[//" > response_times.dat

# response_times.dat
11/May/2011:17:08:53   505.780
11/May/2011:17:08:54   311.455
11/May/2011:17:08:53   376.745
11/May/2011:17:08:54   322.278
# response_times.dat
11/May/2011:17:08:53   505.780
11/May/2011:17:08:54   311.455
11/May/2011:17:08:53   376.745
11/May/2011:17:08:54   322.278

# scatter.gp
plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff", 
     800 t "SLA" lw 2 lc rgb "#f28705"
# scatter.gp
plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff", 
     800 t "SLA" lw 2 lc rgb "#f28705"
#!/usr/local/bin/gnuplot

set terminal pngcairo enhanced font "arial,11" size 800,600
set output 'response_times.png'

# Graph formatting
set title "Listings, generation times (ms)"
set ylabel "millisec"
set key off
set yrange [0:2000]

# Setup time formatting
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"

plot 
  'response_times.dat' using 1:2 with points lc rgb "#33ccff", 
   800 t "SLA" lw 2 lc rgb "#f28705"
# scatter.gp
plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff", 
     800 t "SLA" lw 2 lc rgb "#ff0000"




                Median? 95th Percentile?
# access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876

$ exercise_for_reader.sh

# response_median_perc95.dat
22/May/2012:15:15:00 417 620
22/May/2012:15:16:00 309 973
22/May/2012:15:17:00 340 860
22/May/2012:15:18:00 303 416
# response_median_perc95.dat
22/May/2012:15:15:00 417 620
22/May/2012:15:16:00 309 973
22/May/2012:15:17:00 340 860
22/May/2012:15:18:00 303 416

# trends.gp
plot 'response_median_perc95.dat' 
using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#548c6c" title "95th Percentile", 
"" using 1:2 with lines linewidth 2 linecolor rgb "#2192bf" title "Median", 
800 title "SLA" linewidth 2 linecolor rgb "#f28705"
# trends.gp
plot 'response_median_perc95.dat' 
using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#0060ad" title "95th Percentile", 
"" using 1:2 with lines linewidth 2 linecolor rgb "#dd181f" title "Median", 
800 title "SLA" linewidth 2 linecolor rgb "#f28705"
# access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876

$ php data_buckets.php <filename> <bucket-size>
# access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876

$ php data_buckets.php access.log 50 > response_histogram.dat

# response_histogram.dat
# bucket (ms), count
0 0
50 0
100 12
150 412
200 1203
250 2312
300 2078
#!/usr/local/bin/gnuplot

set terminal pngcairo enhanced font "arial,11"   size 800,600
set output 'histogram.png'

# Graph formatting
set title "Listings response times (ms)"
set key off
set yrange [0:*]

# Histogram rendering
set style data histograms
set style histogram clustered gap 1
set style fill solid 1.00 border lt -1
set boxwidth 0.9 absolute

plot 'response_histogram.dat' using 2:xtic(1)
# histogram.gp
plot 'response_histogram.dat' using 2:xtic(1)
Warning for Mac / homebrew users
The pngcairo terminal does not work with pango 1.30 (latest).
Use 1.28 or 1.29 instead.

set terminal pngcairo enhanced font "arial,11"           size 800,600




“Gnuplot pdfcairo problem”
https://github.com/mxcl/homebrew/issues/11901
(Near) Real-Time Trending
Logster
Run by cron, maintaining a cursor on log files
Simple parsing & aggregation
Output to Ganglia, Graphite, stdout

Combined log, via syslog
1 minute schedule
Output to Graphite                        github.com/etsy/logster
Report script execution time (ms) segmented by signed-in/out visitors?

web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176

web0047 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-
gold-filled-hook-earrings HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 10813080 377090 402038

web0031 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-
diamonds-ooak-red-on-custom HTTP/1.1" 200 13835 "http://www.etsy.com/listing/
72550205/krylon-dots-black-on-white-tank-womens-l" "Mozilla/5.0 (iPhone; U; CPU like
Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/
419.3" mobile 8513282 15249296 563978 586038
web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176


# Parse fields from log file
self.reg = re.compile(‘.* (?P<display_mode>w+) (?P<etsy_user_id>[d-]+)
(?P<php_bytes>d+) (?P<php_usec>d+) (?P<apache_usec>d+)$’)
web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176


# Collect data from each matching line
# signed_out_ms and signed_in_ms are lists.
fields = regMatch.groupdict()
php_ms = int(fields['php_usec']) / 1000
user_id = fields['user_id']

if (user_id == "-"):
    self.signed_out_ms.append(php_ms)
else:
    self.signed_in_ms.append(php_ms)
web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176


return [
  MetricObject("signed_in_reqs",        len(self.signed_in_ms), "Requests"),
  MetricObject("signed_in_ms_min",      self.signed_in_ms[0], "ms"),
  MetricObject("signed_in_ms_median",   percentile(self.signed_in_ms, 0.5), "ms"),
  MetricObject("signed_in_ms_perc95",   percentile(self.signed_in_ms, 0.95), "ms"),
  MetricObject("signed_in_ms_max",      self.signed_in_ms[-1], "ms"),
]
web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176


$ logster --output stdout PerfLogster access.log

signed_out_reqs 8
signed_out_ms_min 201
signed_out_ms_median 227.5
signed_out_ms_perc95 236.9
signed_out_ms_max 239
signed_in_reqs 3
signed_in_ms_min 258
signed_in_ms_median 268
signed_in_ms_perc95 268.0
signed_in_ms_max 268
web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176


$ logster --output stdout PerfLogster access.log

$ logster --output graphite --graphite-host graphite.example.com:2003 
    PerfLogster access.log
web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176


$ logster --output stdout PerfLogster access.log

$ logster --output graphite --graphite-host graphite.example.com:2003 
    PerfLogster access.log


# Put that into a crontab to run every minute, and...
600




450




300
      MEDIAN PAGE GENERATION TIMES (MS)
           BY SIGNED-IN/SIGNED-OUT


150




 0
Thanks!
 Code samples and slides at bit.ly/velocity-etsy
 github.com/etsy/logster




 We are hiring... etsy.com/careers
 (or just come talk to me afterward)

More Related Content

Similar to Simple Log Analysis and Trending

Google Analytics for Beginners - Training
Google Analytics for Beginners - TrainingGoogle Analytics for Beginners - Training
Google Analytics for Beginners - TrainingRuben Vezzoli
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeWim Godden
 
Premier Income Plan
Premier Income PlanPremier Income Plan
Premier Income PlanJp Symoens
 
Beyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeBeyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeWim Godden
 
Social Analytics Reporting Workshop - Dallas Digital Summit 2014
Social Analytics Reporting Workshop - Dallas Digital Summit 2014Social Analytics Reporting Workshop - Dallas Digital Summit 2014
Social Analytics Reporting Workshop - Dallas Digital Summit 2014Eric Swayne
 
Hasil sql php myadmin 3.4.11
Hasil sql   php myadmin 3.4.11Hasil sql   php myadmin 3.4.11
Hasil sql php myadmin 3.4.11Rochmat Salim
 
Foreman Single Sign-On Made Easy with Keycloak
Foreman Single Sign-On Made Easy with KeycloakForeman Single Sign-On Made Easy with Keycloak
Foreman Single Sign-On Made Easy with KeycloakNikhil Kathole
 
213140048박민경
213140048박민경213140048박민경
213140048박민경민경 박
 
Seo Mapping Tips 2022
Seo Mapping Tips 2022Seo Mapping Tips 2022
Seo Mapping Tips 2022SeoMaping
 
Smashing the stats for fun (and profit)
Smashing the stats for fun (and profit)Smashing the stats for fun (and profit)
Smashing the stats for fun (and profit)Security B-Sides
 
Cloudy Open Source and DevOps
Cloudy Open Source and DevOpsCloudy Open Source and DevOps
Cloudy Open Source and DevOpsMatt O'Keefe
 
Open Data, Visualization & Usability for Online News Delivery
Open Data,  Visualization &  Usability for  Online News DeliveryOpen Data,  Visualization &  Usability for  Online News Delivery
Open Data, Visualization & Usability for Online News DeliveryMohan Krishnan
 
Taller abstracción: Mesopotamia
Taller abstracción: MesopotamiaTaller abstracción: Mesopotamia
Taller abstracción: MesopotamiaSilvana961105
 
Taller abstracción
Taller abstracciónTaller abstracción
Taller abstracción212114
 
2分鐘快速開發跨平台iot
2分鐘快速開發跨平台iot2分鐘快速開發跨平台iot
2分鐘快速開發跨平台iotJohn 張治中
 
Sgp car performance
Sgp car performance Sgp car performance
Sgp car performance nomansyed
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeWim Godden
 
Mike lawell executionplansformeremortals_2015
Mike lawell executionplansformeremortals_2015Mike lawell executionplansformeremortals_2015
Mike lawell executionplansformeremortals_2015mlawell
 

Similar to Simple Log Analysis and Trending (20)

Google Analytics for Beginners - Training
Google Analytics for Beginners - TrainingGoogle Analytics for Beginners - Training
Google Analytics for Beginners - Training
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
 
Premier Income Plan
Premier Income PlanPremier Income Plan
Premier Income Plan
 
Presentación1
Presentación1Presentación1
Presentación1
 
Bussines
BussinesBussines
Bussines
 
Beyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeBeyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the code
 
Social Analytics Reporting Workshop - Dallas Digital Summit 2014
Social Analytics Reporting Workshop - Dallas Digital Summit 2014Social Analytics Reporting Workshop - Dallas Digital Summit 2014
Social Analytics Reporting Workshop - Dallas Digital Summit 2014
 
Hasil sql php myadmin 3.4.11
Hasil sql   php myadmin 3.4.11Hasil sql   php myadmin 3.4.11
Hasil sql php myadmin 3.4.11
 
Foreman Single Sign-On Made Easy with Keycloak
Foreman Single Sign-On Made Easy with KeycloakForeman Single Sign-On Made Easy with Keycloak
Foreman Single Sign-On Made Easy with Keycloak
 
213140048박민경
213140048박민경213140048박민경
213140048박민경
 
Seo Mapping Tips 2022
Seo Mapping Tips 2022Seo Mapping Tips 2022
Seo Mapping Tips 2022
 
Smashing the stats for fun (and profit)
Smashing the stats for fun (and profit)Smashing the stats for fun (and profit)
Smashing the stats for fun (and profit)
 
Cloudy Open Source and DevOps
Cloudy Open Source and DevOpsCloudy Open Source and DevOps
Cloudy Open Source and DevOps
 
Open Data, Visualization & Usability for Online News Delivery
Open Data,  Visualization &  Usability for  Online News DeliveryOpen Data,  Visualization &  Usability for  Online News Delivery
Open Data, Visualization & Usability for Online News Delivery
 
Taller abstracción: Mesopotamia
Taller abstracción: MesopotamiaTaller abstracción: Mesopotamia
Taller abstracción: Mesopotamia
 
Taller abstracción
Taller abstracciónTaller abstracción
Taller abstracción
 
2分鐘快速開發跨平台iot
2分鐘快速開發跨平台iot2分鐘快速開發跨平台iot
2分鐘快速開發跨平台iot
 
Sgp car performance
Sgp car performance Sgp car performance
Sgp car performance
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
 
Mike lawell executionplansformeremortals_2015
Mike lawell executionplansformeremortals_2015Mike lawell executionplansformeremortals_2015
Mike lawell executionplansformeremortals_2015
 

More from Mike Brittain

From Building a Marketplace to Building Teams
From Building a Marketplace to Building TeamsFrom Building a Marketplace to Building Teams
From Building a Marketplace to Building TeamsMike Brittain
 
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCContinuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCMike Brittain
 
Distributed Release Management
Distributed Release ManagementDistributed Release Management
Distributed Release ManagementMike Brittain
 
Advanced Topics in Continuous Deployment
Advanced Topics in Continuous DeploymentAdvanced Topics in Continuous Deployment
Advanced Topics in Continuous DeploymentMike Brittain
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyMike Brittain
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsMike Brittain
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsMike Brittain
 
On Failure and Resilience
On Failure and ResilienceOn Failure and Resilience
On Failure and ResilienceMike Brittain
 
Web Performance Culture and Tools at Etsy
Web Performance Culture and Tools at EtsyWeb Performance Culture and Tools at Etsy
Web Performance Culture and Tools at EtsyMike Brittain
 
Take My Logs. Please!
Take My Logs. Please!Take My Logs. Please!
Take My Logs. Please!Mike Brittain
 
Metrics-Driven Engineering
Metrics-Driven EngineeringMetrics-Driven Engineering
Metrics-Driven EngineeringMike Brittain
 
Web Performance Culture and Tools at Etsy
Web Performance Culture and Tools at EtsyWeb Performance Culture and Tools at Etsy
Web Performance Culture and Tools at EtsyMike Brittain
 
Metrics-Driven Engineering at Etsy
Metrics-Driven Engineering at EtsyMetrics-Driven Engineering at Etsy
Metrics-Driven Engineering at EtsyMike Brittain
 
How to Get to Second Base with Your CDN
How to Get to Second Base with Your CDNHow to Get to Second Base with Your CDN
How to Get to Second Base with Your CDNMike Brittain
 

More from Mike Brittain (14)

From Building a Marketplace to Building Teams
From Building a Marketplace to Building TeamsFrom Building a Marketplace to Building Teams
From Building a Marketplace to Building Teams
 
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCContinuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYC
 
Distributed Release Management
Distributed Release ManagementDistributed Release Management
Distributed Release Management
 
Advanced Topics in Continuous Deployment
Advanced Topics in Continuous DeploymentAdvanced Topics in Continuous Deployment
Advanced Topics in Continuous Deployment
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty Details
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty Details
 
On Failure and Resilience
On Failure and ResilienceOn Failure and Resilience
On Failure and Resilience
 
Web Performance Culture and Tools at Etsy
Web Performance Culture and Tools at EtsyWeb Performance Culture and Tools at Etsy
Web Performance Culture and Tools at Etsy
 
Take My Logs. Please!
Take My Logs. Please!Take My Logs. Please!
Take My Logs. Please!
 
Metrics-Driven Engineering
Metrics-Driven EngineeringMetrics-Driven Engineering
Metrics-Driven Engineering
 
Web Performance Culture and Tools at Etsy
Web Performance Culture and Tools at EtsyWeb Performance Culture and Tools at Etsy
Web Performance Culture and Tools at Etsy
 
Metrics-Driven Engineering at Etsy
Metrics-Driven Engineering at EtsyMetrics-Driven Engineering at Etsy
Metrics-Driven Engineering at Etsy
 
How to Get to Second Base with Your CDN
How to Get to Second Base with Your CDNHow to Get to Second Base with Your CDN
How to Get to Second Base with Your CDN
 

Recently uploaded

Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700
 

Recently uploaded (20)

Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
 

Simple Log Analysis and Trending

  • 1. Simple Log Analysis and Trending Mike Brittain DIRECTOR OF ENGINEERING, ETSY @mikebrittain EXAMPLES: bit.ly/velocity-etsy
  • 2. sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel %20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; . 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1| ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0| ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356 web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork? section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6 - LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO 412157 437768 web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and +green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin 1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754 web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha %2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_n ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1| ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038 web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.c externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no A ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176 web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725 dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www. ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heart ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152 586038 web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet? ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0 Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_quer ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_po ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669 web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url? sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel %20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; . 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1| ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0| ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356 web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork? section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6 - LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO 412157 437768 web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and +green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
  • 3. PAGE GENERATION TIME (MS) 600 450 300 B 150 0
  • 4. sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel %20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; . 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1| ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0| ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356 web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork? section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6 - LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO 412157 437768 web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and +green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin 1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754 web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha %2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_n ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1| ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038 web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.c externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no A ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176 web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725 dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www. ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heart ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152 586038 web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet? ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0 Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_quer ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_po ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669 web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url? sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel %20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; . 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1| ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0| ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356 web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork? section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6 - LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO 412157 437768 web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and +green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
  • 5. PAGE GENERATION TIME (MS) 600 450 300 B 150 0
  • 6. A B web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url? sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel %20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0. 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_state ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang= ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356 web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Cen - LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0 ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5I 412157 437768 web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and +green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5 AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_s ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_ 1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754 web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_ %2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy. ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1| ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038 web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.fa externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorit ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gat Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176 web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/lis dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listi ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 586038 web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet? ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Moz Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_l ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_communit ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669 web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
  • 7. $ tail -f access.log
  • 8. Plotting log data for ad hoc analysis and real-time trending EXAMPLES: bit.ly/velocity-etsy
  • 9. Performance data belongs in your app logs (e.g. web server access log)
  • 10. Access log format Time, remote address, http method, request uri, referrer, user-agent, response size, response code, execution time, memory consumed, plus custom fields... • Signed-in/out (user_id vs. “-”) • display mode (“desktop” vs. “mobile”) • i10n/i18n (“en-US”) • A/B experiment buckets
  • 11. apache_note(“custom_field”, $whatever); LogFormat %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{custom_field}n ...
  • 12. apache_note(“custom_field”, $whatever); LogFormat %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{custom_field}n ... LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
  • 13. LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D” web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/ 12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)" desktop - 13399576 505780 554876
  • 14. We have logs. Cool.
  • 15. Ad hoc analysis using the newest, hottest, bleeding-edge plotting tool.
  • 17. 1986
  • 18. # access.log web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/ 12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)" desktop - 13399576 505780 554876
  • 19. # access.log web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/ 12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)" desktop - 13399576 505780 554876 $ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}'
  • 20. # access.log web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/ 12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)" desktop - 13399576 505780 554876 $ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}' | sed "s/[//"
  • 21. # access.log web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/ 12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)" desktop - 13399576 505780 554876 $ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}' | sed "s/[//" > response_times.dat # response_times.dat 11/May/2011:17:08:53 505.780 11/May/2011:17:08:54 311.455 11/May/2011:17:08:53 376.745 11/May/2011:17:08:54 322.278
  • 22. # response_times.dat 11/May/2011:17:08:53 505.780 11/May/2011:17:08:54 311.455 11/May/2011:17:08:53 376.745 11/May/2011:17:08:54 322.278 # scatter.gp plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"
  • 23. # scatter.gp plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"
  • 24. #!/usr/local/bin/gnuplot set terminal pngcairo enhanced font "arial,11" size 800,600 set output 'response_times.png' # Graph formatting set title "Listings, generation times (ms)" set ylabel "millisec" set key off set yrange [0:2000] # Setup time formatting set xdata time set timefmt "%d/%B/%Y:%H:%M:%S" set format x "%H:%M" plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"
  • 25. # scatter.gp plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#ff0000" Median? 95th Percentile?
  • 26. # access.log web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/ 12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)" desktop - 13399576 505780 554876 $ exercise_for_reader.sh # response_median_perc95.dat 22/May/2012:15:15:00 417 620 22/May/2012:15:16:00 309 973 22/May/2012:15:17:00 340 860 22/May/2012:15:18:00 303 416
  • 27. # response_median_perc95.dat 22/May/2012:15:15:00 417 620 22/May/2012:15:16:00 309 973 22/May/2012:15:17:00 340 860 22/May/2012:15:18:00 303 416 # trends.gp plot 'response_median_perc95.dat' using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#548c6c" title "95th Percentile", "" using 1:2 with lines linewidth 2 linecolor rgb "#2192bf" title "Median", 800 title "SLA" linewidth 2 linecolor rgb "#f28705"
  • 28. # trends.gp plot 'response_median_perc95.dat' using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#0060ad" title "95th Percentile", "" using 1:2 with lines linewidth 2 linecolor rgb "#dd181f" title "Median", 800 title "SLA" linewidth 2 linecolor rgb "#f28705"
  • 29. # access.log web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/ 12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)" desktop - 13399576 505780 554876 $ php data_buckets.php <filename> <bucket-size>
  • 30. # access.log web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/ 12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)" desktop - 13399576 505780 554876 $ php data_buckets.php access.log 50 > response_histogram.dat # response_histogram.dat # bucket (ms), count 0 0 50 0 100 12 150 412 200 1203 250 2312 300 2078
  • 31. #!/usr/local/bin/gnuplot set terminal pngcairo enhanced font "arial,11" size 800,600 set output 'histogram.png' # Graph formatting set title "Listings response times (ms)" set key off set yrange [0:*] # Histogram rendering set style data histograms set style histogram clustered gap 1 set style fill solid 1.00 border lt -1 set boxwidth 0.9 absolute plot 'response_histogram.dat' using 2:xtic(1)
  • 33. Warning for Mac / homebrew users The pngcairo terminal does not work with pango 1.30 (latest). Use 1.28 or 1.29 instead. set terminal pngcairo enhanced font "arial,11" size 800,600 “Gnuplot pdfcairo problem” https://github.com/mxcl/homebrew/issues/11901
  • 35. Logster Run by cron, maintaining a cursor on log files Simple parsing & aggregation Output to Ganglia, Graphite, stdout Combined log, via syslog 1 minute schedule Output to Graphite github.com/etsy/logster
  • 36. Report script execution time (ms) segmented by signed-in/out visitors? web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling- silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http:// www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176 web0047 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k- gold-filled-hook-earrings HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 10813080 377090 402038 web0031 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and- diamonds-ooak-red-on-custom HTTP/1.1" 200 13835 "http://www.etsy.com/listing/ 72550205/krylon-dots-black-on-white-tank-womens-l" "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/ 419.3" mobile 8513282 15249296 563978 586038
  • 37. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling- silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http:// www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176 # Parse fields from log file self.reg = re.compile(‘.* (?P<display_mode>w+) (?P<etsy_user_id>[d-]+) (?P<php_bytes>d+) (?P<php_usec>d+) (?P<apache_usec>d+)$’)
  • 38. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling- silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http:// www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176 # Collect data from each matching line # signed_out_ms and signed_in_ms are lists. fields = regMatch.groupdict() php_ms = int(fields['php_usec']) / 1000 user_id = fields['user_id'] if (user_id == "-"): self.signed_out_ms.append(php_ms) else: self.signed_in_ms.append(php_ms)
  • 39. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling- silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http:// www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176 return [ MetricObject("signed_in_reqs", len(self.signed_in_ms), "Requests"), MetricObject("signed_in_ms_min", self.signed_in_ms[0], "ms"), MetricObject("signed_in_ms_median", percentile(self.signed_in_ms, 0.5), "ms"), MetricObject("signed_in_ms_perc95", percentile(self.signed_in_ms, 0.95), "ms"), MetricObject("signed_in_ms_max", self.signed_in_ms[-1], "ms"), ]
  • 40. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling- silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http:// www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176 $ logster --output stdout PerfLogster access.log signed_out_reqs 8 signed_out_ms_min 201 signed_out_ms_median 227.5 signed_out_ms_perc95 236.9 signed_out_ms_max 239 signed_in_reqs 3 signed_in_ms_min 258 signed_in_ms_median 268 signed_in_ms_perc95 268.0 signed_in_ms_max 268
  • 41. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling- silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http:// www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176 $ logster --output stdout PerfLogster access.log $ logster --output graphite --graphite-host graphite.example.com:2003 PerfLogster access.log
  • 42. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling- silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http:// www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176 $ logster --output stdout PerfLogster access.log $ logster --output graphite --graphite-host graphite.example.com:2003 PerfLogster access.log # Put that into a crontab to run every minute, and...
  • 43. 600 450 300 MEDIAN PAGE GENERATION TIMES (MS) BY SIGNED-IN/SIGNED-OUT 150 0
  • 44. Thanks! Code samples and slides at bit.ly/velocity-etsy github.com/etsy/logster We are hiring... etsy.com/careers (or just come talk to me afterward)