As an ecommerce site with more than 800,000 different sellers, Etsy is particularly interested in understanding how shoppers find the items they seek. Part of this understanding involves attributing successful purchases to specific features on the site. This attribution model allows us to compare and refine Etsy’s features, but also provides valuable signals for A/B testing, search quality, and recommenders. However, the path to a successful handmade purchase often involves multiple features over the course of several visits.
This talk will discuss the challenges of funnel analysis at Etsy and the corresponding deficiencies of several widely used web analytics tools. We’ll then dive into our event sequence matching tool, which we’ve successfully applied to hundreds of millions of visits in a single Hadoop job and is widely used across our big data stack. Finally, we’ll take a look at some of our applications of the tool and compare it to related work.
Funnels ++ • Funnels are more than just an optimization tool • Use them to understand different pathways throughout our site • Partition and compare these pathways by attributes • A/B tests, categories, queries, cohorts • Attribution modelTuesday, March 12, 13
Attribution Models • Tie conversions and successes to speciﬁc products and actions • Use to gain understanding of our users’ interaction with Etsy • Help us to measure gains in A/B testing • Easily compare different varieties of the same product • Attribution techniques for internal and external attributionTuesday, March 12, 13
Segmenting Within FunnelsOld Algorithm Search Clicked Listing Purchased Counts 50,000 20,000 15,000 New Hotness Search Clicked Listing Purchased Counts 100,000 60,000 15,000Tuesday, March 12, 13
Segmenting Within FunnelsOld Algorithm Search Clicked Listing Purchased Step 100% 50% 40% To t a l 100% 50% 20% New Hotness Search Clicked Listing Purchased Step 100% 60% 25% To t a l 100% 60% 15%Tuesday, March 12, 13
Segmenting Across Funnels * Clicked Listing Purchased Search 100% 50% 40% Browse 100% 40% 30% Home 100% 60% 36% Activity Feed 100% 62% 28% Ta s t e Te s t 100% 47% 31% Search Ads 100% 45% 38%Tuesday, March 12, 13
Democratized Funnels How do we make this awesomeness available for everyone?Tuesday, March 12, 13
• Funnels had to be setup ahead of time (no backﬁlls) • Reconciliation is hard • Limited to events in our web clickstream (ios/android would be excluded) • Scaling and Operational issues • Difﬁcult to maintain multiple stacksTuesday, March 12, 13
Turns out that we don’t make Product decisions in real time http://mcfunley.com/whom-the-gods-would-destroy-they-ﬁrst-give-real-time-analyticsTuesday, March 12, 13
Getting back on the elephant • Able to carry over the User Interface from v1 • Standardized event sessionization • Operationally supported infrastructure • Nightly batch processTuesday, March 12, 13
Built in metrics • Click through rate • Listing impressions • Purchase rate - • Purchase from shop rate • Favorite from shop rate • More...Tuesday, March 12, 13
Built in metrics • Click through rate - Direct Attribution • Listing impressions - Direct Attribution • Purchase rate - Direct Attribution • Purchase from shop rate - Indirect Attribution • Favorite from shop rate - Indirect Attribution • More... -Tuesday, March 12, 13
Segmentation • Primarily used for A/B test analysis • Arbitrary segmentationTuesday, March 12, 13
How do we get it? select (clicks/visits * 100.0) as "CTR" from feature_funnel where event_type = search and ab_test = sitewide and epoch_s = 1361318400 and group_name = ALL_GROUPINGS;Tuesday, March 12, 13