A Taxonomist, a Software Engineer, and a UX Researcher Walk Into a Bar: Bridging AI and User Experience Methods at Etsy
•
12 likes•3,253 views
video available here: https://blueprintdigital.com/ia-summit-2017/jenny-benevento-giovanni-fernandez-kincade-jill-fruchter/ This was a talk given at IA Summit 2017 in Vancouver, BC by Jenny Benevento, Gio Fernandez-Kincade, and Jill Fruchter. Etsy is a marketplace where people around the world connect, both online and offline, to make, sell and buy unique goods. Etsy is also a tech company that invests in the craft of coding and data-driven product development as a strategic priority. Etsy has employed AI and machine learning to tackle personalization, recommendations, image understanding, item similarity, search relevance, spelling correction, and many other tasks. We’ll talk through several examples of how Etsy leverages data, where it’s excelled, and where this hammer hasn’t quite hit the nail on the head. We will be asking ourselves hard questions, recognizing the limitations of decisions driven purely by big data: - Who are we satisfying? Our customers or our mathematical models? - Are those models even an accurate reflection of the outcomes we want? - In a dual marketplace, where complex changes depend on interactions between both sides of the market, can one metric or measure of success tell the full story? - How do we consider the impact our models are having on our users? - Are we even addressing real human needs and motivations in the first place? - How do we inform and enrich AI with expert created & applied taxonomy & metadata?
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
Similar to A Taxonomist, a Software Engineer, and a UX Researcher Walk Into a Bar: Bridging AI and User Experience Methods at Etsy
Similar to A Taxonomist, a Software Engineer, and a UX Researcher Walk Into a Bar: Bridging AI and User Experience Methods at Etsy (13)
More from Jenny Benevento
Recently uploaded
Recently uploaded (20)
A Taxonomist, a Software Engineer, and a UX Researcher Walk Into a Bar: Bridging AI and User Experience Methods at Etsy
- 1. Bridging AI and UX Methods at Etsy _________ IA Summit March 25, 2017
- 3. Etsy is a global marketplace where people around the world connect, both online and offline, to make, sell and buy unique goods. 3
- 4. By the Numbers 1.7M active sellers AS OF DECEMBER 31, 2016 28M active buyers AS OF DECEMBER 31, 2016 $2.84B annual GMS IN 2016 45+M items for sale AS OF DECEMBER 31, 2016 Photo by Kirsty-Lyn Jameson
- 6. SELLER Pursues craft, grows business ETSY Invests in the platform and delivers a global base of buyers ETSY Facilitates the transaction BUYER Finds unique goods that are hard to find elsewhere ETSY EMPOWERMENT LOOP ETSY EMPOWERMENT LOOP 6
- 7. SELLER Pursues craft, grows business ETSY Invests in the platform and delivers a global base of buyers ETSY Facilitates the transaction BUYER Finds unique goods that are hard to find elsewhere ETSY EMPOWERMENT LOOP ETSY EMPOWERMENT LOOP 7 SEARCH
- 10. Frame the problem “E-Search” - A case study WINTER 2015 Iterative evaluation Concept testing SPRING 2014 SUMMER 2014
- 11. “E-Search” - A Case Study
- 12. Parallel streams of effort UX RESEARCH
- 13. UX Research 13
- 14. Etsy.com What are people shopping for on Etsy?
- 17. People shop for ideas on Etsy 17 GIFTS LIFE EVENTS SEASONAL CELEBRATIONS STYLE/EMOTION POP CULTURE/ TRENDS NOT “known item” searches
- 18. People shop for ideas on Etsy 18 GIFTS LIFE EVENTS STYLE/EMOTION NOT “known item” searches Often, using broad, thematic search terms SEASONAL CELEBRATIONS POP CULTURE/ TRENDS
- 20. Existing “browse”pages were low performers, i.e. not converting. PROBLEM HOW MIGHT WE… Make finding on Etsy easier with a combination of search & browse improvements? 20
- 21. Customers are doing broad searches and getting “overwhelmed” with search results. PROBLEM HOW MIGHT WE… Turn broad search queries into satisfying experiences of finding and discovery? 21
- 22. UX research as the new kid in town FEB 2014 MARCH 2014 MARCH 2015 JUNE 2015 JULY 2014 AUGUST 2014 ● Etsy was founded in 2005 ● No UX Research until 2014 (Thick data) ● Since 2010, data-driven design and evaluation (Big data) RESEARCH BRAND DESIGN PRODUCT DESIGN
- 23. Etsy almost solely relied on AI and experimentation to frame and solve problems. PROBLEM HOW MIGHT WE… Use this new research capacity to help frame the problem and build a better product? 23
- 24. Jewelry 8M items
- 26. GOAL: Observe “jewelry” search behavior on- and off- Etsy to introduce product teams to shoppers’ expectations and needs TEST STIMULI: - The Internet - Self-guided searches Contextual observation (2x)
- 28. Goals 1 2 3 UX Insights Searching on Etsy is conceptually different
- 29. Goals 1 2 3 Highly visual UX Insights Searching on Etsy is conceptually different “I’m not on track” “I’m looking for something to catch my eye”
- 30. Goals 1 2 3 Highly visual Use search to browse UX Insights Searching on Etsy is conceptually different “I’m not on track” “I’m looking for something to catch my eye” “6PM is like running into the drugstore. On Etsy, I’m always finding surprises.” “It’s like watching Reality TV. I like the process.”
- 31. Goals 1 2 3 Highly visual Use search to browse UX Insights Searching on Etsy is conceptually different “I’m not on track” “I’m looking for something to catch my eye” “6PM is like running into the drugstore. On Etsy, I’m always finding surprises.” “It’s like watching Reality TV. I like the process.” FOMO “Even if it were $10,000. I want to know about it.” “Patience on Etsy is rewarded.”
- 33. E-Search SINGLE CATEGORY: JEWELRY > NECKLACE CROSS-CATEGORY: MID-CENTURY MODERN, MOTHER’S DAY Concept testing (2x)
- 34. Goals 1 2 3 UX Insights Until items feel more similar than different, the more categorization the better
- 35. Goals 1 2 3 Less FOMO UX Insights "It lets me see how it's organized and go on. It guides you." "It's nice the categories are right here. It prompts me to narrow down an extremely broad search." Until items feel more similar than different, the more categorization the better
- 36. Goals 1 2 3 Less FOMO UX Insights "It lets me see how it's organized and go on. It guides you." "It's nice the categories are right here. It prompts me to narrow down an extremely broad search." Until items feel more similar than different, the more categorization the better More intuitive “I’m making more progress with minimal effort.”
- 37. Goals 1 2 3 Less FOMO UX Insights "It lets me see how it's organized and go on. It guides you." "It's nice the categories are right here. It prompts me to narrow down an extremely broad search." Until items feel more similar than different, the more categorization the better More intuitive “I’m making more progress with minimal effort.” More “Etsy” “Expands for me what is on Etsy because I haven't thought of that being there.”
- 38. UX Vision Find more, search less
- 39. Ye Olde Librarian Knowledge
- 41. Berrypicking “the query is satisfied...by a series of selections of individual references and bits of information at each stage of the ever-modifying search...retrieval of this sort is here called berrypicking.” Source: Marcia Bates (1989). "The Design of Browsing and Berrypicking Techniques for the Online Search Interface."
- 42. UX Vision Question: Can we build that?
- 44. Jewelry
- 45. “silver necklace” AI Jewelry! Query Classification
- 48. Jewelry > Necklaces > Charm Necklaces Document Classification
- 49. 49
- 50. Disaster 50
- 51. Goals 1 2 3 Data is Noisy Disaster
- 52. AI Data is Noisy
- 53. AI Data is Noisy
- 56. 56 “hose”
- 57. 57 “hose”
- 58. 58 “chair”
- 59. 59 “chair”
- 60. Goals 1 2 3 Data is Noisy Disaster Expensive False Positives
- 61. “sunglasses”
- 62. “shoes”
- 63. Goals 1 2 3 Data is Noisy Disaster Expensive False Positives Sellers Last
- 64. Sellers last 64
- 66. Reboot GOOD DATA USER AGENCY
- 67. Reboot TAXONOMY
- 69. Reboot 69
- 70. A taxonomy to serve both user groups 70
- 71. TAXONOMY MERCH MARKETING POLICY ENG MAKERS LEGAL DATAUX
- 72. Hat?
- 77. ● Handmade > Clothing > Dresses ● Handmade > Women > Clothing > Dresses ● Handmade>Weddings>Dresses ● Vintage > Clothing > Dresses ● Vintage > Women > Clothing > Dresses ● Weddings > Dresses ● Women > Clothing > Dresses
- 78. title Knitting Technique Glass Material Geekery Subculture Vintage Age Children Who Holidays When Old Categories Aspect
- 79. Seller Control is Good UX
- 80. Not One. Two Taxonomies
- 81. Seller Control
- 83. Sellers Buyers
- 85. • Transparent • Art not SEO • Translatable • Flexible • Evolves • Buyer Friendly • Powers Structured Data 85 The Power of Two Taxonomies
- 86. AI + Inventory Data Taxonomy AI + Search
- 87. Epilogue 87
- 89. Takeaways 89
- 91. AI is not enough. 91 TAKEAWAYS
- 92. Holistic solutions require multi-disciplinary thinking. 92 TAKEAWAYS
- 93. Respect and trust your users. 93 TAKEAWAYS
- 94. Thanks!