eBay Search Science: Leveraging Behavioral Data Analysis for Effective Query Reformulation
Brian will talk about combing through behavioral log files with Scala on Hadoop in order to generate large data sets used to drive dynamic, online query rewrites at eBay. He’ll cover the product/feature pipeline from ideation to data mining, prototyping, statistical analysis, offline side by side analysis, human judgment, online experimentation, and finally launch.
1. Welcome To
Director of Engineering
Search Science Recall & Spam
April 3, 2015
BRIAN JOHNSON
With more than 100 million active users globally, eBay is
the world's largest online marketplace, where practically
anyone can buy and sell practically anything. Founded in
1995, eBay connects a diverse and passionate community
of individual buyers and sellers, as well as small
businesses. Their collective impact on ecommerce is
staggering: In 2014, the total value of goods sold on eBay
was $82 billion -- more than $2,500 every second.
13. What is Special for eBay Search?
•Commercial Intentions
– Both the sellers and buyers have strong and clear intention
– Transactions happen on eBay, hence more behavior data
•Listings (Supply Side)
– Given by sellers
– Semi-structured data
•Buyers (Demand Side)
– Relevance matters (Browser vs. Searching)
– Price matters
– Seller trust/credit matters
– 70% eBay revenue starts with Search
14. Fish Sticks
Demand Category
42%
Business & Industrial > Electrical & Test Equipment > Electrical Equipment & Tools >
Electrical Tools > Cable Pullers
10% Business & Industrial > Construction > Building Materials & Supplies > Electrical
9% Home & Garden > Tools > Other
7% Pet Supplies > Fish & Aquariums > Fish Pond Supplies
6% Business & Industrial > Light Equipment & Tools > Air Tools > Staplers
19. Context & Specificity
Context
ATC Armored Troop Carrier in Toys and Hobbies
ATC Artist trading card in ART
ATC Automatic Tool Change in Business and Industrial
Specificity/Directionality
Old Antique
Yoga towels/mats Yogitoes
22. German Compounds
•Syntactically, words can be combined and split in many ways
•Multiple candidates
Granitpflastersteine (granite paving stones)
Granit(granite) pflastersteine(cobblestones)
Granit(granite) pflaster(paving/band-aid) steine(stones)
•Binding characters
Hochzeitsschuhe (grammatically correct, 593 hits on ebay.de)
Hochzeitschuhe (129 hits on ebay.de)
•Some words shouldn’t be de-compounded.
beiden (both) – bei(at) den(the)
24. Hadoop Graph/Session Analysis
Bipartite Graphs
Keyword | Keyword Synonym Expansion
Keyword | Attribute Aspect Expansions
Keyword | Category Category Expansions, Related Search Diversity
Query | Query Related Search
Query | Item Related Search
Query Session Analysis
Successive Queries Synonyms, Related Search
Query Substitutions Synonyms
Same Session Correlation Related Search
Query Metrics
Click Through Rate
Purchase Attribution
Time to First Click, View Item, Purchase
Query Pair Price & Category Divergence
Query Pair Result Set Overlap
Result Count
25. Why we’re excited about data mining…
•We’re at an inflection point – customers are defining how they shop
– We are a data company
– 40+ Pb of data (listings, pictures, queries, clicks, sales, feedback, …)
– Many tests running orthogonally (in parallel with overlapping user slices)
– Nearly all users in one of more tests
– Many users per test, often millions
•Find patterns and insights drives our customer experience
•We’ve built successful teams of data scientists
27. METRICS
•What should we optimize
–Page Views
–Time on Site
–Click Through Rate
–Normalized Discounted Cumulative Gain
–Purchases per User per Session/Day/Week
–Revenue per User per Session/Day/Week
–Net Promoter Score
•How likely would you be to recommend …?
33. Query Rewrites at eBay
Query Rewrite
Search
User Query
eBay Results
Search Query
User Query: pilzlampe {mushroom lamp}
Search Query: OR(pilzlampe, PHRASE(OR(pilz,pilze),OR(lampe,lampen)))
34. Example Query Services/Rewrites
• Stemming (ipod OR ipods)
• Spelling (cannon OR canon)
• Condition (new OR condition=new)
• Synonyms (boat carpet OR marine carpet)
• Space Synonyms (MarioKart OR Mario-Kart)
• Item Specifics (blue OR color=blue)
• Acronyms (hp OR hewlett-packard OR horsepower)
• Category (shoes OR Category=Shoes)
• Cross Border (site=0 AND category =123) OR (site=3 AND
category=456)
• Fitment (fits model=corolla)
• Term Removal (Harry Potter and the Order of the Phoenix (daily deal))
35. Acronym/Abbreviation Mining
•Acronyms/Abbreviations mined from raw text and query logs
•Look for patterns of text:
long form (short form)
short form (long form)
• Employ intelligent matching algorithms to mine candidates
• Schwartz et al: Greedy Match Algorithm
new cheap Playstation portable (PSP)
PlayStation 3 (PS3)
• Acronym discovered
PSP => PlayStation Portable
PS3 => PlayStation 3
• Candidates mined are fed to an ML classifier to remove false positives
Editor's Notes
Why
What
How
http://www.ebayinc.com/who
You are in business to make money
How do you know if changes you make, make money
You HAVE to test
You can’t manage what you don’t measure
Testing is crucial
Image http://www.wallpapertimes.com/files/q/Yf/4j/qYf4jp9q86379020_800x600.jpg
Documents not enough anymore
Need behavioral data – Yandex beating Google in Russia, why, they have users, refrigerators in Moscow vs. isolated small town
(It was very hard to find a good example of this that brought in obviously wrong data above the fold: these issues are generally more subtle, showing up in deterministic sorts and in slower processing time. If you come up with another good example to include, that would be great.)
There are many entity names, including many brands, which are identical to (Cowboys) or share components with (e.g. Red Bull) common terms that describe our inventory. By identifying entities and by using whole query context, we can provide expansions only when appropriate (e.g. no Redder Bull or Crimson Bull). We can also decide the confidence of an expansion compared to the original (e.g. as is usually done in spell check).
For the cowboy(s) hats, Cowboys seems to mainly refer to the football team; there are a few cowboy hats where someone used “cowboys” instead of the possessive, but not many. For the toys, the plural form is definitely more common but the singular is also used in titles even in sets (bottom row of pictures has the singular; top row the plural); so, we want to use both forms to get the maximum inventory for this.
Detail matters
Context is important
"beef labeling regulation & delegation of supervision law” - long word
How do we do this
Simple counting – that’s it, you “just” have to count
Image http://www.csie.ntnu.edu.tw/~u91029/Matching.html
We did some great query rewrite work for UK/DE in 2011. Germans love compound words and the changes we made in 2011 to support them really paid off. We’ve continued to ramp up our machine learning, data mining, and natural language processing efforts to make sure that eBay search delivers again in 2012.