Slides from sharing at Strata + Hadoop Singapore 2016 (http://conferences.oreilly.com/strata/hadoop-big-data-sg/public/schedule/detail/54542) Ecommerce has enabled retailers to make all of their products available to consumers and consumers to access niche products not found in brick-and-mortar stores. This growth provides consumers with unparalleled choice. Nonetheless, the sheer number of products brings with it the challenge of helping users find relevant products with ease. Lazada has tens of millions of products on its platform, and this number grows by approximately one million monthly. Lazada’s challenge: How can we help users easily discover good quality products they will like? How can we ensure product selection remains fresh and constantly updated? One way to do this is through the ranking of products. Via ranking, Lazada helps customers easily find products that will delight them by ensuring these products appear in the first few pages. I’ll share how Lazada ranks products on our website. (Note: Google “how amazon ranks products” for some industry background) Topics include how we: * Develop methodology (and tricks) to solve not-so-well-defined problems * Collect and store user-behavior data from our website and app * Clean and prepare the data (e.g., handling outliers) * Discover and create features useful features * Build models to improve customer experience and meet business objectives * Measure and test outcomes on our website * Built this end-to-end on our Hadoop infrastructure, with tools including Kafka and Spark

1. How Lazada
ranks products
to improve customer experience and conversion
Strata Hadoop Singapore 2016

2. How Lazada
ranks products
to improve customer experience and conversion
Strata Hadoop Singapore 2016

3. Leading
e-commerce
platform in
South-East
Asia

4. Lazada Data Science
Data App Devs expose, integrate, platform-ize
Data Scientists explore, prepare, model
Data Engineers collect, store, maintain
Start from bottom up

5. Ranking
affects what
appears
on top

6. Ranking is
different
from recom-
mendation

7. “How can I
rank well on an
e-commerce
platform?”

8. “How can I
rank well on an
e-commerce
platform?”

9. “How can I
rank well on an
e-commerce
platform?”

10. “How can I
rank well on an
e-commerce
platform?”

11. “How can I
rank well on an
e-commerce
platform?”

12. “How can I
rank well on an
e-commerce
platform?”

13. “How can I
rank well on an
e-commerce
platform?”

14. “How can I
rank well on an
e-commerce
platform?”

16. Ranking products for
catalog and search
Introducing
new products
Emphasizing
product quality

17. Web Tracker
(JavaScript)
Mobile Tracker
(Adjust)
3rd
Party
(e.g. ,ZenDesk,
SurveyGizmo)
Kafka Queues
Bulk Loaders
(Spark)
Hadoop
Hadoop
Data
Exploration
+
Data
Preparation
+
Feature
Engineering
+
Modelling
(Spark)
Manual
Boosting
(Django)
Local
Validation
A/B
Testing
Product
Seller
Transaction
Product rankings
Split traffic and measure outcomes
(Category Managers)
(User devices)

18. Overall results
Better ranking improved conversion and revenue per session
Introducing new products improved new product engagement
Emphasizing product quality had neutral to positive outcomes

19. Ranking products for
catalog and search

20. Intent
Provide shoppers quick access to best products
in catalog/search results, making shopping easy

21. Problem
Lazada has millions of products—not easy to navigate
How to identify products that interest users in the future?
How do we measure interest?

22. Methodology
Measure shoppers’ interest through product
engagement as a proxy
Clicks, add-to-cart, checkouts, etc.
Predict future interest

23. Collecting
behavioral data
Track and collect events on web (JavaScript)
and app (Adjust)
Stream and process via Kafka
Store in Hive tables

24. Data preparation
Filter and categorize online behavioral events
(e.g., impressions, clicks, etc.)
Merge various views of product data (e.g. price, stock, etc.)
Exclude outliers and potentially fraudulent events

25. Feature engineering
Calculate product engagement metrics
(e.g., average clicks, conversion rate, etc.)
Derive product attributes (e.g., age, discount, etc.)
Exclude outliers (e.g., conversion rate > 1.00)

26. Modelling
(i.e., machine learning)
Predict future (tomorrow’s) product clicks/checkouts
Examine results against a benchmark model
Pandas + XGBoost is faster and more effective than
Spark + MLlib; assessing XGBoost4J-Spark

27. Boosting products
(manually)
Manually increase rank of certain products
(e.g., highly anticipated products, campaign tie-ups)
User-friendly interface to drag-and-drop products
Limits on how many products can be boosted

28. Validation and
A/B testing
Local validation is easy, but difficult to ensure
similar results via A/B testing
A/B test all updates before production

29. Results
Increased conversion rate by 3 – 8%
Increased revenue per session by 5 – 20%

30. Introducing
new products

31. Intent
Provide potentially good new products with exposure
Provide shoppers with new products they like
Keep catalog fresh

32. Problem
Products with strong engagement stay on top
Products without engagement don’t get traffic
How can we identify new products that are likely
to interest users?

33. Methodology (demand)
Find what people need
Measure needs through internal/external data
Rank new products in terms of demand

34. Methodology (supply)
Find products similar to top products
Measure similarity with top products
Rank new products based on similarity and top
product volume

35. Data preparation and
feature engineering
Parse (log) data to identify shoppers’ needs
Measure potential product demand
Model product similarity (Spark GraphX / ElasticSearch)

36. Validation and
A/B testing
Limited capability on existing A/B testing platforms
to track specific products
Measure performance of new products across
experimental groups using in-house tracker

37. Results
Increased new product click-thru rate by 30 – 80%
Increased new product add-to-cart by 20 – 90%
Expected overall conversion to decrease—increased
instead (though not statistically significant)

38. Emphasizing
product quality

39. Intent
Improve customer experience throughout purchase journey
From online browsing to receiving of product
Product quality identified as key driver

40. Problem
How do we measure product “quality”?

41. Methodology (online)
Content (e.g., title quality, richness of content)
Reviews (e.g., average rating, negative reviews)
Performance (e.g., click-thru rate, browsing time)

42. Methodology (offline)
Perfect order rate (i.e., not cancelled, not returned, etc.)
Negative feedback (e.g., counterfeit, complaints, etc.)
Seller metrics (e.g., timely shipped-rate, return rate, etc.)

43. Data preparation and
feature engineering
Derive product features (e.g., title quality, image quality, etc.)
Measure content richness (e.g., attributes available,
grouping, etc.)
Measure delivery performance and customer feedback

44. Results
Improved quality of products displayed
Increased conversion by 3 – 5% for some countries
Small conversion change in other countries (non-significant)

45. Key takeaways
Data science is (i) team sport, (ii) partly R&D, (iii) iterative
How you use data to solve problems (methodology), data
preparation, and feature engineering > machine learning

46. Thank you!
eugene.yan@lazada.com