A/B Experimentation at Pinterest

1. Krishna Gade
Why everything is an A/B test at
Pinterest

2. Confidentia
l
2

3. What is Pinterest?
A visual bookmarking tool
Discover an inspiring idea
Save it to a board
Go do it

4. Pinterest is a data product

6. Data Flow
App
events
Kafka
Secor
Skyline
Pinball
Redshift
Pinalytics
Features
Qubole
(Hadoop)
Singer

7. Confidentia
l
Metrics
7

8. Confidentia
l
User Lifecycle
New
Users
Active Users
Dormant Users
Acquisition
Activation
Churn
Churn Resurrection

9. Confidentia
l
Terminology
● MAU – Monthly Active User
● WAU – Weekly Active User
● DAU – Daily Active User
● WAR – Weekly Active Repinner
● WAC – Weekly Active Clickthrougher
● WARC – Weekly Active Repinner or Clickthrougher
● 1d7 – User who comes back at-least once in 7 days after signup
● 1rc7 - User who repins/clicks at-least once in 7 days after signup

10. Confidentia
l
MAUs
● Current progress to quarterly goal
● MAU forecast
● MAUs by app/gender/country
● MAU Accounting

11. Confidentia
l
MAU Accounting

12. Confidentia
l
Acquisition Metrics
● Total Signups
● Signups by App
● Signups by Referrer
● Invites Sent
● Unique Invite Senders
● Invite Signups

13. Confidentia
l
Activation Metrics
Activation Rate = 1rc7/signups
● Activation Rate by App/Gender/Referrer
● Overall signups to 1rc7
● Signup engagement funnel
● Signups to WAU 35 days after signup

14. Confidentia
l
Engagement Metrics
● *AU ratios
● # of Emails sent
● # of Emails opened
● # of Emails clicked
● # of Notifications sent
● # of Notifications
clicked

15. Confidentia
l
Resurrection Metrics
● # of Users who were
dormant for 28+ days
but came back, split by
platform, referrer etc

16. Confidentia
l
Takeaways
● Emphasize on activation (process of converting
a new user to MAU)
● Activation critical to long-term growth
● Deeply understand how different segments of
users interact

17. Confidentia
l
Automated Metrics
17

18. Confidentia
l

19. Confidentia
l
Chart creation

20. Confidentia
l
Reuse existing query result

21. Confidentia
l
Metrics creation

22. Confidentia
l
A/B Experimentation
22

23. Confidentia
l
Categories of Experiments
● A/B testing new features
● Copy testing
● SEO changes
● Infrastructure updates

24. Confidentia
l
A/B Testing Framework

25. Confidentia
l

26. Confidentia
l

27. Confidentia
l

28. Confidentia
l
A/B Testing Dashboard

29. Confidentia
l

30. Confidentia
l
p-value (via chi-squared test)
Statistical tests for validation

31. Confidentia
l
Impact of Badging

32. Confidentia
l
Impact of Badging
● 7% lift in DAU
● 2.5% lift over long term
● Effective at increasing
engagement levels

33. Confidentia
l
Copy Testing
Definition: Testing different variations of text we
display to the user.
● A/B Testing copy was a tedious process
● Even when we did this we were optimizing only
for English

34. Confidentia
l
New Approach
● Ability to run a copy test w/o changing a line of
code.
● Support for translating text dynamically
● Use Multi-armed bandits to converge to showing
best performing text

35. Confidentia
l
Multi Armed Bandits

36. Confidentia
l
SEO
36
50+ Billion Pins
categorized by people
into more than
1 Billion Boards

37. Confidentia
l
37
What makes SEO so challenging?
What is the
Impact of the
new SEO
feature?
Will this new
change affect
SEO?
Uhhh...
Dunno...
Uhhh...
Ask Google.

38. Confidentia
l
KEEP
CALM
and
RUN AN
SEO EXPERIMENT

39. Confidentia
l
What is an SEO experiment?
39
Regular AB Experiment SEO Experiment
Metrics User behavior Search engine referred traffic
Unique id for hashing userid / sessionid page url

40. Confidentia
l
What is an SEO experiment?
Launch
● Measure the gap
● Before vs After launch

41. Confidentia
l
Dashboard
41
DIFF = enabled - control
Traffic Gain

42. Confidentia
l
42
Decision Making
1 2
3
4
SHIP!
SHIP!
SHUT DOWN!
VALIDATE

43. Confidentia
l
43
Experiment Driven SEO
What is the
Impact of the
new SEO
feature?
We got XX% of
traffic growth from it!
Can we use more
resources to make
bigger impact?
No, the
experiment
result is flat, so
let’s ship it.
SEO
experiment
SEO
experiment
Is my UI change
affecting SEO
negatively?

44. Confidentia
l
Why you should be A/B testing
even your infrastructure

45. Confidentia
l
Infrastructure Changes
● 2013: A new web framework
● 2014: Pyapns
● 2015: User service

46. Confidentia
l
Experiment results for replacing PyAPNs

47. Confidentia
l
Conclusions
● Figure out what success means
● Define your metrics
● Automate metric computation
● A/B test everything!

48. Confidentia
l
Thank You!