06/18/2014 - Billing & Payments Engineering Meetup @ Netflix For this Meetup, we have invited speakers from several tech companies to give a series of lightening talks on challenges related to billing & payments systems. This event is for engineers who are interested in learning more about billing & payments systems. No previous experience with this kind of system is required to attend. Presenters: - Mathieu Chauvin - Engineering Manager for Payments @ Netflix - Taylor Wicksell - Sr. Software Engineer for Billing @ Netflix - Jean-Denis Greze - Engineer @ Dropbox - Alec Holmes - Software Engineer @ Square - Emmanuel Cron - Software Engineer III, Google Wallet @ Google - Paul Huang - Engineering Manager @ Survey Monkey - Anthony Zacharakis - Lead Engineer @ Lumos Labs - Shengyong Li / Feifeng Yang - Dir. Engineering Commerce / Tech Lead Payment @ Electronic Arts

1. Billing & Payments
Engineering Meetup
June 18th, 2014

2. • Mathieu Chauvin – Netflix
Validating Payment Information
• Taylor Wicksell – Netflix
Flexible Billing Integrations Using Events
• Jean-Denis Greze – Dropbox
Mistakes (and Wins) Building Payments at Dropbox
• Anthony Zacharakis – Lumosity
Billing Migrations: Maintaining Your (Data) Sanity
• Alec Holmes – Square
Scaling Merchant Payouts at Square
=== Break ===
• Paul Huang – SurveyMonkey
Billing Globalization and Auto Renewal Optimization
• Emmanuel Cron – Google Wallet
Moving from SE to Host Card Emulation
• Feifeng Yang / Michael Chen – Electronic Arts
Ecommerce at EA
• Krishnan Sridhar – LinkedIn
Real-Time Analytics and Smart Routing

3. Validating Payment Information
Mathieu Chauvin
@matchauvin – linkedin/matchauvin

4. Signing-Up for Netflix
• Subscription Based
• 30 Days Free Trial
• Payment Information
Collection

5. • Pro Forma
• Luhn Check (Mod10)
• Card Type vs Card# RegEx
• Authorization / Reversal
Payment Validation 101

6. But Wait! You Can Do More!

7. Authorization Amount
• $0.00, $1.00, or Full Price Authorization
• Don’t Forget the Reversal!
• Constraints by Country or by Card Type

8. Response Types
• Hard Declines
• Soft Declines

9. MCC
• What is an MCC?
• Figure out the Best MCC Fit

10. BINs
• What is a BIN?
• Card Types: Credit, Debit, Prepaids
• What About Non-Reloadable Prepaid Cards?

11. Forking Traffic
• Do You Use Several Payment Processors?
• Why Don’t You Test Which One Performs
Better?

12. Use Cases
• Sign-Up
• Updating Payment Information
• Returning Customer

13. Country Specific Behavior
• Adapt To Your Market

14. The Feedback Loop
• A/B Tests
• Analyze, Project & Evaluate Risks
• Test Again

15. Conclusion
• Try Everything
• Build Your Application Dynamically
• The Feedback Loop!
Thank You!

16. Flexible Billing Integrations
Using Events
Taylor Wicksell

17. The Billing Team
• Signups/Cancellations
• Recurring Billing
• Price and Tax Calculations
• Discounts / Gifts
• Financial Reporting

18. Current Architecture
• Netflix Data Center
• Batch Processing and
Customer Requests
• Oracle (Single-Master)
• Data driven

19. Cloud Migration - Primary Goals
• Join Netflix in the Cloud
• High Availability
• Multi-Regional Active/Active
• Scalability

20. New Design
• Amazon EC2
• Multi-Regional
• Cassandra
• Event Driven

21. Order Flow

22. Cross-Cutting Concerns
• Publishing Data To Interested Parties
– Hold Messaging
– Financial Reporting
• Metrics/Analytics
• Velocity Checks
• Debugging

23. Wiretap to Common Event Pipeline

24. Common Event Pipeline
Current Implementation
• SQS Entry Point
• Custom routing service
• Custom code for each
endpoint integration
• Weak ordering of events
Future Implementation
• Suro / Kafka Entry Point
• Configurable Routing and
Transformation
• Pluggable endpoint
integration
• Option for strong ordering
of events

26. Sting – Broad Analytics

27. Druid – Event Aggregation Metrics

28. ElasticSearch – Event Level Debugging

29. Mistakes (and Wins) Building
Payments At Dropbox
Jean-Denis Grèze & Dan Wheeler
June 18th, 2014

30. Backend Tips
• Not about increasing conversion
• Not about pricing
• Not about plan and feature optimizations
• Not about upselling
• Not about consumer SaaS at scale
• Not about self-serve in
SMB/SME/Enterprise

31. Pains of Scaling Payments
• Thousands of customers to millions of
customers
• SMB to Enterprise
– Custom flows!
• International expansion
– Fraud
– New payment methods (delayed settlements)
– Different price points

32. Out Of The Dark Ages
• For a long time, only 0.5 engineers
worked on payments and billing
• March 2013: consult w/ leading payment
engineers, PMs and executives on how to
build an amazing payments team
– 15+ in 1 year

33. Advice
• Build a payments + billing backend that is:
– Flexible
• Migrations (sadface)
• Requirements will change – often
– Auditable
• Always know why and state changed
– Append Only
• Never lose data

34. Team Ca$h

35. Team Ca$h
• Engineers
– Some finance background
– A lot of systems/analytics background
• PMs
– Some finance/accounting background
– Some product/marketing background
• Advisors
– Payments experts
– Tight feedback loop w/ finance
• Designers

36. Migrations

37. Migrations
• You will have to migrate
– 3rd-party vaulting to self vaulting
– New markets = new processors
– If you are a growing company, your internals will require
migrations
• Stakes are high
– Double-billing? Forgetting to charge some users?
Inadvertently moving users from one pricing category to
another?
• Old way:
– Ad hoc
– Tons of tests

38. I. Leverage Existing Code:
Equivalence
• Write equivalence between old and new
implementations (database, API, 3rd party
providers, tests, etc.)
• Run everything through both systems at
once, with equivalence being tested
• Every step of the way check that equivalence
relations hold (e.g., old-style invoice has a
new-style invoice equivalent)
• Turn off old system when everything works
for X amount of time

39. Migration Pro-Tip
• If you can migrate in both directions at
will on a per-endpoint basis, your life
will be awesome and people will love you.

40. Moneytree

41. Logging
• Dark Ages = tons of logging
• Very comprehensive, but ad-hoc = too
much effort to re-create state
• Human error

42. II. Automated Logging
• Automatically log
– Any DB write (graph, relational, etc.)
– Any 3rd party API call (and some internal calls like
email)
• Pre-log
– Any incoming 3rd party payload
• Can recreate past actions if we introduced a
regression
• LOG A REASON (and code version)
– 1 year is a long time

43. Automated Logging
=
Time Machine

44. Not too much data
• Dropbox = large scale for SaaS
• Hundreds of millions of users
(provisioning is hard)
• Tons of data, but still payments data <<
file storage data
– Although we do have the benefits of amazing
infrastructure

45. III. States, Not Deltas
• Generally # states << # deltas
• States
– Pro 100 + Packrat
– DfB + Harmony Enabled
• Deltas
– Add 5 licenses, 6 licenses, etc.
– Add 20 licenses and switch from monthly to yearly
• Use states and let the system figure out how
to get from start to end

46. IV. Possibility & Transitions Use
Same Code Paths
• No difference between:
– entity.is_valid_transition(end_state)
– entity.perform_transition(end_state)
• Except that writes are turned off for the
former.
• No change for “is something possible”
logic to be different than “do the thing”
logic.

47. States Are Nice
transition_space = MoneytreeTransitionsSpace.build_cross_product(
entity=me,
gateways=ALL_GATEWAYS,
plans=[Pro100, Pro200, Pro500, DfB, DfBTrial],
schedules=[Monthly, Yearly],
currencies=ALL_CURRENCIES,
features=ALL_FEATURES,
tax_profile=[NoTax, SimpleTax, ComplexTax],
)
# …
If transition_space.supports(FEATURE_PACKRAT):
# …

48. Write Protection
• Seems dumb, but need to be careful not
to accidentally change values in payments
world. Have clearly-defined code paths
that can touch state, talk to 3rd party
components, etc.

49. V. One More Lesson
• Payments + Billing != Finance
• Business requirements don’t always
translate to what’s best/easiest in the
world of accounting. You need to flexibly
work in both worlds – can’t risk the user
experience to make your finance dep’t
happy (and vice-versa)
• Get a great PM

50. VI. Why Payments Are Cool?
• Infrastructure?
– Payments service
– Provisioning service
• Product?
– Upsells? (+50% increase in revenue per user)
– Gating features?
• Product Infrastructure!
– Build a successful structure by emphasizing
hard problems in both worlds!

52. Now + The Future
• Other Cool Projects
– ML for risk/anomaly detection (e.g., for payment
methods that don’t settle immediately)
– Price AB testing (*)
– Cross-platform upsell framework
• Questions?
– dan@dropbox.com
– jeandenis@dropbox.com
• Hiring
– Get in touch!

53. Team Ca$h

54. Lumos Labs, Inc.
Data Sanity
A short treatise on
rebuilding a payment system

55. Lumos Labs, Inc.
the background

56. Lumos Labs, Inc.
Circa 2012
Decided to rewrite our payment system

57. Lumos Labs, Inc.
Reasons
Old payment system has a lot of
limitations

58. Lumos Labs, Inc.
Payment system limitations
• Hard to add additional gateways
• Models don’t reflect business reality
• Not built for reporting
• Code is brittle

59. Lumos Labs, Inc.
3 months later…

60. Lumos Labs, Inc.
New payment system features
• Trivial to add new payment methods
• Subscriptions are the core model
• Built with reporting in mind
• Separate, well encapsulated library

61. Lumos Labs, Inc.
Works like a charm!

62. Lumos Labs, Inc.
wait… just one problem

63. Lumos Labs, Inc.
Having two parallel
payment systems is not
sustainable
Surprise!
As it turns out

64. Lumos Labs, Inc.
Just deprecate the old one

65. Lumos Labs, Inc.
Just deprecate the old one
• Don't migrate anyone, let old users
churn out naturally (will take forever)

66. Lumos Labs, Inc.
Just deprecate the old one
• Don't migrate anyone, let old users
churn out naturally (will take forever)
• Migrate everyone, but only most
critical/current subscription info
(loses history)

67. Lumos Labs, Inc.
Just deprecate the old one
• Don't migrate anyone, let old users
churn out naturally (will take forever)
• Migrate everyone, but only most
critical/current subscription info
(loses history)
• Migrate everyone + full history
(tricky, lots of edge cases)

68. Lumos Labs, Inc.
so, what’d we
choose?
(drum roll)

69. Lumos Labs, Inc.
We decided to migrate
everyone and their entire
billing history

70. Lumos Labs, Inc.
Why?
One system
One history
One source of truth

71. Lumos Labs, Inc.

72. Lumos Labs, Inc.
enter the
sanity check

73. Lumos Labs, Inc.
enter the
sanity check
Just make a sanity check before and
after the migration to ask questions

74. Lumos Labs, Inc.
enter the
sanity check
Both models should answer certain
questions the same way, e.g:
• How much did the user pay for a subscription?
• How many total transactions did the user make?
• Was auto-renewal enabled on X date?

75. Lumos Labs, Inc.
class SanityCheck
Methods = [:subscriber?, :transaction_count] # etc.
def initialize(record)
@record = record
@before_values = SanityCheck.values(record)
end
def self.values(record)
Methods.map { |m| [m, record.send(m)] }.to_h
end
end
enter the
sanity check

76. Lumos Labs, Inc.
class SanityCheck
...
def check
@after_values = SanityCheck.values(record)
@diff = diff(@before_values, @after_values)
end
def diff(a, b)
a.delete_if { |k, v| b[k] == v }
.merge!(b.dup.delete_if { |k, v| a.has_key?(k) })
end
end
enter the
sanity check

77. Lumos Labs, Inc.
enter the
sanity check
User.each do |user|
sanity_check = SanityCheck.new(user)
user.migrate!
sanity_check.check
if sanity_check.diff.any?
# sanity check failed -- log an error, rollback, etc.
raise ActiveRecord::Rollback
else
# woo hoo, success!
user.update_attributes(:migrated => true)
end
end
config/initializers/user_auth.rb

78. Lumos Labs, Inc.
the result

79. Lumos Labs, Inc.
A resounding success!
the result

80. Lumos Labs, Inc.
(Mostly)
the result

81. Lumos Labs, Inc.
Did not work in all cases
● Payment system behavior changed over
time

82. Lumos Labs, Inc.
Did not work in all cases
● Payment system behavior changed over
time
● Some concepts did not map between
systems

83. Lumos Labs, Inc.
Handling the edge cases
● Replayed history as it would play out in
the new system
● Still kept most critical information the
same (e.g. transaction timestamps)

84. Lumos Labs, Inc.
Skip ahead to today
Migrated 99.9994%* of all our users
successfully to the new system
*The remaining .0006% live on for business, not technical reasons

85. Lumos Labs, Inc.
Thanks
Anthony Zacharakis
anthony@lumoslabs.com
@azacharax

86. Scaling Merchant Payouts at
Square
Alec Holmes

87. ‣ Infrastructure team that moves money
‣ Card processing (money in)
‣ Settlements (money out)
Payments Team

88. ‣ Monolithic Rails app
‣ Database storage and IO limits
‣ Controls around accounting data
Disaster Ahead

89. Monorail
Database
Payments Bills
Deposits
Where We Started

90. Ledger
Database
Payments Bills
Deposits
Monorail
Database
API
Current System

91. Challenge:
Migrating Live
Data
‣ Stricter money-moving event model
‣ Same event streams to all systems
‣ Smart queueing of events

92. Challenge:
Moving to APIs
‣ Replace direct data access with API calls
‣ Hadoop for analytics queries

93. That's it. All pennies accounted fo

95. Billing Globalization and Auto-
Renewal Optimization
• Paul Huang
• Engineering Manager
• SurveyMonkey
• June 18th, 2014

96. Our mission is to help people make
better decisions

97. We help people get answers to their questions
customers create surveys distribute to others recipients respond customers analyze
and gain insight

98. Our customers have invented many ways to use our tool
98

99. 90MUnique visitors
every month
17K
New sign-ups daily
“SurveyMonkey turns online
surveys into a hot business.”
“ Start-up companies using
‘freemium’ business models,
including SurveyMonkey,
are thriving as the cost of
computer power and storage
falls.”
One of the hottest
startups to watch.
2.4MSurvey responses
are generated daily
SurveyMonkey is the world’s largest survey company

100. Billing Globalization
In 2010...
Currencies:1 currency (USD).
Payment Methods: Credit Card, Invoice.
Pricing: one price per package.
Revenue: $ MM

101. Billing Globalization
In 2014...
Currencies: 39 international currencies.
Payment Methods: Credit Card, PayPal, Debit Card, Bank Transfer, iTunes, Invoice.
Pricing: each package with different price per currency, multiple prices allowed for
price testing.
Revenue: $$$$ MM

102. Auto-Renewal Optimization
Retry Logic: retry failed payments at different intervals, ~60% retry success rate
Account Updater: get users’ updated credit card data (number / expiration date)
before next renewal date, ~4% users at 96% success rate
Immediate Renewal: charge pending invoices when users update payment accounts,
~4% retry success rate improvement

103. The
Road
Ahead

104. Upcoming Billing Projects in 2014
VAT 2015: preparations for VAT rule changes in Europe in 2015
Brazil Payment Processing: set up local entity, integrate with a new Payment Service
Provider in Brazil
Continue Improving Retry Logic: increase retry frequencies

105. The End (BTW, we’re hiring…)

112. 112
* Jun 1st to Oct 17th