Building a messaging platform on MongoDB

1. Messaging Platform with
MongoDB Atlas
@eHarmony
Vijay Vangapandu
(Senior Principal Software Engineer)
Email: vijayvangapandu@eharmony.com
LinkedIn: http://bit.ly/vijay-ln
Elankumaran Srinivasan
(Senior Software Engineer)
Email: esrinivasan@eharmony.com
LinkedIn: http://bit.ly/elan-ln
June 20, 2017

2. • Our comprehensive & revealing
Relationship Questionnaire
• 29 Dimensions® of Compatibility
• 15 million matches per day globally
(10 billion since inception)
• 101 thousand photos uploaded per day
• 1 million communications per day
• 70 million registrants globally
Who is eHarmony?

5. eH Product Experience
Match ProfileMatch Listing Communication Newsfeed

6. 18 Step Guided Communication Flow

7. Match Comm Drop Off (match level)
• Decline in Guided communication stages completion since Jan 2016
 Users completing CEQ’s : -36.5%
 Users completing MHCS : -36.0%
 Users completing OEQ’s : -17.7%
 Users reaching open comm: -36.8%

8. Goals of the Redesign
• Simplified communication flow
• No lockdown or dependent stages
• Real time messaging system - Chat
• Extend the system to support richer content types
• Giphy
• Video/Photos
• Stickers
• Support various custom business requirements
• Improved user experience

9. New Communication System
Conversations Chat History User’s Recent Questions

10. RDBMS Table Distribution
20
3
2
10
3
1
1
Open Ended Questions
Smiles (Icebreaker)
Close Ended Questions
Must Have & Can’t Stands
Open Communication
Workflow & Metadata

11. Communication: RDBMS Table

12. Limitations With Current Relational Store
• Antiquated
• Large number of tables
• Complex Joins
• Performance Concerns
• Rigid database model

13. Communication – Microservice architecture
Redis Oracle
XMPP
HTTPS
Aggregation
Service
Data API Service
SSL
Match ServiceUser Service
Auth
Service
API Gateway
NodeJS APP Stack
FIREWALL
???

14. Characteristics of eH Use Cases
• Data Consistency
• Read Heavy
• Scalable
• Flexible Schema
• High Performance
• Rich Query Support

15. Communication – Microservice architecture
Redis Oracle
XMPP
HTTPS
Aggregation
Service
Data API Service
SSL
Match ServiceUser Service
Auth
Service
API Gateway
NodeJS APP Stack
FIREWALL
MongoDB

16. Why MongoDB Atlas?
• NoSQL DB as a Service
• High Availability
• Low Management Overhead
• Monitoring & Alerting
• Automated Backups
• Disaster Recovery
• Workload Isolation
• Strong Security
• No need for on-site DBA

17. Schema Modeling

18. Modeling Challenges
• Related Data
• Normalized Vs Denormalized
• Homogenous Vs Heterogeneous

19. Schema Modeling - 1
Send
Mark As Read
Read Receipt
Message Document
Receive

20. Schema Modeling - 2
Match ID
Send
Receive
Mark As Read
Read Receipt
123123123
UID-190
UID-530
UID-121
UID-700
MatchIds

21. Modeling – Array Of Messages
Pros
Cons
• Easy retrieval
• Logical grouping
• Memory usage & document size limitation
• Query complexities
• Document locking

22. Modeling – Individual Documents
Pros
• Simple Schema
• Memory efficient
• Simple queries

23. Final Document Model
Messages
User
Use Cases
• Conversations
• History
• Count
Use Cases
• Recently Asked Questions

24. Flexible Schema Model

25. Collection Sharding
Messages
User
Use Cases
• Conversations
• Chat History
• Count of new messages
Use Cases
• Recently Asked Questions
Shard Key
Shard Key

26. Query - Conversations
Conversations
User Match - 335567
Match - 112334
Match - 233456
Message 1
Message 2
Message N
Message N
Message N
Message 1
Message 1
....
........
Shard
Shard
Shard

27. Query - Conversations
db.getCollection('message').find({"matchId":NumberLong("11791804342")}).sort({"createdTimeStamp":-1}).limit(1)
API Services
Page & Size
.......
MongoDB AtlasUser-Match Store
Retrieve User Contacts
matchIds
M
O
N
G
O
S
.......
Shard-1
Shard-2
Shard-n

28. Query - Counts
Output

29. Query – Chat History
Output

30. Performance & Load Testing

31. Performance – Load Testing
eHarmony’s Data Center
Cloud Tests DAO as API Service
Mongo
Driver
MongoDB Atlas
• Network Latency with multiple hops
• Query Performance Impact
• Data Growth
• Sharding

32. Performance – Best Practices
• Indexing
– Too many or none.
– Compound Indexes
– Covered Queries
– Background Index creation.
• Leverage in place updates.

33. Configuration

34. Sizing
Chat Quick
Questions
Total Daily
Transactions
1 Year
Projection
2 year
Projection
Daily Non Peak 75, 000 290,000 365,000 100,375,000
Daily Peak 160, 000 360,000 520,000 46,800,000
Total 147,175,000 300,000,000
Data Size Projection For 2 Year Period
# Documents 300,000,000
Average Doc Size 433 Bytes
Compressed Doc Size 168.37 Bytes
Compression Ratio 0.67
Storage Size 47.18 GB
Index Size 12.49 GB

35. Mongo Atlas Configration
Primary
Secondary Secondary
Primary
Secondary Secondary
Mongos
Shard #1 Shard #2
32 GB 160 GBM50

36. Monitoring

37. Real Time Metrics

38. Mongo Atlas Stats

39. Alerting
MongoDB Atlas
• Replica set has no Primary
• Cluster missing active Mongos
• Number of connections
• Write Tickets
• Read Tickets
• Replication Lag
• Disk space
• Number of read
• Number of writes

40. Users In Communication

41. Total Communication Volume

42. Summary
Schema
Design
Load
Testing
Analyze
MongoDB Atlas
- Focus on the feature.
- Highly robust worry free
deployment.
- Excellent professional support.

43. Future Road Map
• Support other message types like videos/images/map location etc.
• Presentation ready feed.
• Monitor impact of additional shards on aggregate functions.
• Text Search.