The following might sound all too familiar: We’re taking and sharing more photos than ever before, and as a result we’re drowning in our photos. Somewhere in that pile or stream of photos are the ones that really matter – if only we could find them. To make this easier Albumprinter created a solution which recently launched with more than 500 million photos for 1 million existing customers. Neo4j is a key component in this new solution. It holds all the information of how photos relate to each other, to which user they belong and with which users they are shared. In total the store size is over 1.3 TB. We will explain why we choose Neo4j, how we use it and how we scaled to handle the massive import. Of course we will also share the lessons learned and how we solved some of our challenges.

1. Building a Scalable
Platform for Sharing
500 Million Photos
Wouter Crooy & Ruben Heusinkveld
Solution Architect & Technical Lead, Albumprinter

2. Wouter Crooy
Solution Architect, Albumprinter
@wcrooy

3. Ruben Heusinkveld
Technical Lead, Albumprinter
@rheusinkveld

4. Who are we
• Wouter Crooy – Solution Architect
• Ruben Heusinkveld – Technical Lead
• Neo4j Certified Professionals

5. The photo organizer
• Deliver well organized, easy to use and secure storage for all
your images
• Ease the process of selecting photos for creating photo
products
• Started as part of a R&D ‘Skunk works’ project

6. The photo organizer

7. The photo organizer

8. The photo organizer

9. The photo organizer – from photos to products

10. The photo organizer – demo
https://minnebanken.no

11. The challenge

12. The challenge
• Replace legacy system with the new photo organizer
• Move 1.3 PB of photos from on premise to cloud storage
• Analyze & organize all photos (511 million)
• Data cleansing while importing
• Using the same technology / architecture during import and
after
• Ability to add features while importing
• Core of the systems are built in .NET

13. The import
• Hard deadline
• Factory closing that holds the data center with all photos
• Started 1st of April
• Minimum processing of 150 images / second
• ~500 queries / second to Neo4j
• Up to 700 EC2 instances on AWS

14. How we did it
• Micro services
• Command Query Responsibility Segregation (CQRS)
• Cluster
• Multiple write nodes
• Single master read only nodes
• HAProxy
• Cypher only via REST interface
• .NET Neo4jClient

15. Architecture
Neo4j
Cluster
HaProxy
Query Command
Frontend
Amazon
ElastiCache
Photo
processors
Other Services
 Notifications
 Authentication
 ....
Preview
Generation
Storage
Other database
(clusters)

16. Why we choose Neo4j
• Close to domain model
• Not an ordinary (relational) database
• Looking for relations between photos/users
• Scalable
• Flexible schema
• Natural / fluent queries
• ACID / data consistency

17. The design

18. Graph model
User
Photo
Photo
Event
BelongsTo
BelongsTo
Contains
Contains
Raw
Exif
HasExif
HasEvent
Raw
ExifHasExif

19. Graph model
User
Photo
BelongsTo Photo
Day
Day
Month
Year
DateTaken
Day Month
DateTaken
Day
Time
line
Year
HasTimeline
BelongsTo

20. Graph model
User PhotoBelongsTo
Collec
tion
OwnsCollection HasItem
UserIsSharedWith

21. Our Neo4j database
• More than 1 billion nodes
• 4.1 billion properties
• 2.6 billion relations
• Total store size of 863 GB

22. Command Query Responsibility Segregation
• Seperation between writing and reading data
• Different model between Query and Command API
• Independent scaling
UI
Cache
DB
Component
Component
Update
Publish
Write
Query
Command

23. Bumps and Solutions

24. CQRS Seperate Reads & Writes
• No active event publishing in place
• Specific scenarios for updating / writing data
• Ability to create seperate model for read and write
• Updates (pieces) the user graph
• Requires reliable and consistent read
• Scale out -> overloading locking of (user) graph
• After import
• Low performance scenarios -> cache with lower update priority

25. Read after write consistency
• All reads should contain the very latest and most accurate data
• Replication delay between servers
• Split on consistency
• Article by Aseem Kishore:
• https://neo4j.com/blog/advanced-neo4j-fiftythree-reading-writing-
scaling/

26. Graph locking
• Concurrency challenge
• Scale-out => more images from the same user
• Manage the input
• High spread of user/image combination
• Prevent concurrent analysis of multiple images from the same user
• :GET
/db/manage/server/jmx/domain/org.neo4j/instance%3Dkernel%
230%2Cname%3DLocking

27. Batch insert vs single insert
• Cypher CSV import per 1000 records
• Prevent locking caused by concurrency issues

28. No infinite scale out
• Find the sweet spot for the amount of cluster nodes
• +1 nodes => more replications updates => higher load on write
master

29. Timeline
• We’re looking for photos which should belong to each other
based on date-taken.
• Moving from full property scan to graph walking via the timeline.
• For large collection 75% less DB-hits
• Walking the timeline if looking for photos within a certain
timeframe
• Less photos to evaluate for property scan (SecondsSinceEpoch)
• Works perfectly for year, month, day selections

30. .NET & Rest interface
• Custom headers to REST Cypher endpoint (Filtered by HaProxy)
• To route to multiple write servers
• Sticky session per user
• Custom additions to .NET Neo4jclient
• Managing JSON resultset

31. Graph design considerations
• Property scan
• (User)<-[:BelongsTo]-(Photo)
• More photos
• Property search => full-graph-scan
• Differentiating property
• Create node
• No path/clustered indexes…. (yet..  )
• Making changes to the schema….
• For 550+ million nodes

32. Graph design improvements
Property search
match (u:User { Id: “001"})<-
[:BelongsTo]-(p:Photo)
where p.Favourite = true
return p
=> 2812 db hits
Node/Relationship search
match (u:User { Id: "001"})-
[:HasFavourites]-(f:Favourites)<-
[:IsFavourite]-(p:Photo)
return p
=> 13 db hits
• dbms.logs.query.* (don’t forget to enable parameters resolving)
• Our alternative: Integrate with Kibana / Elasticsearch
• https://neo4j.com/docs/operations-manual/current/reference/

33. The future

34. The future
• Neo4j 3.x
• Bolt
• Datamining
• Procedures / APOC

35. That’s a wrap