These slides were from my Hippo GetTogether 2013 presentation. During this presentation I went into detail about the architecture behind our high performance relevance platform. The talk will also cover why we chose CouchBase for storage and how Elasticsearch can be used for search and analytics. I shared how we integrated and leverage both products full-circle from within our Hippo CMS product.

1. Building a relevance platform
with Couchbase and
Elasticsearch
Hippo GetTogether, 21 June 2013
Jeroen Reijn | @jreijn | #hgt2013
Hippo GetTogether 2013
follow the Hippo trail

2. follow the Hippo trail
Hippo GetTogether 2013
About me
• Architect @ Hippo
• DevOps guy
• Blogger @ http://blog.jeroenreijn.com

3. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Relevance?

4. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
“The capability of a search
engine or function to
retrieve data appropriate
to a user's needs.”
http://www.thefreedictionary.com/relevance

5. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto

6. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
How we deliver
relevant content
@Hippo

7. follow the Hippo trail
Hippo GetTogether 2013
Registration
Visitor - entity making HTTP requests
Collector - records data about a visitor or his behavior
Example: location collector (GeoIPCollector)
Targeting Data - all data about a specific visitor
Example: IP address is located in Amsterdam

8. follow the Hippo trail
Hippo GetTogether 2013
Matching
Characteristic - a type of fact about visitors
Example: "comes from a city", "experiences a type of
weather"
Target Group - the specification of a Characteristic
Example: "comes from a European city", "comes from
Amsterdam"
Persona - one or more target groups that describe a
certain type of visitor
Example: "Jim, the European urban consumer",
"Alice, the Pet owner"

9. follow the Hippo trail
Hippo GetTogether 2013
What do we store?
Request log
Targeting data
Statistics
Averages, e.g. how many visitors became which persona

10. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
BIG DATA !!

11. follow the Hippo trail
Hippo GetTogether 2013
Real-time analysis

12. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Architecture

13. follow the Hippo trail
Hippo GetTogether 2013
RDBMS
Hippo Delivery Tier
Hippo Repository
App server
XMLJSON (X)HTML

14. follow the Hippo trail
Hippo GetTogether 2013
Delivery Tier
URL Matching
Fetch content
Compose output
Request
Response

15. follow the Hippo trail
Hippo GetTogether 2013
Delivery Tier
URL Matching
Targeting Data Collection
Compose output
Request
Response
Fetch content
Scoring

16. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Scaling

17. follow the Hippo trail
Hippo GetTogether 2013
RDBMS
Hippo Delivery Tier
Hippo Repository
App server
Hippo Delivery Tier
Hippo Repository
App server
Scaling out

18. follow the Hippo trail
Hippo GetTogether 2013
RDBMS
Delivery Tier
Repository
App server
Delivery Tier
Repository
App server
Scaling out
Targeting
Datastore

19. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
What kind of ‘storage’?

20. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Question?

21. follow the Hippo trail
Hippo GetTogether 2013
Distributed Cache?

22. follow the Hippo trail
Hippo GetTogether 2013
We have a winner!

23. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Requirements
change!

24. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
NoSQL to the rescue

25. follow the Hippo trail
Hippo GetTogether 2013
Suitable types
• Key-value store
• Document database

26. follow the Hippo trail
Hippo GetTogether 2013
Assessment Criteria
Maturity Data model
Consistency model
PerformanceReplication
Caching model Query model
Monitoring
Scalability
Reliability
Support

27. follow the Hippo trail
Hippo GetTogether 2013
Selection Criteria
• Performance
• Scalability
• Schema flexibility
• Simplicity
• Monitoring
• Support

28. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Performance !!
Performance !!!!

29. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Scalability

30. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Schema flexibility

31. follow the Hippo trail
Hippo GetTogether 2013
{
"visitorId": "7a1c7e75-8539-40",
"pageUrl": "http://localhost:8080/site/news",
"pathInfo": "/news",
"remoteAddr": "127.0.0.1",
"referer": "http://localhost:8080/site/",
"timestamp": 1371419505909,
"collectorData": {
"geo": {
"country": "",
"city": "",
"latitude": 0,
"longitude": 0
},
"returningvisitor": false,
"channel": "English Website"
},
"personaIdScores": [],
"globalPersonaIdScores": []
}
Request log document

32. follow the Hippo trail
Hippo GetTogether 2013
{
"geo": {
"collectorId": "geo",
"city": "",
"country": "",
"latitude": 0,
"longitude": 0
},
"channel": {
"collectorId": "channel",
"channels": [
"English Website"
],
"lastVisitedChannel": "English Website"
}
}
Visitor document

33. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Simplicity

34. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Monitoring

35. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Support

36. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Couchbase

37. follow the Hippo trail
Hippo GetTogether 2013
Why Couchbase?
• Drop-in replacement for memcached
• Read/Write-through cache
• High throughput
• Easy scalability
• Schema flexibility
• Low latency

38. follow the Hippo trail
Hippo GetTogether 2013
Couchbase
• Open Source
• Document-oriented
• Easy Scalable
• Consistent High Performance
• Apache license

39. follow the Hippo trail
Hippo GetTogether 2013
Performance
• Object managed cache
• Write Queue to disk
• Avoids Cold Cache

40. follow the Hippo trail
Hippo GetTogether 2013
Source: http://www.slideshare.net/Couchbase/benchmarking-couchbase
Copyright © Altoros Systems, Inc.

41. follow the Hippo trail
Hippo GetTogether 2013
Easy scalable
• Auto sharding
• Cross cluster replication (XDCR)
• Master - Master replication

42. follow the Hippo trail
Hippo GetTogether 2013
Flexible data model
• Native JSON support
• Incremental Map Reduce
• Gives power to the developer

43. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
How we run
Couchbase @Hippo

44. follow the Hippo trail
Hippo GetTogether 2013
Load Balancer
Database cluster
Hippo Delivery Tier
Couchbase cluster
•Request log data
•Targeting data
•Statistics data

45. follow the Hippo trail
Hippo GetTogether 2013
Query capabilities
• Querying via views
• Secondary indexes via views
• Views based on Map - Reduce
• Lacks some advanced query capabilities

46. follow the Hippo trail
Hippo GetTogether 2013
Elasticsearch
• Apache Lucene
• Designed to be distributed
• Schema free
• Apache license
• RESTful API

47. follow the Hippo trail
Hippo GetTogether 2013
Added value of ES
• Full text search
• Faceted search
• Geo spatial search
• All in (near) real-time

48. follow the Hippo trail
Hippo GetTogether 2013
Couchbase Server Cluster Elasticsearch Server Cluster
Hippo Delivery Tier
Java API
Write
Read
XDCR Couchbase ES
Transport plugin
Replicating to ES

49. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
What’s Next?

50. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
What’s Next?

51. follow the Hippo trail
Hippo GetTogether 2013
Advanced analytics

52. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Demo time!

53. follow the Hippo trail
Hippo GetTogether 2013
OneHippo @ Goto
Thank you!
Questions?
j.reijn@onehippo.com | @jreijn
ps. We’re hiring!