SpringOne Platform 2018 Geode is Not a Cache, it's an Analytics Engine - Sharif Ghazzawi and Evan Benoit, Resonate

1. Geode is Not a Cache,
it's an Analytics Engine!
By Evan Benoit (evan.benoit@resonate.com)
and Sharif Ghazzawi (sharif.ghazzawi@resonate.com)

2. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Who is Resonate?
Marketing and Advertising Technology Company
Located in Reston, VA
Give our clients insights into their customers’ values and motivations
Hiring Spring and Big Data Engineers!
http://www.resonate.com
2

3. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
What Kind of Data Do We Have?
We model predictions for thousands of different attributes
• Likes, Dislikes, Motivations, Behavior, Sentiments
3
1000’s of attributes
200 million
cookies
1.7 trillion total
predictions!
1000’s of sites
21.9 billion total
site hits

4. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
What Do We Do With the Data?
Our SaaS platform computes thousands of insights for our clients’ sites
• Example: “How many cookies hit my homepage yesterday that we’ve modeled
as female democrats, and how does that compare to the general population?”
4
Women Dems
Home
page
Women Dems

5. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/ 5
Each Insights Report requires
thousands of set operations
to be performed ad hoc,
within seconds!

6. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Key Take-aways
Geode can be used as more than a simple Key-Value cache; it can run functions on
data in-memory.
Probabilistic Data Structures can be used in many industries to perform set
operations at scale.
A Spring/Geode architecture can improve performance and scalability.
6

7. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
What Didn’t Work? HBase
Brute force approach: HBase co-processors sequentially scanning bitmaps
Completely inappropriate use of HBase!
40-node cluster, 30 second queries
Essentially using HBase as an in-memory database
7
1000’s of attributes
200 million
cookies
1000’s of sites
Sequential scan
Sequential scan
Sequential scan

8. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Probabilistic Data Structures for Estimating
Cardinality of a Set
We have a counting problem. You probably do, too.
Our users don’t require exact precision. We’re not a bank!
Probabilistic data structures can estimate the cardinality of a set
Data uses in fixed amount of Time and Space
8

9. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Yahoo Theta Sketch
Yahoo’s Theta Sketches give you estimated counts in a fixed amount of space…
… and they also support set operations!!
9
Example from https://datasketches.github.io/docs/Theta/ThetaJavaExample.html

10. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Sketches Begin Multiplying Like Rabbits
Sketches can’t contain any additional metadata
We need a sketch for each attribute, for each tag
Next thing we know, we have 150 Million sketches, 2 Terabytes total
We need a place to store all these sketches
11
Example from https://datasketches.github.io/docs/Theta/ThetaJavaExample.html
1000’s of attributes
1000’s of
sites

11. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
We Need a Distributed In-memory Database…

12. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
System Architecture

13. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
System Characteristics
Data Locality
• We register Java methods built with the Theta Sketch library into Geode
• These set operations run close to the data. No need to shuffle data between
nodes. The sketches never leave Geode; Geode just returns the final count.
Performance
• Computing the cardinality of a set is now an O(1) lookup instead of O(n) full
table scan
• Output of a set operation is a sketch rather than a number, allowing multiple
set operations to be chained together efficiently
14

14. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
System Characteristics
Fault Tolerance/Resiliency
• Geode Locators and Servers can be added/removed with zero downtime
• AWS Elastic Load Balancer (ELB) detects when a Geode ECS node is
unhealthy, kills the Docker container, spawns a new one
• Nodes are distributed across multiple AWS Availability Zones
Scalability
• Just add more servers and rebalance
15

15. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Geode Regions
Geode gives a lot of options for how to persist and replicate your data
Original design called for persistent, replicated, partitioned Geode regions
• But persistence and replication made it difficult to swap out bad Geode nodes
• It checks filesystem to ensure that no data was lost – Slow!
• Data is shuffled to honor the replication config – Slow!
Solution: We use AWS S3 as our persistent, replicated layer, not Geode
• Geode reads-through from S3 whenever it doesn’t have the data
• We read-through "parcels" containing thousands of sketches instead of
individually one at-a-time
16

16. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Geode and Docker
Geode doesn’t easily support Docker/ECS
• Recommended way of starting locators and servers is via Gfsh
• Gfsh starts locator/server in the background then exits
• Docker container exits/dies once there is no process running in the foreground
Solution: We added a dummy foreground process to keep Docker container up
17

17. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Geode and ECS
Geode Locators keep state on local disk, which is transient in AWS ECS
• Don't assume existence of a local disk
• Makes it difficult to honor "12 factor app" principles
Solution: We deploy and associate Locator docker instances to EC2 nodes with
storage
18

18. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Geode and Spring Boot
Spring-data-geode didn’t fit our production architecture
• Initially we tried embedding Geode in Spring Boot
• No lifecycle hooks for Spring apps to tap into for heath checks
• Makes designing fault tolerance/resiliency and scalability difficult
Solution: We run Geode as a standalone process, not embedded in spring
19

19. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Geode and Configuration Data
Improved configuration management flexibility
• Geode comes with a tightly integrated configuration management sub-system
• Configs are uploaded to locators, distributed to servers
Many organizations already have a configuration management system
• e.g. consul, zookeeper, spring-cloud-config
We’d love to see Geode’s configuration system be pluggable/swappable
20

20. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Testing Distributed Systems
As with any distributed system, make sure you understand the consistency,
availability and partition-tolerance guarantees provided by your tools, and
ultimately your system
• Identify what parts of your system will provide redundancy
• How does your system respond to various failure scenarios?
• Test, Test, Test those scenarios
21

21. U nless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Com m ons
A ttribution-NonCom mercial license: http://creativecom m ons.org/licenses/by -nc/3.0/
Summary
We Deployed a Spring and Geode Architecture
Containing Yahoo Theta Sketches
Significantly improved our main report’s performance
Reduced operating costs by 95% over our previous HBase implementation
Increased scalability
Simplified operations
Increased resiliency
22

22. Resonate is HIRING in RESTON!
Spring Engineers
Big Data Engineers (Spark, Geode, Hadoop, Kafka)
Dev Ops Engineers (AWS)
UX Engineers (Ember.js)
https://www.resonate.com/about/careers/
#springone@s1p