VoltDB Shard It: Java Morning slides

1. VoltDB: Shard it!
By Vitalii Torshyn

2. VoltDB Designed for
● Network Activity Monitoring / Security
● Real-Time Analytics/Monitoring
● Telecom: billing, QoS, Policy
Management
● Financial Services
● Gaming

3. Agenda
#1: Academic issues: Shard, HP, VP
#2: What is Volt DB? Why ! VoltDB?
#3: Real VoltDB application
#4: Tips and Tweaks: you are not
supposed to do that at all

4. Agenda
#1: Academic issues: Shard, HP, VP
#2: What is Volt DB? Why ! Volt DB?
#3: Simple Volt DB application
#4: Tips and Tweaks: you are not
supposed to do that at all

5. Academic issues
What is:
● Shard, Horizontal Partitioning
● ACID complaint DBMS
● In-memory and real time DB

6. Academic issues
What is:
● Shard, Horizontal Partitioning
● ACID compliant DBMS
● In-memory and real time DB

9. Straight forward approach

10. What is Horizontal
Partitioning

11. What is shard

12. Use over time for shard

13. Academic issues
What is:
● Shard, Horizontal Partitioning
● ACID compliant DBMS
● In-memory and real time DB

14. ACID
● Atomicity
● Consistency
● Isolation
● Durability

15. Academic issues
What is:
● Shard, Horizontal Partitioning
● ACID compliant DBMS
● In-memory and real time database

16. In-memory/real time
database
● Storage is option, not requirement
● Key value? No!
● Data access is cheap (no I/O)
● Real time processing

17. Agenda
#1: Academic issues: Shard,HP,VP
#2: What is Volt DB? Why?! Volt DB?
#3: Simple Volt DB application
#4: Tips and Tweaks: you are not
supposed to do that at all

18. db-engines.com: Ranking

19. What is VoltDB?
●ACID compliant DBMS
●In-memory database
●Real time
●Shared nothing architecture (SN)
●SQL support
●Java Stored procedures

20. Java over JNI
SP, Query Prepare,
Transfer
C++ Engine:
Indexing,
Lookup, Mem.
Management ...

21. Why Volt DB?
● Low cost of scaling
● Low latency
● Automatic Cross-partition joins (no app.
Code)
●Multi-master replication (HA, K-Safety)
●No buffer management (In memory)
● Lockless
● Licensing (GPLv3, Enterprise)
●Client libraries: Java, Python, C++, C#...

22. Requests execution

23. Access and Networking
●RESTful HTTP/JSON API
● Socket connections
● Java API
● JDBC

24. Is VoltDB fast enough?
● Sharding
●High Speed Small transactions
●No journaling required
● Buffering is not required

27. Replication: K-Safety = 1

29. Replication: K-Safety = 1

30. Replication: Active-Passive

31. K-Factor: performance

32. Volt DB tools
● csvloader — can load data from CSV
file
● exporttofile — exports to CSV/TSV file
● sqlcmd — mysql like client
● voltadmin - administrative functions
● voltdb — catalog (DB) management
and server

33. Questions?

34. Volt DB: Super Chat

35. Agenda
#1: Academic issues: Shard,HP,VP
#2: What is Volt DB? Why?! Volt DB?
#3: Simple Volt DB application(Super Chat)
#4: Tips and Tweaks: you are not
supposed to do that at all

36. Super Chat: flow

37. DB Schema file
1: CREATE TABLE messages (
uid BIGINT NOT NULL,
nick VARCHAR(64) NOT NULL,
ip VARCHAR(16) NOT NULL,
text VARCHAR(1024)
);

38. DB Schema file
1: CREATE TABLE messages (
uid BIGINT NOT NULL,
nick VARCHAR(64) NOT NULL,
ip VARCHAR(16) NOT NULL,
text VARCHAR(1024)
);
2: CREATE INDEX messages_idx ON messages (nick, ip);

39. DB Schema file
1: CREATE TABLE messages (
uid BIGINT NOT NULL,
nick VARCHAR(64) NOT NULL,
ip VARCHAR(16) NOT NULL,
text VARCHAR(1024)
);
2: CREATE INDEX messages_idx ON messages (nick, ip);
3: PARTITION TABLE messages ON COLUMN nick;

40. DB Schema file
1: CREATE TABLE messages (
uid BIGINT NOT NULL,
nick VARCHAR(64) NOT NULL,
ip VARCHAR(16) NOT NULL,
text VARCHAR(1024)
);
2: CREATE INDEX messages_idx ON messages (nick, ip);
3: PARTITION TABLE messages ON COLUMN nick;
4: CREATE PROCEDURE FROM CLASS vposter.procedures.AddMessage;

41. DB Schema file
1: CREATE TABLE messages (
uid BIGINT NOT NULL,
nick VARCHAR(64) NOT NULL,
ip VARCHAR(16) NOT NULL,
text VARCHAR(1024)
);
2: CREATE INDEX messages_idx ON messages (nick, ip);
3: PARTITION TABLE messages ON COLUMN nick;
4: CREATE PROCEDURE FROM CLASS vposter.procedures.AddMessage;
5: PARTITION PROCEDURE AddMessage ON TABLE messages COLUMN
nick;

42. DB Schema file
1: CREATE TABLE messages (
uid BIGINT NOT NULL,
nick VARCHAR(64) NOT NULL,
ip VARCHAR(16) NOT NULL,
text VARCHAR(1024)
);
2: CREATE INDEX messages_idx ON messages (nick, ip);
3: PARTITION TABLE messages ON COLUMN nick;
4: CREATE PROCEDURE FROM CLASS vposter.procedures.AddMessage;
5: PARTITION PROCEDURE AddMessage ON TABLE messages COLUMN
nick;

43. Simple Java Procedure

44. Putting All Together
sh $ javac -cp "$CLASS_PATH:/opt/voltdb-3.7/voltdb/*"
java/vposter/procedures/AddMessage.java
sh$ voltdb compile --classpath=./java/ -o voltdb-catalog.jar
/path/to/schema/schema-ddl.sql
# Finaly, run the server
sh$ voltdb create catalog voltdb-catalog.jar

45. Questions?

46. Agenda
#1: Academic issues: Shard,HP,VP
#2: What is Volt DB? Why?! Volt DB?
#3: Simple Volt DB application(Super Chat)
#4: Tips and Tweaks: you are not
supposed to do that at all

47. Tips and Tweaks
● Java Stored procedures
● Schema file: restrictions and syntax
● Client code vs Server code
● TPS: Performance Testing
● Deployment configuration

48. Java Stored procedures
● Minimize hard math. calculations, I.e. let's
client do what it needs
● Minimize SQL queue, i.e. usage of
lists/arrays as parameters is real
optimization
● Do not return huge chunk of data
● To Throw or Not To Throw
● Use statuses (application and response)

49. Schema file: restrictions and
syntax
● Use comments
● Renaming columns/tables in DDL
● Constraint LIMIT PARTITION ROWS
● Standard constraints
● Views as mechanism of aggregation

50. Client code vs Server code
● Let server aggregate data, let client
process data
● Table-Of-Tables
● Optimize Insertions
● Why SELECT * requests are
dangerous?

51. TPS: Performance Testing
● @STATISTICS usage
● @EXPLAIN[PROC] usage
● @SystemInformation

52. Questions?

53. References
l http://odbms.org/download/VoltDBTechnicalOverview.pdf
l http://www.mysqlperformanceblog.com/2011/02/28/is-voltdb-really-as-scalable-as-they-claim/
l http://voltdb.com
l http://techledger.wordpress.com/2011/07/08/voltdb-faq/
l http://highscalability.com/blog/2010/6/28/voltdb-decapitates-six-sql-urban-myths-and-delivers-internet.html
l http://www.perfdynamics.com/Manifesto/USLscalability.html#tth_sEc1
l git@github.com:vtorshyn/voltdb-shardit-src.git