Adding a new technology to your development process can be challenging, and the distributed nature of Apache Cassandra can make it daunting. However, recent improvements in drivers, utilities and tooling have simplified the process making it easier than ever before to develop software with Apache Cassandra. In this presentation we will cover essential knowledge for all developers wanting to efficiently create reliable Apache Cassandra based solutions. Topics will include: - Language and Driver selection - Optimizing Driver configuration - Productive Developer environments using ccm, Vagrant and DataStax DevCenter - Creating appropriate test data - Unit testing - Automated integration testing New and existing users will leave this presentation with the necessary knowledge to make their next Apache Cassandra project a success.

1. CASSANDRA-SF 2014
SUCCESSFUL SOFTWARE
DEVELOPMENT WITH
CASSANDRA
Nate McCall
@zznate
#CassandraSummit
Co-Founder & Sr. Technical Consultant
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

2. About The Last Pickle.
!
Work with clients to deliver and improve
Apache Cassandra based solutions.
!
Based in New Zealand & USA.

3. OVERVIEW

4. Overview:
What makes
a software development
project successful?

5. Overview: Successful Software Development
- it ships
- maintainable
- good test coverage
- check out and build

6. Overview:
Impedance mismatch:
distributed systems
development
on a laptop.

7. GETTING STARTED:
FOLLOW THE PATH OF LEAST
RESISTANCE

8. Getting Started:
!
JVM-Based
if at all Possible.

9. Getting Started:
!
Python Otherwise.
https://github.com/datastax/python-driver

10. Getting Started:
!
C#?
https://github.com/datastax/csharp-driver

11. Getting Started:
!
Ruby?
https://github.com/datastax/ruby-driver

12. Getting Started:
!
ORM?
maybe - only if it’s very simple
more later…
http://www.datastax.com/documentation/developer/java-driver/2.1/java-driver/reference/crudOperations.html

13. DATA MODELING

14. Data Modeling:
!
… a topic unto itself.
But quickly:

15. Data Modeling - Quickly
!
• It’s Hard
• Do research
• #1 performance problem
• Tip: don’t “port” your schema

16. DEVELOPER PRODUCTIVITY

17. Productivity:
!
use CQL

18. Productivity - Using CQL:
!
• tools support
• easy tracing (and trace discovery)
• documentation*
*Maintained in-tree:
https://github.com/apache/cassandra/blob/cassandra-1.2/doc/cql3/CQL.textile
https://github.com/apache/cassandra/blob/cassandra-2.0/doc/cql3/CQL.textile
https://github.com/apache/cassandra/blob/cassandra-2.1/doc/cql3/CQL.textile

19. Productivity:
!
Use the Java Driver

20. Productivity - Java Driver:
!
• Reference implementation
• Well written, extensive coverage
• open source
https://github.com/datastax/java-driver/

21. Productivity - Java Driver:
!
Existing Spring Users:
Spring Data
Integration
http://projects.spring.io/spring-data-cassandra/

22. Productivity - Java Driver:
!
Guice Users:
“GuicyFig:”
Archaius + Guice
https://stash.safehaus.org/projects/GFIG/repos/main/browse

23. Productivity - Java Driver:
!
Configuration is Similar to
Other DB Drivers
(with caveats**)
http://www.datastax.com/documentation/developer/java-driver/2.1/common/drivers/reference/clusterConfiguration_c.html

24. Productivity - Java Driver - Configuration:
!
Major Difference:
it’s a Cluster!

25. Productivity - Java Driver - Configuration:
!
Two groups of configurations
!
• policies
• connections

26. Productivity - Java Driver - Configuration:
!
Three Policy Types:
• load balancing
• connection
• retry

27. Productivity - Java Driver - Configuration:
!
Connection Options:
• protocol*
• pooling
• socket
*https://github.com/apache/cassandra/blob/cassandra-2.1/doc/native_protocol_v3.spec

28. Productivity - Java Driver:
!
Embrace Asynchronicity
(but use RxJava)
https://github.com/ReactiveX/RxJava

29. Productivity - Java Driver:
!
A note about
User Defined Types (UTDs)

30. Productivity - Java Driver - Using UDTs:
!
Wait.
- serialized as blobs !!?!
- new version already being discussed*
- will be a painful migration path
* https://issues.apache.org/jira/browse/CASSANDRA-7423

31. Productivity:
!
Tools:
DataStax DevCenter
http://www.datastax.com/what-we-offer/products-services/devcenter

32. Productivity:
!
Metrics API
for your own code
https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/metrics/ColumnFamilyMetrics.java
https://dropwizard.github.io/metrics/3.1.0/

33. Productivity - Instrumentation via Metrics API:
!
Run Riemann
locally
http://riemann.io/

35. Productivity:
!
Trace Frequently

36. Productivity - Tracing:
!
Trace per query
via cqlsh
http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/tracing_r.html

37. cqlsh> tracing on;
Now tracing requests.
cqlsh> SELECT doc_version FROM data.documents_by_version
... WHERE application_id = myapp
... AND document_id = foo
... AND chunk_index = 0
... ORDER BY doc_version ASC
... LIMIT 1;
!
doc_version
-------------
65856
!
!
Tracing session: 46211ab0-2702-11e4-9bcf-8d157d448e6b

38. Preparing statement | 18:05:44,845 | 192.168.1.197 | 22337
Enqueuing data request to /192.168.1.204 | 18:05:44,845 | 192.168.1.197 | 22504
Sending message to /192.168.1.204 | 18:05:44,847 | 192.168.1.197 | 24498
Message received from /192.168.1.197 | 18:05:44,854 | 192.168.1.204 | 872
Executing single-partition query on documents_by_version | 18:05:44,888 | 192.168.1.204 | 35183
Acquiring sstable references | 18:05:44,888 | 192.168.1.204 | 35459
Merging memtable tombstones | 18:05:44,889 | 192.168.1.204 | 35675
Key cache hit for sstable 2867 | 18:05:44,889 | 192.168.1.204 | 35792
Seeking to partition beginning in data file | 18:05:44,889 | 192.168.1.204 | 35817
…

39. Preparing statement | 18:05:44,845 | 192.168.1.197 | 22337
Enqueuing data request to /192.168.1.204 | 18:05:44,845 | 192.168.1.197 | 22504
Sending message to /192.168.1.204 | 18:05:44,847 | 192.168.1.197 | 24498
Message received from /192.168.1.197 | 18:05:44,854 | 192.168.1.204 | 872
Executing single-partition query on documents_by_version | 18:05:44,888 | 192.168.1.204 | 35183
Acquiring sstable references | 18:05:44,888 | 192.168.1.204 | 35459
Merging memtable tombstones | 18:05:44,889 | 192.168.1.204 | 35675
Key cache hit for sstable 2867 | 18:05:44,889 | 192.168.1.204 | 35792
Seeking to partition beginning in data file | 18:05:44,889 | 192.168.1.204 | 35817
…

40. …
Merging data from memtables and 8 sstables | 18:05:44,892 | 192.168.1.204 | 38605
Read 1 live and 2667 tombstoned cells | 18:05:54,135 | 192.168.1.204 | 9282428
Enqueuing response to /192.168.1.197 | 18:05:54,136 | 192.168.1.204 | 9283423
Sending message to /192.168.1.197 | 18:05:54,138 | 192.168.1.204 | 9284753
Message received from /192.168.1.204 | 18:05:54,155 | 192.168.1.197 | 9332505
Processing response from /192.168.1.204 | 18:05:54,158 | 192.168.1.197 | 9335372
Request complete | 18:05:54,158 | 192.168.1.197 | 9335592

41. …
Merging data from memtables and 8 sstables | 18:05:44,892 | 192.168.1.204 | 38605
Read 1 live and 2667 tombstoned cells | 18:05:54,135 | 192.168.1.204 | 9282428
Enqueuing response to /192.168.1.197 | 18:05:54,136 | 192.168.1.204 | 9283423
Sending message to /192.168.1.197 | 18:05:54,138 | 192.168.1.204 | 9284753
Message received from /192.168.1.204 | 18:05:54,155 | 192.168.1.197 | 9332505
Processing response from /192.168.1.204 | 18:05:54,158 | 192.168.1.197 | 9335372
Request complete | 18:05:54,158 | 192.168.1.197 | 9335592
!!?!

42. …
Merging data from memtables and 8 sstables | 18:05:44,892 | 192.168.1.204 | 38605
Read 1 live and 2667 tombstoned cells | 18:05:54,135 | 192.168.1.204 | 9282428
Enqueuing response to /192.168.1.197 | 18:05:54,136 | 192.168.1.204 | 9283423
Sending message to /192.168.1.197 | 18:05:54,138 | 192.168.1.204 | 9284753
Message received from /192.168.1.204 | 18:05:54,155 | 192.168.1.197 | 9332505
Processing response from /192.168.1.204 | 18:05:54,158 | 192.168.1.197 | 9335372
Request complete | 18:05:54,158 | 192.168.1.197 | 9335592

43. Productivity - Tracing:
!
Enable traces
in the driver
http://www.datastax.com/documentation/developer/java-driver/2.0/java-driver/tracing_t.html

44. Productivity - Tracing:
!
`nodetool settraceprobability`

45. Productivity - Tracing:
!
…then make sure
you try it again
with a node down!

46. Productivity - Tracing:
!
Final note on tracing:
do it sparingly

47. Productivity:
!
Logging Verbosity
can be changed
dynamically**
!
!
** since 0.4rc1
http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configLoggingLevels_r.html

48. Productivity:
!
nodetool for developers
• cfstats
• cfshistograms
• proxyhistograms

49. Productivity - nodetool - cfstats:
cfstats:
per-table statistics about size
and performance
(single most useful command)

50. Productivity - nodetool - cfhistograms:
cfhistograms:
column count and partition
size vs. latency distribution

51. Productivity - nodetool - proxyhistograms:
proxyhistograms:
performance of inter-cluster
requests

52. Productivity:
!
Running Cassandra
during development

53. Productivity - Running Cassandra:
!
Local Cassandra
• easy to setup
• you control it
• but then you control it!

54. Productivity - Running Cassandra:
!
CCM
• supports multiple versions
• clusters and datacenters
• up/down individual nodes
https://github.com/pcmanus/ccm

55. Productivity - Running Cassandra:
!
Vagrant
• isolated, controlled environment
• configuration mgmt integration
• same CM for production!
http://www.vagrantup.com/

56. server_count = 3
network = '192.168.2.'
first_ip = 10
!
servers = []
seeds = []
cassandra_tokens = []
(0..server_count-1).each do |i|
name = 'node' + (i + 1).to_s
ip = network + (first_ip + i).to_s
seeds << ip
servers << {'name' => name,
'ip' => ip,
'initial_token' => (2**64 / server_count * i) - 2**63}
end

57. server_count = 3
network = '192.168.2.'
first_ip = 10
!
servers = []
seeds = []
cassandra_tokens = []
(0..server_count-1).each do |i|
name = 'node' + (i + 1).to_s
ip = network + (first_ip + i).to_s
seeds << ip
servers << {'name' => name,
'ip' => ip,
'initial_token' => (2**64 / server_count * i) - 2**63}
end

58. server_count = 3
network = '192.168.2.'
first_ip = 10
!
servers = []
seeds = []
cassandra_tokens = []
(0..server_count-1).each do |i|
name = 'node' + (i + 1).to_s
ip = network + (first_ip + i).to_s
seeds << ip
servers << {'name' => name,
'ip' => ip,
'initial_token' => (2**64 / server_count * i) - 2**63}
end

59. chef.json = {
:cassandra => {'cluster_name' => 'VerifyCluster',
'version' => '2.0.8',
'setup_jna' => false,
'max_heap_size' => '512M',
'heap_new_size' => '100M',
'initial_token' => server['initial_token'],
'seeds' => "192.168.2.10",
'listen_address' => server['ip'],
'broadcast_address' => server['ip'],
'rpc_address' => server['ip'],
'conconcurrent_reads' => "2",
'concurrent_writes' => "2",
'memtable_flush_queue_size' => "2",
'compaction_throughput_mb_per_sec' => "8",
'key_cache_size_in_mb' => "4",
'key_cache_save_period' => "0",
'native_transport_min_threads' => "2",
'native_transport_max_threads' => "4"
},
}

60. ENCAPSULATE ENVIRONMENTS

61. Environments:
!
Configuration Management
is Essential

62. Environments:
!
Laptop to Production
with NO
Manual Modifications!

63. TESTING

64. Testing:
Use a Naming Scheme
!
• *UnitTest.java: no external resources
• *ITest.java: uses external resources
• *PITest.java: safely parallel “ITest”

65. Testing:
Tip:
wildcards on the CLI
are not
a naming schema.

66. Testing:
Group tests
into
logical units
(“suites”)

67. Testing - Suites:
Benefits of Suites:
• share test data
• share Cassandra instance(s)
• build profiles

68. <profile>
<id>short</id>
<properties>
<env>default</env>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<groups>unit,short</groups>
<useFile>false</useFile>
<systemPropertyVariables>
<cassandra.version>${cassandra.version}</cassandra.version>
<ipprefix>${ipprefix}</ipprefix>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
</profile>

69. <profile>
<id>short</id>
<properties>
<env>default</env>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<groups>unit,short</groups>
<useFile>false</useFile>
<systemPropertyVariables>
<cassandra.version>${cassandra.version}</cassandra.version>
<ipprefix>${ipprefix}</ipprefix>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
</profile>

70. Testing - Suites:
Using annotations
for suites in code

72. Testing:
!
Use Mocks
where possible

73. Testing:
!
Unit Integration Testing

74. Testing:
Verify Assumptions:
test failure scenarios
explicitly

75. Testing - Integration:
Runtime Integrations:
• local
• in-process
• forked-process

76. Testing - Integration - Runtime:
EmbeddedCassandra

77. Testing - Integration - Runtime:
ProcessBuilder to
fork Cassandra(s)

78. Testing - Integration - Runtime:
CCMBridge:
delegate to CCM
https://github.com/datastax/java-driver/blob/2.1/driver-core/src/test/java/com/datastax/driver/core/CCMBridge.java

79. Testing - Integration - Runtime:
Vagrant:
delegate to vagrant cli

80. Testing - Integration:
Best Practice:
Jenkins should be able to
manage your cluster

81. Testing - Integration - Best Practices:
Vagrant vs. CCMBridge?
!
• choice of style, really
• developer integration with CM
• what else is in the architecture?

82. Testing:
!
Load Testing Goals
• reproducible metrics
• catch regressions
• test to breakage point

83. Testing - Load Testing:
!
Stress.java
(lot’s of changes recently)

84. Testing - Load Testing:
!
CassandraJMeter
https://github.com/Netflix/CassJMeter

85. Testing - Load Testing:
!
Workload recording
and playback coming soon
https://issues.apache.org/jira/browse/CASSANDRA-6572

86. Testing:
!
Primary testing goal:
Don’t let
cluster behavior
surprise you.

87. Summary:
• Go slowly with bite sized chunks
• Segment your tests and use build profiles
• Monitor and Instrument
• Use reference implementation drivers
• Control your environments
• Verify any assumptions about failures

88. Thanks.
!

89. Nate McCall
@zznate
!
Co-Founder & Sr. Technical Consultant
www.thelastpickle.com
#CassandraSummit