SlideShare a Scribd company logo
1 of 203
Troubleshooting Cassandra
nodetool & system.log deep dive
J.B. Langston, Senior Support Engineer
Company Confidential© 2014 DataStax, All Rights Reserved.
Agenda
2
1 Things to keep in mind
2 Useful Linux & Java tools
3 Useful nodetool commands
4 Things to look for in system.log
Company Confidential© 2014 DataStax, All Rights Reserved.
Troubleshooting Process
3
1 Ask what changed
2 Determine which nodes have problems
3 Find and understand errors
4 Examine bottlenecks
5 Determine root cause
6 Take corrective action
Company Confidential© 2014 DataStax, All Rights Reserved.
Troubleshooting Process
4
1 Ask what changed
2 Determine which nodes have problems
3 Find and understand errors
4 Examine bottlenecks
5 Share relevant information
6 Ask for help
Company Confidential© 2014 DataStax, All Rights Reserved. 5
• Does it work in another
environment?
• Did you upgrade?
• Cassandra
• Kernel
• JVM
• Driver
• Change one thing at a time!
What changed?
• Did it work before?
• What did you change?
• Settings
• Application Code
• Read/Write Load
• Data Volume
• Hardware
• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 6
• Background
• Flushes
• Compactions
• Garbage collections
• Gossip
• Hinted Handoff
• Read Repair
Cassandra Processes
• Startup
• Foreground
• Coordinating requests
• Local reads
• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk
• Disk space
• I/O bandwidth
• Network
• OS Resources/Limits
• File Handles
• Processes
• Mapped Memory
System Resources
• CPU
• Single Core utilization
• Multi Core utilization
• Memory
• Heap space
• Off-heap space
• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…
top CPU utilization and memory use per process
top -H CPU utilization per thread, memory use is still per process
df Free disk space
iostat -x I/O bandwidth utilization
free -m Memory and cache usage
netstat -an Network connections established
iftop Network bandwidth utilization
sar All (most) of the above, with history!
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Command What it tells you…
jstack -l Status and stack trace of each thread
jmap -histo Types of objects on the heap (optionally only live objects)
jmap -heap Size and usage of each java heap generation
jstat -gccause Causes of gc activity
jmap -dump Take a heap dump for further analysis
MemoryAnalyzer Post-mortem heap-dump analysis
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…
status / ring Overall cluster status
info Status, memory usage, and caches for a single node
tpstats Statistics about each thread pool on a single node
cfstats Summary statistics for all tables and keyspaces on a single node
cfhistograms Detailed statistics for a specific table on the local node
proxyhistograms Latency statistics for requests coordinated the local node
compactionstats Compactions pending and in progress
netstats Network activity: streams, read repair, and in-flight commands
Company Confidential© 2014 DataStax, All Rights Reserved. 11
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 12
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 13
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 14
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 15
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 16
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 17
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 18
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 19
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 20
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 21
nodetool status
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b
DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b
UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a
UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a
UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 22
nodetool ring
Note: Ownership information does not include topology; for complete information,
specify a keyspace
Datacenter: us-east
==========
Address Rack Status State Load Owns Token
-3074457345618258603
54.174.19.98 1b Down Normal 22.82 MB 16.67% -9223372036854775808
54.174.245.247 1b Up Normal 22.68 MB 16.67% -6148914691236517206
54.173.171.164 1b Up Normal 22.78 MB 16.67% -3074457345618258603
Datacenter: us-west
==========
Address Rack Status State Load Owns Token
6148914691236517205
54.153.39.203 1a Up Normal 22.67 MB 16.67% 0
54.153.107.100 1a Up Normal 22.72 MB 16.67% 3074457345618258602
54.153.108.157 1a Up Normal 22.79 MB 16.67% 6148914691236517205
Company Confidential© 2014 DataStax, All Rights Reserved. 23
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 24
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 25
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 26
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 27
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 28
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 29
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 30
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 31
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 32
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 33
nodetool info
Token : -6148914691236517206
ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 22.68 MB
Generation No : 1426523950
Uptime (seconds) : 1557
Heap Memory (MB) : 270.85 / 1842.00
Off Heap Memory (MB) : 0.11
Data Center : us-east
Rack : 1b
Exceptions : 0
Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests,
0.813 recent hit rate, 14400 save period in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 34
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 35
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 36
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 37
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 38
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 39
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 40
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 41
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 42
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 43
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 44
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 45
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 46
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 47
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 48
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 49
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 50
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 51
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 52
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 53
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 54
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 2 20487 0 0
RequestResponseStage 0 0 2787111 0 0
MutationStage 32 8713 1735129 0 0
ReadRepairStage 0 0 2641 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 34421 0 0
CacheCleanupExecutor 0 0 1 0 0
MigrationStage 0 0 1 0 0
MemoryMeter 1 1 71 0 0
FlushWriter 1 4 73 0 5
ValidationExecutor 0 0 42 0 0
InternalResponseStage 0 0 188 0 0
AntiEntropyStage 0 0 610 0 0
MemtablePostFlusher 1 4 358 0 0
MiscStage 0 0 43 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
CompactionExecutor 2 5 125 0 0
AntiEntropySessions 1 1 13 0 0
HintedHandoff 2 4 34 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 7
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 42993
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 55
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 56
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 57
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 58
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 59
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 60
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 61
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 62
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 63
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 64
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 65
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 66
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 67
nodetool cfstats
Keyspace: Keyspace1
Read Count: 282781
Read Latency: 1.0155043372786716 ms.
Write Count: 2700184
Write Latency: 0.07415620861393149 ms.
Pending Tasks: 0
Table: Standard1
SSTable count: 5
Space used (live), bytes: 656644536
Space used (total), bytes: 656644536
Off heap memory used (total), bytes: 4221251
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2182272
Memtable cell count: 215155
Memtable data size, bytes: 67222188
Memtable switch count: 72
Local read count: 282781
Local read latency: 1.016 ms
Local write count: 2700943
Local write latency: 0.074 ms
Pending tasks: 0
Bloom filter false positives: 379
Bloom filter false ratio: 0.00097
Bloom filter space used, bytes: 3897360
Bloom filter off heap memory used, bytes: 3897320
Index summary off heap memory used, bytes: 323931
Compression metadata off heap memory used, bytes: 0
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 310
Average live cells per slice (last five minutes): 5.0
Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 68
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 69
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 70
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 71
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 72
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 73
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 74
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 75
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 76
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 77
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 78
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 79
nodetool cfhistograms
Keyspace1/Standard1 histograms
SSTables per Read
1 sstables: 14250
2 sstables: 42908
3 sstables: 97878
4 sstables: 126087
5 sstables: 745
6 sstables: 295
7 sstables: 122
8 sstables: 448
10 sstables: 37
12 sstables: 10
14 sstables: 1
Partition Size (bytes)
1131752 bytes: 1093
1358102 bytes: 479
1629722 bytes: 364
1955666 bytes: 343
2346799 bytes: 73
2816159 bytes: 134
3379391 bytes: 7
4055269 bytes: 0
4866323 bytes: 2
Cell Count per Partition
103 cells: 112
124 cells: 5130
149 cells: 2458
179 cells: 1704
215 cells: 1460
258 cells: 823
310 cells: 1327
372 cells: 1331
446 cells: 49295
535 cells: 9091
642 cells: 11892
770 cells: 3908
924 cells: 14532
1109 cells: 20898
1331 cells: 8491
1597 cells: 8140
1916 cells: 4010
2299 cells: 19476
2759 cells: 15112
3311 cells: 10812
3973 cells: 12763
4768 cells: 18597
5722 cells: 5765
6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 80
nodetool cfhistograms
Write Latency (microseconds)
4 us: 83
5 us: 9004
6 us: 73288
7 us: 211239
8 us: 321305
10 us: 607511
12 us: 438101
14 us: 440503
17 us: 459036
20 us: 211763
24 us: 97240
29 us: 43564
35 us: 29579
42 us: 22575
50 us: 10442
60 us: 4842
72 us: 3566
86 us: 4323
103 us: 2038
124 us: 6192
149 us: 6850
179 us: 2657
215 us: 4536
258 us: 1839
Read Latency (microseconds)
42 us: 29
50 us: 132
60 us: 1225
72 us: 4791
86 us: 13031
103 us: 18526
124 us: 21151
149 us: 21085
179 us: 16528
215 us: 19259
258 us: 39965
310 us: 55015
372 us: 29344
446 us: 11347
535 us: 5692
642 us: 5395
770 us: 3550
924 us: 2649
1109 us: 1379
1331 us: 1084
1597 us: 1266
1916 us: 1492
2299 us: 2032
2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 81
nodetool cfhistograms
Write Latency (microseconds)
4 us: 83
5 us: 9004
6 us: 73288
7 us: 211239
8 us: 321305
10 us: 607511
12 us: 438101
14 us: 440503
17 us: 459036
20 us: 211763
24 us: 97240
29 us: 43564
35 us: 29579
42 us: 22575
50 us: 10442
60 us: 4842
72 us: 3566
86 us: 4323
103 us: 2038
124 us: 6192
149 us: 6850
179 us: 2657
215 us: 4536
258 us: 1839
Read Latency (microseconds)
42 us: 29
50 us: 132
60 us: 1225
72 us: 4791
86 us: 13031
103 us: 18526
124 us: 21151
149 us: 21085
179 us: 16528
215 us: 19259
258 us: 39965
310 us: 55015
372 us: 29344
446 us: 11347
535 us: 5692
642 us: 5395
770 us: 3550
924 us: 2649
1109 us: 1379
1331 us: 1084
1597 us: 1266
1916 us: 1492
2299 us: 2032
2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 82
nodetool cfhistograms
Write Latency (microseconds)
4 us: 83
5 us: 9004
6 us: 73288
7 us: 211239
8 us: 321305
10 us: 607511
12 us: 438101
14 us: 440503
17 us: 459036
20 us: 211763
24 us: 97240
29 us: 43564
35 us: 29579
42 us: 22575
50 us: 10442
60 us: 4842
72 us: 3566
86 us: 4323
103 us: 2038
124 us: 6192
149 us: 6850
179 us: 2657
215 us: 4536
258 us: 1839
Read Latency (microseconds)
42 us: 29
50 us: 132
60 us: 1225
72 us: 4791
86 us: 13031
103 us: 18526
124 us: 21151
149 us: 21085
179 us: 16528
215 us: 19259
258 us: 39965
310 us: 55015
372 us: 29344
446 us: 11347
535 us: 5692
642 us: 5395
770 us: 3550
924 us: 2649
1109 us: 1379
1331 us: 1084
1597 us: 1266
1916 us: 1492
2299 us: 2032
2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 83
nodetool cfhistograms
Write Latency (microseconds)
4 us: 83
5 us: 9004
6 us: 73288
7 us: 211239
8 us: 321305
10 us: 607511
12 us: 438101
14 us: 440503
17 us: 459036
20 us: 211763
24 us: 97240
29 us: 43564
35 us: 29579
42 us: 22575
50 us: 10442
60 us: 4842
72 us: 3566
86 us: 4323
103 us: 2038
124 us: 6192
149 us: 6850
179 us: 2657
215 us: 4536
258 us: 1839
Read Latency (microseconds)
42 us: 29
50 us: 132
60 us: 1225
72 us: 4791
86 us: 13031
103 us: 18526
124 us: 21151
149 us: 21085
179 us: 16528
215 us: 19259
258 us: 39965
310 us: 55015
372 us: 29344
446 us: 11347
535 us: 5692
642 us: 5395
770 us: 3550
924 us: 2649
1109 us: 1379
1331 us: 1084
1597 us: 1266
1916 us: 1492
2299 us: 2032
2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 84
nodetool cfhistograms
Write Latency (microseconds)
4 us: 83
5 us: 9004
6 us: 73288
7 us: 211239
8 us: 321305
10 us: 607511
12 us: 438101
14 us: 440503
17 us: 459036
20 us: 211763
24 us: 97240
29 us: 43564
35 us: 29579
42 us: 22575
50 us: 10442
60 us: 4842
72 us: 3566
86 us: 4323
103 us: 2038
124 us: 6192
149 us: 6850
179 us: 2657
215 us: 4536
258 us: 1839
Read Latency (microseconds)
42 us: 29
50 us: 132
60 us: 1225
72 us: 4791
86 us: 13031
103 us: 18526
124 us: 21151
149 us: 21085
179 us: 16528
215 us: 19259
258 us: 39965
310 us: 55015
372 us: 29344
446 us: 11347
535 us: 5692
642 us: 5395
770 us: 3550
924 us: 2649
1109 us: 1379
1331 us: 1084
1597 us: 1266
1916 us: 1492
2299 us: 2032
2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 85
nodetool proxyhistograms
Read Latency (microseconds)
124 us: 18
149 us: 445
179 us: 5866
215 us: 14541
258 us: 19123
310 us: 20941
372 us: 32847
446 us: 37464
535 us: 26451
642 us: 25291
770 us: 37870
924 us: 35865
1109 us: 30432
1331 us: 22929
1597 us: 17592
1916 us: 12895
2299 us: 9760
2759 us: 7887
3311 us: 6024
3973 us: 4751
4768 us: 3691
5722 us: 3038
6866 us: 2453
8239 us: 1991
Write Latency (microseconds)
29 us: 4
35 us: 60
42 us: 331
50 us: 471
60 us: 1334
72 us: 2856
86 us: 4661
103 us: 65741
124 us: 37999
149 us: 42697
179 us: 45598
215 us: 30515
258 us: 23481
310 us: 21831
372 us: 22741
446 us: 26045
535 us: 33045
642 us: 43533
770 us: 51021
924 us: 57496
1109 us: 61548
1331 us: 60741
1597 us: 57538
1916 us: 52255
Company Confidential© 2014 DataStax, All Rights Reserved. 86
nodetool compactionstats
pending tasks: 7
compaction type keyspace table completed total unit progress
Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66%
Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68%
Active compaction remaining time : 0h00m15s
pending tasks: 1
compaction type keyspace table completed total unit progress
Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47%
Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 87
nodetool compactionstats
pending tasks: 7
compaction type keyspace table completed total unit progress
Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66%
Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68%
Active compaction remaining time : 0h00m15s
pending tasks: 1
compaction type keyspace table completed total unit progress
Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47%
Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 88
nodetool compactionstats
pending tasks: 7
compaction type keyspace table completed total unit progress
Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66%
Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68%
Active compaction remaining time : 0h00m15s
pending tasks: 1
compaction type keyspace table completed total unit progress
Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47%
Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 89
nodetool compactionstats
pending tasks: 7
compaction type keyspace table completed total unit progress
Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66%
Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68%
Active compaction remaining time : 0h00m15s
pending tasks: 1
compaction type keyspace table completed total unit progress
Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47%
Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 90
nodetool compactionstats
pending tasks: 7
compaction type keyspace table completed total unit progress
Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66%
Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68%
Active compaction remaining time : 0h00m15s
pending tasks: 1
compaction type keyspace table completed total unit progress
Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47%
Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 91
nodetool compactionstats
pending tasks: 7
compaction type keyspace table completed total unit progress
Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66%
Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68%
Active compaction remaining time : 0h00m15s
pending tasks: 1
compaction type keyspace table completed total unit progress
Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47%
Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 92
nodetool compactionstats
pending tasks: 7
compaction type keyspace table completed total unit progress
Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66%
Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68%
Active compaction remaining time : 0h00m15s
pending tasks: 1
compaction type keyspace table completed total unit progress
Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47%
Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 93
nodetool compactionstats
pending tasks: 7
compaction type keyspace table completed total unit progress
Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66%
Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68%
Active compaction remaining time : 0h00m15s
pending tasks: 1
compaction type keyspace table completed total unit progress
Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47%
Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 94
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 95
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 96
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 97
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 98
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 99
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 100
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 101
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 102
nodetool compactionhistory
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out
5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543
654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892
59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0
4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264
68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472
f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816
3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550
e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517
54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552
c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778
1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676
rows_merged
{1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
Company Confidential© 2014 DataStax, All Rights Reserved. 103
nodetool netstats
Mode: NORMAL
Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.19.98
Receiving 6 files, 117949006 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98
Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.245.247
Receiving 4 files, 93304584 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247
Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30
/54.153.39.203 (using /172.31.10.65)
Receiving 3 files, 49959102 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics:
Attempted: 39576
Mismatch (Blocking): 0
Mismatch (Background): 746
Pool Name Active Pending Completed
Commands n/a 58 2545817
Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 104
nodetool netstats
Mode: NORMAL
Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.19.98
Receiving 6 files, 117949006 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98
Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.245.247
Receiving 4 files, 93304584 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247
Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30
/54.153.39.203 (using /172.31.10.65)
Receiving 3 files, 49959102 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics:
Attempted: 39576
Mismatch (Blocking): 0
Mismatch (Background): 746
Pool Name Active Pending Completed
Commands n/a 58 2545817
Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 105
nodetool netstats
Mode: NORMAL
Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.19.98
Receiving 6 files, 117949006 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98
Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.245.247
Receiving 4 files, 93304584 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247
Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30
/54.153.39.203 (using /172.31.10.65)
Receiving 3 files, 49959102 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics:
Attempted: 39576
Mismatch (Blocking): 0
Mismatch (Background): 746
Pool Name Active Pending Completed
Commands n/a 58 2545817
Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 106
nodetool netstats
Mode: NORMAL
Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.19.98
Receiving 6 files, 117949006 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98
Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.245.247
Receiving 4 files, 93304584 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247
Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30
/54.153.39.203 (using /172.31.10.65)
Receiving 3 files, 49959102 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics:
Attempted: 39576
Mismatch (Blocking): 0
Mismatch (Background): 746
Pool Name Active Pending Completed
Commands n/a 58 2545817
Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 107
nodetool netstats
Mode: NORMAL
Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.19.98
Receiving 6 files, 117949006 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98
Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.245.247
Receiving 4 files, 93304584 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247
Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30
/54.153.39.203 (using /172.31.10.65)
Receiving 3 files, 49959102 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics:
Attempted: 39576
Mismatch (Blocking): 0
Mismatch (Background): 746
Pool Name Active Pending Completed
Commands n/a 58 2545817
Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 108
nodetool netstats
Mode: NORMAL
Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.19.98
Receiving 6 files, 117949006 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98
Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.245.247
Receiving 4 files, 93304584 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247
Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30
/54.153.39.203 (using /172.31.10.65)
Receiving 3 files, 49959102 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics:
Attempted: 39576
Mismatch (Blocking): 0
Mismatch (Background): 746
Pool Name Active Pending Completed
Commands n/a 58 2545817
Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 109
nodetool netstats
Mode: NORMAL
Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.19.98
Receiving 6 files, 117949006 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98
Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.245.247
Receiving 4 files, 93304584 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247
Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30
/54.153.39.203 (using /172.31.10.65)
Receiving 3 files, 49959102 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics:
Attempted: 39576
Mismatch (Blocking): 0
Mismatch (Background): 746
Pool Name Active Pending Completed
Commands n/a 58 2545817
Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 110
nodetool netstats
Mode: NORMAL
Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.19.98
Receiving 6 files, 117949006 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98
Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30
/54.174.245.247
Receiving 4 files, 93304584 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247
Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30
/54.153.39.203 (using /172.31.10.65)
Receiving 3 files, 49959102 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203
Sending 2 files, 47709526 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics:
Attempted: 39576
Mismatch (Blocking): 0
Mismatch (Background): 746
Pool Name Active Pending Completed
Commands n/a 58 2545817
Responses n/a 0 2833081
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)
Cassandra Troubleshooting (for 2.0 and earlier)

More Related Content

What's hot

DataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax Academy
 
Advanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXAdvanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXzznate
 
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...DataStax
 
Hadoop - Lessons Learned
Hadoop - Lessons LearnedHadoop - Lessons Learned
Hadoop - Lessons Learnedtcurdt
 
Postgres in Amazon RDS
Postgres in Amazon RDSPostgres in Amazon RDS
Postgres in Amazon RDSDenish Patel
 
DataStax | Advanced DSE Analytics Client Configuration (Jacek Lewandowski) | ...
DataStax | Advanced DSE Analytics Client Configuration (Jacek Lewandowski) | ...DataStax | Advanced DSE Analytics Client Configuration (Jacek Lewandowski) | ...
DataStax | Advanced DSE Analytics Client Configuration (Jacek Lewandowski) | ...DataStax
 
DataStax | DSE: Bring Your Own Spark (with Enterprise Security) (Artem Aliev)...
DataStax | DSE: Bring Your Own Spark (with Enterprise Security) (Artem Aliev)...DataStax | DSE: Bring Your Own Spark (with Enterprise Security) (Artem Aliev)...
DataStax | DSE: Bring Your Own Spark (with Enterprise Security) (Artem Aliev)...DataStax
 
Building Apache Cassandra clusters for massive scale
Building Apache Cassandra clusters for massive scaleBuilding Apache Cassandra clusters for massive scale
Building Apache Cassandra clusters for massive scaleAlex Thompson
 
Improving HDFS Availability with Hadoop RPC Quality of Service
Improving HDFS Availability with Hadoop RPC Quality of ServiceImproving HDFS Availability with Hadoop RPC Quality of Service
Improving HDFS Availability with Hadoop RPC Quality of ServiceMing Ma
 
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...DataStax
 
Percona Toolkit for Effective MySQL Administration
Percona Toolkit for Effective MySQL AdministrationPercona Toolkit for Effective MySQL Administration
Percona Toolkit for Effective MySQL AdministrationMydbops
 
Cassandra summit 2013 how not to use cassandra
Cassandra summit 2013  how not to use cassandraCassandra summit 2013  how not to use cassandra
Cassandra summit 2013 how not to use cassandraAxel Liljencrantz
 
Profiling the logwriter and database writer
Profiling the logwriter and database writerProfiling the logwriter and database writer
Profiling the logwriter and database writerKyle Hailey
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTanel Poder
 
AddThis: Scaling Cassandra up and down into containers with ZFS
AddThis: Scaling Cassandra up and down into containers with ZFSAddThis: Scaling Cassandra up and down into containers with ZFS
AddThis: Scaling Cassandra up and down into containers with ZFSDataStax Academy
 
Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...
Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...
Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...Ceph Community
 
Apache Cassandra multi-datacenter essentials
Apache Cassandra multi-datacenter essentialsApache Cassandra multi-datacenter essentials
Apache Cassandra multi-datacenter essentialsJulien Anguenot
 
Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009) Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009) PostgreSQL Experts, Inc.
 
TechTalk v2.0 - Performance tuning Cassandra + AWS
TechTalk v2.0 - Performance tuning Cassandra + AWSTechTalk v2.0 - Performance tuning Cassandra + AWS
TechTalk v2.0 - Performance tuning Cassandra + AWSPythian
 

What's hot (20)

DataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The Sequel
 
Advanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXAdvanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMX
 
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
Lessons Learned From Running 1800 Clusters (Brooke Jensen, Instaclustr) | Cas...
 
Hadoop - Lessons Learned
Hadoop - Lessons LearnedHadoop - Lessons Learned
Hadoop - Lessons Learned
 
Postgres in Amazon RDS
Postgres in Amazon RDSPostgres in Amazon RDS
Postgres in Amazon RDS
 
DataStax | Advanced DSE Analytics Client Configuration (Jacek Lewandowski) | ...
DataStax | Advanced DSE Analytics Client Configuration (Jacek Lewandowski) | ...DataStax | Advanced DSE Analytics Client Configuration (Jacek Lewandowski) | ...
DataStax | Advanced DSE Analytics Client Configuration (Jacek Lewandowski) | ...
 
DataStax | DSE: Bring Your Own Spark (with Enterprise Security) (Artem Aliev)...
DataStax | DSE: Bring Your Own Spark (with Enterprise Security) (Artem Aliev)...DataStax | DSE: Bring Your Own Spark (with Enterprise Security) (Artem Aliev)...
DataStax | DSE: Bring Your Own Spark (with Enterprise Security) (Artem Aliev)...
 
Building Apache Cassandra clusters for massive scale
Building Apache Cassandra clusters for massive scaleBuilding Apache Cassandra clusters for massive scale
Building Apache Cassandra clusters for massive scale
 
Improving HDFS Availability with Hadoop RPC Quality of Service
Improving HDFS Availability with Hadoop RPC Quality of ServiceImproving HDFS Availability with Hadoop RPC Quality of Service
Improving HDFS Availability with Hadoop RPC Quality of Service
 
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
 
Percona Toolkit for Effective MySQL Administration
Percona Toolkit for Effective MySQL AdministrationPercona Toolkit for Effective MySQL Administration
Percona Toolkit for Effective MySQL Administration
 
Cassandra summit 2013 how not to use cassandra
Cassandra summit 2013  how not to use cassandraCassandra summit 2013  how not to use cassandra
Cassandra summit 2013 how not to use cassandra
 
Profiling the logwriter and database writer
Profiling the logwriter and database writerProfiling the logwriter and database writer
Profiling the logwriter and database writer
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
 
Advanced Operations
Advanced OperationsAdvanced Operations
Advanced Operations
 
AddThis: Scaling Cassandra up and down into containers with ZFS
AddThis: Scaling Cassandra up and down into containers with ZFSAddThis: Scaling Cassandra up and down into containers with ZFS
AddThis: Scaling Cassandra up and down into containers with ZFS
 
Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...
Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...
Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...
 
Apache Cassandra multi-datacenter essentials
Apache Cassandra multi-datacenter essentialsApache Cassandra multi-datacenter essentials
Apache Cassandra multi-datacenter essentials
 
Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009) Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009)
 
TechTalk v2.0 - Performance tuning Cassandra + AWS
TechTalk v2.0 - Performance tuning Cassandra + AWSTechTalk v2.0 - Performance tuning Cassandra + AWS
TechTalk v2.0 - Performance tuning Cassandra + AWS
 

Viewers also liked

Stateless authentication for microservices applications - JavaLand 2015
Stateless authentication for microservices applications -  JavaLand 2015Stateless authentication for microservices applications -  JavaLand 2015
Stateless authentication for microservices applications - JavaLand 2015Alvaro Sanchez-Mariscal
 
NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?Eberhard Wolff
 
JavaLand - Integration Testing How-to
JavaLand - Integration Testing How-toJavaLand - Integration Testing How-to
JavaLand - Integration Testing How-toNicolas Fränkel
 
Legacy lambda code
Legacy lambda codeLegacy lambda code
Legacy lambda codePeter Lawrey
 
Cassandra 3.0 advanced preview
Cassandra 3.0 advanced previewCassandra 3.0 advanced preview
Cassandra 3.0 advanced previewPatrick McFadin
 
C* Summit 2013: How Not to Use Cassandra by Axel Liljencrantz
C* Summit 2013: How Not to Use Cassandra by Axel LiljencrantzC* Summit 2013: How Not to Use Cassandra by Axel Liljencrantz
C* Summit 2013: How Not to Use Cassandra by Axel LiljencrantzDataStax Academy
 
Cassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSCassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSAdrian Cockcroft
 
Anwendungsfälle für Elasticsearch JavaLand 2015
Anwendungsfälle für Elasticsearch JavaLand 2015Anwendungsfälle für Elasticsearch JavaLand 2015
Anwendungsfälle für Elasticsearch JavaLand 2015Florian Hopf
 
RSA暗号運用でやってはいけない n のこと #ssmjp
RSA暗号運用でやってはいけない n のこと #ssmjpRSA暗号運用でやってはいけない n のこと #ssmjp
RSA暗号運用でやってはいけない n のこと #ssmjpsonickun
 
Cassandra for Sysadmins
Cassandra for SysadminsCassandra for Sysadmins
Cassandra for SysadminsNathan Milford
 

Viewers also liked (13)

Stateless authentication for microservices applications - JavaLand 2015
Stateless authentication for microservices applications -  JavaLand 2015Stateless authentication for microservices applications -  JavaLand 2015
Stateless authentication for microservices applications - JavaLand 2015
 
NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?
 
JavaLand - Integration Testing How-to
JavaLand - Integration Testing How-toJavaLand - Integration Testing How-to
JavaLand - Integration Testing How-to
 
Javaland keynote final
Javaland keynote finalJavaland keynote final
Javaland keynote final
 
Legacy lambda code
Legacy lambda codeLegacy lambda code
Legacy lambda code
 
Cassandra 3.0 advanced preview
Cassandra 3.0 advanced previewCassandra 3.0 advanced preview
Cassandra 3.0 advanced preview
 
Cassandra Metrics
Cassandra MetricsCassandra Metrics
Cassandra Metrics
 
C* Summit 2013: How Not to Use Cassandra by Axel Liljencrantz
C* Summit 2013: How Not to Use Cassandra by Axel LiljencrantzC* Summit 2013: How Not to Use Cassandra by Axel Liljencrantz
C* Summit 2013: How Not to Use Cassandra by Axel Liljencrantz
 
Cassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSCassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWS
 
Anwendungsfälle für Elasticsearch JavaLand 2015
Anwendungsfälle für Elasticsearch JavaLand 2015Anwendungsfälle für Elasticsearch JavaLand 2015
Anwendungsfälle für Elasticsearch JavaLand 2015
 
Cassandra+Hadoop
Cassandra+HadoopCassandra+Hadoop
Cassandra+Hadoop
 
RSA暗号運用でやってはいけない n のこと #ssmjp
RSA暗号運用でやってはいけない n のこと #ssmjpRSA暗号運用でやってはいけない n のこと #ssmjp
RSA暗号運用でやってはいけない n のこと #ssmjp
 
Cassandra for Sysadmins
Cassandra for SysadminsCassandra for Sysadmins
Cassandra for Sysadmins
 

Similar to Cassandra Troubleshooting (for 2.0 and earlier)

Webinar: Untethering Compute from Storage
Webinar: Untethering Compute from StorageWebinar: Untethering Compute from Storage
Webinar: Untethering Compute from StorageAvere Systems
 
20150918 klug el performance tuning-v1.4
20150918 klug el performance tuning-v1.420150918 klug el performance tuning-v1.4
20150918 klug el performance tuning-v1.4Jinkoo Han
 
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016DataStax
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceBrendan Gregg
 
Practice and challenges from building IaaS
Practice and challenges from building IaaSPractice and challenges from building IaaS
Practice and challenges from building IaaSShawn Zhu
 
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringOSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringNETWAYS
 
The Data Center and Hadoop
The Data Center and HadoopThe Data Center and Hadoop
The Data Center and HadoopDataWorks Summit
 
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...Ontico
 
Virtualization & Network Connectivity
Virtualization & Network Connectivity Virtualization & Network Connectivity
Virtualization & Network Connectivity itplant
 
TGorman Collab16 UnixTools 20160411.pdf
TGorman Collab16 UnixTools 20160411.pdfTGorman Collab16 UnixTools 20160411.pdf
TGorman Collab16 UnixTools 20160411.pdfTricantinoLopezPerez
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceBrendan Gregg
 
Analyzing OS X Systems Performance with the USE Method
Analyzing OS X Systems Performance with the USE MethodAnalyzing OS X Systems Performance with the USE Method
Analyzing OS X Systems Performance with the USE MethodBrendan Gregg
 
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全維泰 蔡
 
OSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
OSDC 2015: Georg Schönberger | Linux Performance Profiling and MonitoringOSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
OSDC 2015: Georg Schönberger | Linux Performance Profiling and MonitoringNETWAYS
 
Linux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringLinux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringGeorg Schönberger
 
Loadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkitLoadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkitFrederic Descamps
 
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...Anne Nicolas
 
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...Orgad Kimchi
 

Similar to Cassandra Troubleshooting (for 2.0 and earlier) (20)

Webinar: Untethering Compute from Storage
Webinar: Untethering Compute from StorageWebinar: Untethering Compute from Storage
Webinar: Untethering Compute from Storage
 
20150918 klug el performance tuning-v1.4
20150918 klug el performance tuning-v1.420150918 klug el performance tuning-v1.4
20150918 klug el performance tuning-v1.4
 
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
 
Practice and challenges from building IaaS
Practice and challenges from building IaaSPractice and challenges from building IaaS
Practice and challenges from building IaaS
 
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringOSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
 
Performance Whackamole (short version)
Performance Whackamole (short version)Performance Whackamole (short version)
Performance Whackamole (short version)
 
The Data Center and Hadoop
The Data Center and HadoopThe Data Center and Hadoop
The Data Center and Hadoop
 
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
 
Virtualization & Network Connectivity
Virtualization & Network Connectivity Virtualization & Network Connectivity
Virtualization & Network Connectivity
 
TGorman Collab16 UnixTools 20160411.pdf
TGorman Collab16 UnixTools 20160411.pdfTGorman Collab16 UnixTools 20160411.pdf
TGorman Collab16 UnixTools 20160411.pdf
 
test
testtest
test
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
 
Analyzing OS X Systems Performance with the USE Method
Analyzing OS X Systems Performance with the USE MethodAnalyzing OS X Systems Performance with the USE Method
Analyzing OS X Systems Performance with the USE Method
 
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
 
OSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
OSDC 2015: Georg Schönberger | Linux Performance Profiling and MonitoringOSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
OSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
 
Linux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringLinux Performance Profiling and Monitoring
Linux Performance Profiling and Monitoring
 
Loadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkitLoadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkit
 
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
 
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
 

Recently uploaded

VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Developmentvyaparkranti
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaHanief Utama
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEEVICTOR MAESTRE RAMIREZ
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024StefanoLambiase
 
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfComparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfDrew Moseley
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesŁukasz Chruściel
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl
 
Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...Velvetech LLC
 
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfA healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfMarharyta Nedzelska
 
Introduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdfIntroduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdfFerryKemperman
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesPhilip Schwarz
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmSujith Sukumaran
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odishasmiwainfosol
 
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfExploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfkalichargn70th171
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa
 
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureImplementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureDinusha Kumarasiri
 

Recently uploaded (20)

VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Development
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
 
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfComparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdf
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New Features
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
 
Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...
 
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfA healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdf
 
Introduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdfIntroduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdf
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a series
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
 
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfExploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
 
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureImplementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
 

Cassandra Troubleshooting (for 2.0 and earlier)

  • 1. Troubleshooting Cassandra nodetool & system.log deep dive J.B. Langston, Senior Support Engineer
  • 2. Company Confidential© 2014 DataStax, All Rights Reserved. Agenda 2 1 Things to keep in mind 2 Useful Linux & Java tools 3 Useful nodetool commands 4 Things to look for in system.log
  • 3. Company Confidential© 2014 DataStax, All Rights Reserved. Troubleshooting Process 3 1 Ask what changed 2 Determine which nodes have problems 3 Find and understand errors 4 Examine bottlenecks 5 Determine root cause 6 Take corrective action
  • 4. Company Confidential© 2014 DataStax, All Rights Reserved. Troubleshooting Process 4 1 Ask what changed 2 Determine which nodes have problems 3 Find and understand errors 4 Examine bottlenecks 5 Share relevant information 6 Ask for help
  • 5. Company Confidential© 2014 DataStax, All Rights Reserved. 5 • Does it work in another environment? • Did you upgrade? • Cassandra • Kernel • JVM • Driver • Change one thing at a time! What changed? • Did it work before? • What did you change? • Settings • Application Code • Read/Write Load • Data Volume • Hardware • Network
  • 6. Company Confidential© 2014 DataStax, All Rights Reserved. 6 • Background • Flushes • Compactions • Garbage collections • Gossip • Hinted Handoff • Read Repair Cassandra Processes • Startup • Foreground • Coordinating requests • Local reads • Local writes
  • 7. Company Confidential© 2014 DataStax, All Rights Reserved. 7 • Disk • Disk space • I/O bandwidth • Network • OS Resources/Limits • File Handles • Processes • Mapped Memory System Resources • CPU • Single Core utilization • Multi Core utilization • Memory • Heap space • Off-heap space • OS page cache • Garbage Collection
  • 8. Company Confidential© 2014 DataStax, All Rights Reserved. 8 Linux monitoring commands Command What it tells you… top CPU utilization and memory use per process top -H CPU utilization per thread, memory use is still per process df Free disk space iostat -x I/O bandwidth utilization free -m Memory and cache usage netstat -an Network connections established iftop Network bandwidth utilization sar All (most) of the above, with history!
  • 9. Company Confidential© 2014 DataStax, All Rights Reserved. 9 Java monitoring commands Command What it tells you… jstack -l Status and stack trace of each thread jmap -histo Types of objects on the heap (optionally only live objects) jmap -heap Size and usage of each java heap generation jstat -gccause Causes of gc activity jmap -dump Take a heap dump for further analysis MemoryAnalyzer Post-mortem heap-dump analysis
  • 10. Company Confidential© 2014 DataStax, All Rights Reserved. 10 nodetool commands Command What it tells you… status / ring Overall cluster status info Status, memory usage, and caches for a single node tpstats Statistics about each thread pool on a single node cfstats Summary statistics for all tables and keyspaces on a single node cfhistograms Detailed statistics for a specific table on the local node proxyhistograms Latency statistics for requests coordinated the local node compactionstats Compactions pending and in progress netstats Network activity: streams, read repair, and in-flight commands
  • 11. Company Confidential© 2014 DataStax, All Rights Reserved. 11 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 12. Company Confidential© 2014 DataStax, All Rights Reserved. 12 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 13. Company Confidential© 2014 DataStax, All Rights Reserved. 13 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 14. Company Confidential© 2014 DataStax, All Rights Reserved. 14 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 15. Company Confidential© 2014 DataStax, All Rights Reserved. 15 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 16. Company Confidential© 2014 DataStax, All Rights Reserved. 16 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 17. Company Confidential© 2014 DataStax, All Rights Reserved. 17 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 18. Company Confidential© 2014 DataStax, All Rights Reserved. 18 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 19. Company Confidential© 2014 DataStax, All Rights Reserved. 19 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 20. Company Confidential© 2014 DataStax, All Rights Reserved. 20 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 21. Company Confidential© 2014 DataStax, All Rights Reserved. 21 nodetool status Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
  • 22. Company Confidential© 2014 DataStax, All Rights Reserved. 22 nodetool ring Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: us-east ========== Address Rack Status State Load Owns Token -3074457345618258603 54.174.19.98 1b Down Normal 22.82 MB 16.67% -9223372036854775808 54.174.245.247 1b Up Normal 22.68 MB 16.67% -6148914691236517206 54.173.171.164 1b Up Normal 22.78 MB 16.67% -3074457345618258603 Datacenter: us-west ========== Address Rack Status State Load Owns Token 6148914691236517205 54.153.39.203 1a Up Normal 22.67 MB 16.67% 0 54.153.107.100 1a Up Normal 22.72 MB 16.67% 3074457345618258602 54.153.108.157 1a Up Normal 22.79 MB 16.67% 6148914691236517205
  • 23. Company Confidential© 2014 DataStax, All Rights Reserved. 23 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 24. Company Confidential© 2014 DataStax, All Rights Reserved. 24 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 25. Company Confidential© 2014 DataStax, All Rights Reserved. 25 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 26. Company Confidential© 2014 DataStax, All Rights Reserved. 26 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 27. Company Confidential© 2014 DataStax, All Rights Reserved. 27 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 28. Company Confidential© 2014 DataStax, All Rights Reserved. 28 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 29. Company Confidential© 2014 DataStax, All Rights Reserved. 29 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 30. Company Confidential© 2014 DataStax, All Rights Reserved. 30 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 31. Company Confidential© 2014 DataStax, All Rights Reserved. 31 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 32. Company Confidential© 2014 DataStax, All Rights Reserved. 32 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 33. Company Confidential© 2014 DataStax, All Rights Reserved. 33 nodetool info Token : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
  • 34. Company Confidential© 2014 DataStax, All Rights Reserved. 34 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 35. Company Confidential© 2014 DataStax, All Rights Reserved. 35 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 36. Company Confidential© 2014 DataStax, All Rights Reserved. 36 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 37. Company Confidential© 2014 DataStax, All Rights Reserved. 37 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 38. Company Confidential© 2014 DataStax, All Rights Reserved. 38 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 39. Company Confidential© 2014 DataStax, All Rights Reserved. 39 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 40. Company Confidential© 2014 DataStax, All Rights Reserved. 40 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 41. Company Confidential© 2014 DataStax, All Rights Reserved. 41 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 42. Company Confidential© 2014 DataStax, All Rights Reserved. 42 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 43. Company Confidential© 2014 DataStax, All Rights Reserved. 43 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 44. Company Confidential© 2014 DataStax, All Rights Reserved. 44 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 45. Company Confidential© 2014 DataStax, All Rights Reserved. 45 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 46. Company Confidential© 2014 DataStax, All Rights Reserved. 46 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 47. Company Confidential© 2014 DataStax, All Rights Reserved. 47 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 48. Company Confidential© 2014 DataStax, All Rights Reserved. 48 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 49. Company Confidential© 2014 DataStax, All Rights Reserved. 49 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 50. Company Confidential© 2014 DataStax, All Rights Reserved. 50 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 51. Company Confidential© 2014 DataStax, All Rights Reserved. 51 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 52. Company Confidential© 2014 DataStax, All Rights Reserved. 52 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 53. Company Confidential© 2014 DataStax, All Rights Reserved. 53 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 54. Company Confidential© 2014 DataStax, All Rights Reserved. 54 nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
  • 55. Company Confidential© 2014 DataStax, All Rights Reserved. 55 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 56. Company Confidential© 2014 DataStax, All Rights Reserved. 56 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 57. Company Confidential© 2014 DataStax, All Rights Reserved. 57 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 58. Company Confidential© 2014 DataStax, All Rights Reserved. 58 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 59. Company Confidential© 2014 DataStax, All Rights Reserved. 59 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 60. Company Confidential© 2014 DataStax, All Rights Reserved. 60 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 61. Company Confidential© 2014 DataStax, All Rights Reserved. 61 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 62. Company Confidential© 2014 DataStax, All Rights Reserved. 62 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 63. Company Confidential© 2014 DataStax, All Rights Reserved. 63 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 64. Company Confidential© 2014 DataStax, All Rights Reserved. 64 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 65. Company Confidential© 2014 DataStax, All Rights Reserved. 65 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 66. Company Confidential© 2014 DataStax, All Rights Reserved. 66 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 67. Company Confidential© 2014 DataStax, All Rights Reserved. 67 nodetool cfstats Keyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
  • 68. Company Confidential© 2014 DataStax, All Rights Reserved. 68 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 69. Company Confidential© 2014 DataStax, All Rights Reserved. 69 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 70. Company Confidential© 2014 DataStax, All Rights Reserved. 70 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 71. Company Confidential© 2014 DataStax, All Rights Reserved. 71 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 72. Company Confidential© 2014 DataStax, All Rights Reserved. 72 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 73. Company Confidential© 2014 DataStax, All Rights Reserved. 73 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 74. Company Confidential© 2014 DataStax, All Rights Reserved. 74 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 75. Company Confidential© 2014 DataStax, All Rights Reserved. 75 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 76. Company Confidential© 2014 DataStax, All Rights Reserved. 76 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 77. Company Confidential© 2014 DataStax, All Rights Reserved. 77 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 78. Company Confidential© 2014 DataStax, All Rights Reserved. 78 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 79. Company Confidential© 2014 DataStax, All Rights Reserved. 79 nodetool cfhistograms Keyspace1/Standard1 histograms SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1 Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2 Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
  • 80. Company Confidential© 2014 DataStax, All Rights Reserved. 80 nodetool cfhistograms Write Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839 Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
  • 81. Company Confidential© 2014 DataStax, All Rights Reserved. 81 nodetool cfhistograms Write Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839 Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
  • 82. Company Confidential© 2014 DataStax, All Rights Reserved. 82 nodetool cfhistograms Write Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839 Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
  • 83. Company Confidential© 2014 DataStax, All Rights Reserved. 83 nodetool cfhistograms Write Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839 Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
  • 84. Company Confidential© 2014 DataStax, All Rights Reserved. 84 nodetool cfhistograms Write Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839 Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
  • 85. Company Confidential© 2014 DataStax, All Rights Reserved. 85 nodetool proxyhistograms Read Latency (microseconds) 124 us: 18 149 us: 445 179 us: 5866 215 us: 14541 258 us: 19123 310 us: 20941 372 us: 32847 446 us: 37464 535 us: 26451 642 us: 25291 770 us: 37870 924 us: 35865 1109 us: 30432 1331 us: 22929 1597 us: 17592 1916 us: 12895 2299 us: 9760 2759 us: 7887 3311 us: 6024 3973 us: 4751 4768 us: 3691 5722 us: 3038 6866 us: 2453 8239 us: 1991 Write Latency (microseconds) 29 us: 4 35 us: 60 42 us: 331 50 us: 471 60 us: 1334 72 us: 2856 86 us: 4661 103 us: 65741 124 us: 37999 149 us: 42697 179 us: 45598 215 us: 30515 258 us: 23481 310 us: 21831 372 us: 22741 446 us: 26045 535 us: 33045 642 us: 43533 770 us: 51021 924 us: 57496 1109 us: 61548 1331 us: 60741 1597 us: 57538 1916 us: 52255
  • 86. Company Confidential© 2014 DataStax, All Rights Reserved. 86 nodetool compactionstats pending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
  • 87. Company Confidential© 2014 DataStax, All Rights Reserved. 87 nodetool compactionstats pending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
  • 88. Company Confidential© 2014 DataStax, All Rights Reserved. 88 nodetool compactionstats pending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
  • 89. Company Confidential© 2014 DataStax, All Rights Reserved. 89 nodetool compactionstats pending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
  • 90. Company Confidential© 2014 DataStax, All Rights Reserved. 90 nodetool compactionstats pending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
  • 91. Company Confidential© 2014 DataStax, All Rights Reserved. 91 nodetool compactionstats pending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
  • 92. Company Confidential© 2014 DataStax, All Rights Reserved. 92 nodetool compactionstats pending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
  • 93. Company Confidential© 2014 DataStax, All Rights Reserved. 93 nodetool compactionstats pending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
  • 94. Company Confidential© 2014 DataStax, All Rights Reserved. 94 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 95. Company Confidential© 2014 DataStax, All Rights Reserved. 95 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 96. Company Confidential© 2014 DataStax, All Rights Reserved. 96 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 97. Company Confidential© 2014 DataStax, All Rights Reserved. 97 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 98. Company Confidential© 2014 DataStax, All Rights Reserved. 98 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 99. Company Confidential© 2014 DataStax, All Rights Reserved. 99 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 100. Company Confidential© 2014 DataStax, All Rights Reserved. 100 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 101. Company Confidential© 2014 DataStax, All Rights Reserved. 101 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 102. Company Confidential© 2014 DataStax, All Rights Reserved. 102 nodetool compactionhistory Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out 5d4adcc0-cbf9-11e4-9f32-098a653a7013 system local 1426523260044 1101 543 654f31c0-cc01-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426526709468 53518892 53518892 59a7b070-cc1c-11e4-a84d-098a653a7013 system hints 1426538286327 106483 0 4b8e2b10-cbf2-11e4-bb39-098a653a7013 system schema_keyspaces 1426520223809 909 264 68429440-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537022340 59878420 59109472 f06de4a0-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537250794 227493794 133487816 3bec3ee0-cbf2-11e4-bb39-098a653a7013 system local 1426520197582 720 550 e263a080-cbfa-11e4-87be-098a653a7013 system schema_columns 1426523912832 27224 11517 54741560-cc19-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426536989110 70198058 61572552 c67f8c90-cc08-11e4-a84d-098a653a7013 system peers 1426529879001 2219 778 1edb6bf0-cc1a-11e4-a84d-098a653a7013 Keyspace1 Standard1 1426537328687 19723822 16552676 rows_merged {1:247562, 2:67934, 3:2945, 4:2042, 5:824, 6:232, 7:2, 8:1}
  • 103. Company Confidential© 2014 DataStax, All Rights Reserved. 103 nodetool netstats Mode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db 851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db 6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db 9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db 2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db 1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 46561942/46561942 bytes(100%) sent to /54.153.39.203 Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746 Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
  • 104. Company Confidential© 2014 DataStax, All Rights Reserved. 104 nodetool netstats Mode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db 851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db 6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db 9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db 2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db 1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 46561942/46561942 bytes(100%) sent to /54.153.39.203 Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746 Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
  • 105. Company Confidential© 2014 DataStax, All Rights Reserved. 105 nodetool netstats Mode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db 851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db 6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db 9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db 2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db 1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 46561942/46561942 bytes(100%) sent to /54.153.39.203 Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746 Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
  • 106. Company Confidential© 2014 DataStax, All Rights Reserved. 106 nodetool netstats Mode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db 851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db 6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db 9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db 2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db 1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 46561942/46561942 bytes(100%) sent to /54.153.39.203 Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746 Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
  • 107. Company Confidential© 2014 DataStax, All Rights Reserved. 107 nodetool netstats Mode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db 851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db 6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db 9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db 2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db 1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 46561942/46561942 bytes(100%) sent to /54.153.39.203 Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746 Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
  • 108. Company Confidential© 2014 DataStax, All Rights Reserved. 108 nodetool netstats Mode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db 851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db 6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db 9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db 2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db 1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 46561942/46561942 bytes(100%) sent to /54.153.39.203 Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746 Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
  • 109. Company Confidential© 2014 DataStax, All Rights Reserved. 109 nodetool netstats Mode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db 851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db 6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db 9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db 2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db 1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 46561942/46561942 bytes(100%) sent to /54.153.39.203 Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746 Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
  • 110. Company Confidential© 2014 DataStax, All Rights Reserved. 110 nodetool netstats Mode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db 851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db 6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db 9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db 2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db 1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db 46561942/46561942 bytes(100%) sent to /54.153.39.203 Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746 Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081

Editor's Notes

  1. Role at DataStax Previous Experience Something about Atlanta
  2. Overall troubleshooting process from a support engineer’s perspective I’ll focus on the tools Cassandra gives you to do steps 1 to 4 Steps 5 and 6 are the hard parts; but luckily that’s what DataStax support, or the mailing list, or StackOverflow can help you with Doing the legwork on the first part will make the second part happen much faster
  3. From a user’s perspective, the troubleshooting process could be revised like this Definitely better to over-share than under-share information Make sure you share all the facts, not just those that support your current theory
  4. When troubleshooting, it’s helpful to keep in mind the various processes Cassandra runs to do its work These can be roughly divided into: startup processes Foreground processes (serving reads and writes); your application has control over these! Background processes (happen periodically); Cassandra decides when to do them, but they can be tuned in many cases
  5. It’s also helpful to keep in mind the various system resources that Cassandra consumes CPU keep in mind both the speed of a single core and of multiple cores necessary because some processes are single-threaded and bottlenecked by a single core Memory Heap space, typically limited to a subset of your total physical memory Cassandra stores many objects off heap to avoid Garbage Collection issues The OS will use whatever memory is left over for page cache; make sure you leave enough free memory for this Garbage collection Driven by memory utilization Drives CPU utiliization Disk Available disk space I/O bandwidth utilization Network Primarily concerned with bandwidth Keep in mind firewalls; the path needs to be open OS Resources/Limits File handles, processes, etc. Make sure you set high enough limits in limits.conf or ulimits
  6. Overview of some of the most useful nodetool commands and what they do
  7. Nodetool status gives an overview of the entire cluster’s status
  8. The cluster is divided into datacenter
  9. The first column shows whether a node is up or down
  10. The second column shows the node’s state, which one of: normal leaving the cluster joining the cluster moving its token
  11. The IP address of each node. This will be the broadcast_address if specified; otherwise, the listen_address
  12. The amount of data on each node It is normal for this to differ slightly Large discrepancies could indicate a problem (in terms percentage) Wide rows/partitions Uneven racks Compaction issues
  13. Number of tokens Normally 256 for vnodes, 1 for non-vnodes
  14. Percentage of ring each node owns Note message at the top Without a keyspace, assumes SimpleStrategy with RF=1 Can cause strange readout when using multiple DCs with a small offset between tokens (nothing to worry about) With keyspace, shows ownership according to RF in that keyspace With keyspace, should add up to RF times 100% May differ slightly from node to node with vnodes due to random token distribution
  15. UUID of the node. Used to uniquely identify the node when running removenode command.
  16. The rack the node is on Used to avoid single point of failure Avoid if not using vnodes If using vnodes, ensure the same number of nodes are in each rack
  17. In this example, we have one node that is down, so that is where we’d focus our investigation
  18. Nodetool ring is an old way of checking status Shows the same information as nodetool status, except for the token When using vnodes, it will show every token on each node and becomes difficult to read
  19. Nodetool info shows information specific to the node where it is run Some of the information is also shown by nodetool ring/status
  20. Same information shown by nodetool status and ring Not going to rehash this
  21. Whether gossip, thrift, and native transport are enabled Can be individually enabled/disabled with nodetool commands
  22. Gossip generation; increases each time the node is restarted
  23. How many seconds the node has been running
  24. Heap and off heap memory usage Heap memory shows amount currently in use and maximum Amount currently in use may include garbage The amount of garbage included varies depending on when GC was last run Off heap memory is stored outside the heap but adds to the process’s overall resident size If a process gets killed by the kernel OOM killer, make sure it isn’t using too much off heap memory
  25. Number of exceptions that occurred since last restart Not every error involves an exception, but it’s still a good indicator
  26. Key Cache size and capacity Size is amount actually in use Capacity is amount specified by key_cache_size_in_mb in cassandra.yaml Capacity defaults to 100MB or 5% of the heap, whichever is less If size is consistently much less than capacity, you have this set too high
  27. Cache hits, total requests, and hit rate - If hit rate is low, try increasing key cache capacity slowly until you see diminishing returns
  28. Key cache is periodically saved to disk and reloaded when a node is restarted This is to avoid a cold key cache after restart
  29. Same information shown for row cache Row cache should be disabled except for extremely narrow set of use cases Small, very hot data set that will fit in memory Data should be read much more than written because writes invalidate row in cache But really, don’t use row cache
  30. tpstats shows thread pool statistics Cassandra has various thread pools that handle important foreground and background tasks This information is also logged to system.log by StatusLogger.java periodically or when a message is dropped
  31. Name of the thread pool
  32. Number of tasks being actively serviced by a thread For several stages, this can be configured in cassandra.yaml For others, it is equal to the number of CPU cores For a few others, it is a hardcoded limit, usually 1
  33. Number tasks waiting to be serviced For most stages, limited to ~2 billion You’ll run out of memory long before you ever hit this limit High number of pending tasks indicates the stage is overloaded
  34. Number of tasks completed since last restart
  35. Number of tasks currently blocked for I/O Should almost always be 0
  36. Total number of tasks blocked since last restart. Usually zero except for FlushWriter
  37. At the bottom is a list of message types and the number of messages dropped Load shedding drops requests that have been pending beyond timeout specified in cassandra.yaml Dropped messages usually indicate overloaded cluster; check for other causes, then add more nodes
  38. Handles local reads for which this node is a replica Number of threads is controlled by concurrent_reads in cassandra.yaml
  39. Various reads, all handled by ReadStage READ - normal read on a single partition RANGE_SLICE - A sequential or secondary index scan over multiple partitions PAGED_RANGE - Used for automatic paging when result size exceeds row limit
  40. Handles local writes for which this node is a replica Number of threads is controlled by concurrent_writes in cassandra.yaml
  41. Various writes, handled by MutationStage READ_REPAIR - write due to a read repair MUTATION - normal write COUNTER_MUTATION - incrementing a counter
  42. Coordinator uses this to process responses from other nodes Roughly indicates how often this node has been a coordinator Roughly because unless using CL ONE, need to handle responses from multiple nodes
  43. Request completed but timed out before coordinator could respond to it Timeout controlled by request_timeout_in_ms in cassandra.yaml May indicate that coordinator is overloaded Ensure that client load balancing is set up correctly Make sure use of batches is appropriate Logged batches only when atomicity is required Unlogged batches only when updating multiple rows with the same partition key Otherwise use asynchronous execution to pipeline requests without overloading a single coordinator
  44. Writes memtables to disk Number of threads controlled by memtable_flush_writers, should equal number of data drives Maximum pending tasks controlled by memtable_flush_queue_size in cassandra.yaml Once queue is full writes are blocked until another flush writer is available Large number of all-time-blocked tasks indicates a disk bottleneck; add more/faster disks or more nodes
  45. Handles compactions Number of threads controlled by concurrent_compactors in cassandra.yaml Constantly pending compactions means compactions can’t keep up with writes; mitigation strategies: Switch from LeveledCompactionStrategy to SizeTieredCompactionStrategy for write-heavy tables Get faster disks (SSD) if needed Increase compaction throughput Get faster CPU cores
  46. Asynchronous read repairs Occur for a certain percentage of reads, configurable per table Pending tasks indicate that you may have read repair chance set too high
  47. Handles hint delivery from the coordinator to a node that’s recently come back up Large number of hints usually means an unhealthy cluster
  48. Nodes gossip with each other once a second Prior to 2.0, when using vnodes with a large cluster, gossip could become CPU-bound and get behind
  49. Migrates schema changes to other nodes If you see pending tasks here, you’re making too many schema changes
  50. Repair related stages AntiEntropyStage coordinates repairs AntiEntropySesssions are active repairs in progress ValidationExecutor builds merkle trees for repair
  51. cfstats shows statistics for individual tables Tables are grouped by keyspace Values apply only to the node where cfstats is run, not cluster-wide
  52. Keyspace and table names Multiple tables grouped under each keyspace
  53. Read and write counts Per table table and at keyspace level
  54. Read and write latency Per table and averaged at keyspace level
  55. Number of thread pool tasks pending against a table Per table and summed at keyspace level
  56. Number of sstables comprising a table Broken down by level when using LeveledCompactionStrategy
  57. Space used by the table on this node Must sum across different nodes to get total space May include deleted or updated data that hasn’t been compacted yet
  58. Number of partition keys in the table on this node Estimated to the nearest index_interval (128 by default) Rows spread across multiple sstables will inflate this number
  59. Space consumed by off-heap data structures Total off head memory Broken down by data structure: bloom filter, index summary, compression metadata
  60. Memtable information number of entries in the memtable bytes of data in the memtable the number of times the memtable has been switched (flushed to disk)
  61. Bloom filter statistics False positives Too high, performance will suffer Reduce false positive chance for table Space used Lower false positive chance requires more space Bloom filter grows linearly with number of partitions Increase false positive if bloom filter is too large
  62. Statistics on partition size, calculated during compaction Maximum, minimum, and average size Helps identify tables containing large partitions
  63. Tombstone statistics Number of live cells versus tombstones encountered when scanning a partition Rolling average for the last 5 minutes
  64. cfhistograms provides deeper insight into a specific table Must specify keyspace and table name when calling nodetool cfhistograms Information shown by cfhistograms is local to the node where it was run
  65. Keyspace and table name
  66. Number of sstables each read hit More sstables means slower reads, higher I/O utilization If reads are hitting too many sstables, consider LeveledCompactionStrategy LCS incurs more I/O due to compaction on write-heavy tables
  67. The size of each partition in bytes Scanning through large partitions generates garbage and hinders performance Large partitions cause uneven data distribution across nodes
  68. Number of cells in each partition Total number of columns belonging to all logical rows within the partition Large number of cells may cause increased garbage generation
  69. How to read cfhistograms Left side shows list of buckets Bucket ranges from previous line (exclusive) to current line (inclusive) Buckets get progressively larger as numbers increase
  70. Count of items that fall in this bucket What’s being counted depends on the statistic
  71. Heading tells you what’s being counted For “SSTables per Read”, we’re counting number of reads For the other two, we’re counting the number of partitions
  72. Also pay attention to the units used for the buckets
  73. Example 126,087 reads hit 4 sstables 37 reads hit between 9 and 10 sstables
  74. Example 73 partitions between approx 195K and 234K bytes
  75. Example 14,532 partitions with between 771 and 924 cells in them
  76. Read latency for local reads
  77. Write latency for local writes
  78. Unit is microseconds, not milliseconds
  79. Example 1379 reads served locally by this node took between 925 and 1109 us
  80. Example 1839 writes served locally by this node took between 216 and 258 us
  81. Shows the latency for reads and writes that this node coordinated Latency for the entire request, including network latency between coordinator and replicas Not broken down by table
  82. Status of current and pending compactions
  83. Number of pending compactions
  84. Compactions in progress
  85. Compaction Type Compaction - normal compaction Validation - building merkle trees for repair
  86. Keyspace and table
  87. Bytes complete and total bytes for each compaction
  88. Percent done
  89. Estimated time remaining for the active tasks Not useful, because: estimates can be wrong doesn’t account for pending tasks
  90. History of recent compactions Shows how much space a compaction reclaimed Same information available in system.log
  91. Unique ID
  92. Keyspace and column family
  93. Unix timestamp when compaction occurred Seconds since Jan 1, 1970
  94. Total bytes before compaction
  95. Total bytes after compaction
  96. Row merge counts Actually the last column on each row Moved to make output fit on slide
  97. Count of sstables
  98. Number of rows spread across that many sstables prior to compaction
  99. Shows network activity
  100. Mode, same as on status: NORMAL, JOINING, LEAVING, MOVING
  101. Active streams
  102. Repair session IDs
  103. Nodes exchanging data
  104. Data is streaming over listen address instead of broadcast address Useful on EC2 because Amazon doesn’t charge for internal traffic
  105. Total number of files and bytes of data to be received from specified node
  106. Total number of files and bytes of data to be sent to specified node
  107. Specific files currently being transferred
  108. Progress for each file
  109. Read repair statistics
  110. Number of read repairs attempted
  111. Number of mismatches resolved in the foreground
  112. Number of mismatches resolved in the background
  113. Commands sent and responses received while acting as coordinator
  114. Number of pending and completed commands sent to other nodes
  115. Number of pending and completed responses received from other nodes
  116. Used to check for schema disagreements
  117. This is not the information we’re interested in
  118. We’re looking to see how many schema versions there are in the cluster.
  119. No disagreement; only one version of the schema shared by all nodes
  120. Schema disagreement; one node has a different version from all the others Schema disagreements must be manually resolved If only one node disagrees, run nodetool resetlocalschema on that node If multiple nodes disagree shut down nodes in the minority and delete system/schema_* sstables start nodes back up one by one
  121. system.log is the most important tool for troubleshooting Cassandra Where it is How to configure it logging level location override logging level for a specific class Basic format Level: INFO, WARN, ERROR by default; DEBUG only if configured Thread: use the ID to correlate messages from the same thread Date & Time: use to correlate messages across multiple nodes, time duration of events Source File/Line No: code that logged the message, not necessarily where an error occurred; talk about stack traces later
  122. Exception - What kind of error occurred?
  123. Stack trace – where did the error happen? Most local at top to most global at bottom Wall of text — we’ll dissect it in the next slides
  124. Organization names – whose fault is it?
  125. Sub-packages usually group major application subsystems
  126. Class Name – specific object Will usually, but not always match the filename
  127. Nested Classes - $ separates outer class from inner class(es) Method belongs to the inner class
  128. Method name – what was the class doing? <init> indicates that the error occurred in a static initialization block or constructor
  129. File name – where the source code is, should you want to look at it Also pay attention to the package name so you can find the file within the nested directory structure When source-diving, start with the most local method and work your way out
  130. Line number – where to look in the code (available on github.com/apache/cassandra) Be careful! Line numbers change between versions, so make sure you select the the right version in github
  131. Pay attention to nested exceptions Each exception has its own stack trace which may be completely different The outer exception may be too general because it’s been rethrown from unrelated code The innermost exception will be the actual root cause of the error Best to use a combination of outer and inner exception as search terms
  132. Exception will usually have an error message Provides additional information about the circumstances of the exception Usually good to search for the exception and message together Look out for embedded numbers or strings; these may change from one message to the next, and including them will undesirably narrow your search
  133. - Some additional examples of organizations and subsystems
  134. - Some additional examples of organizations and subsystems
  135. Use exception and several package+class+method names Exception alone often isn’t sufficient because the same error can occur many different places You’ll find the same exception in unrelated software if it’s a standard java exception. Add several package/class/method combinations to narrow down the exception Use at least the topmost method and the first org.apache.cassandra method Use quotes around individual elements (especially if they contain spaces) Line numbers shouldn’t be part of your search criteria because you may not find the same error in a different version Likewise, exclude specific numbers and strings like names and counts from your search Use Google’s site: feature to narrow search terms to apache JIRA, cassandra mailing list, or stackoverflow Add or remove additional methods as needed to narrow or broaden search
  136. These might be a good set of search terms for this exception Include both exceptions and methods from both stack traces Include exception and error message grouped together inside quotes
  137. Know how to recognize a restart Check versions of major components and JVM Confirm settings are what you think they are Make sure you have JNA installed Know when node is ready to serve requests
  138. Cassandra writes updates to the commit log on disk and the memtables in memory When memtables get full, they’re flushed to disk and the associated commit log is reclaimed Flushes can also be triggered when the node is running low on memory
  139. First the flush is enqueued There are a limited number of FlushWriter threads Flushes wait in the queue until a FlushWriter is available
  140. - When a FlushWriter becomes available, the flush begins
  141. Eventually the flush completes. This is not an instantaneous process because disks are slow.
  142. This is the name of the table and the unique identifier of the memtable You can use this to link the enqueuing and writing of the flush
  143. Make note of the FlushWriter thread doing the flush You can use this to link the beginning and end of the flush
  144. This shows the number of serialized and live bytes Serialized vs live is the size of the data on disk vs the size in heap It also shows the number of individual write operations stored in the memtable
  145. - This shows the name of the sstable that the memtable was written to on disk
  146. The size of the sstable This is typically smaller than the size in memory because of compression
  147. This shows the segment ID for the commit log and the position within the segment The commit log is broken up into small files called segments When all the commits in a segment are flushed, the segment is reclaimed
  148. Note the times on the messages Time between first and second messages is how long the flush waited in the queue Time between second and third messages is how long the flush took to complete
  149. Sstables are immutable; updates go into new sstables Reads scan over multiple sstables to stitch together a row SSTables must eventually be compacted together to keep reads fast Size-tiered compactions occur when a sufficient number of similarly sized sstables exist In leveled compaction, sstables are written to Level 0 and moved to higher levels as they are compacted Leveled compactions occur continuously as long as sstables exist in Level 0
  150. Note the CompactionExecutor thread doing the compaction The thread ID can be used to link together the messages
  151. The compaction is beginning
  152. The sstables that are going to be compacted
  153. The compaction is complete
  154. How many tables were compacted
  155. The name of the new sstable created by the compaction
  156. The number of bytes in the original files The number of bytes in the new file The percentage of the original size after: Updates were merged Tombstones and expired TTLs were removed
  157. The time the compaction took and the rate in MB/sec
  158. The sum of the number of rows in each sstable The number of unique rows across all compacted sstables
  159. X:Y where Y rows were split across X sstables
  160. During compaction, you may see one or more messages about partitions being compacted incrementally Logical CQL 3 rows sharing the same partition key form a physical row when stored in the cluster Newer versions of Cassandra say partition instead of row Large partitions can cause a number of problems for Cassandra Uneven distribution of data between nodes Large memory usage when large partitions are read all at once Generating lots of garbage compacting a wide row Slower compactions because they’re done incrementally on disk These messages can help identify large rows
  161. The keyspace, table, and partition key
  162. The size of the partition
  163. Garbage collections are a necessary evil Some garbage collections run concurrently with Cassandra, but others stop the world Cassandra logs any stop-the-world collections that last longer than 200ms Stop the world collections cause nodes to stop responding to gossip and client requests
  164. This shows the type of collection Java allocates new objects into the young generation Objects that survive a specified number of collections get promoted to the tenured generation ParNew collections occur when the young gen is collected. These are stop-the-world. The young gen is usually small so ParNew is usually fast If there’s not enough contiguous space in the old gen to promote an object, the old gen must be compacted, which takes a long time ConcurrentMarkSweep normally runs concurrently with the application and does not stop the world If the concurrent collection can’t keep up with the rate at which garbage is generated, a stop-the-world collection occurs Stop-the-world CMS can take a very long time because the old gen is usually big
  165. This shows the number of ms elapsed for the collection Long collections increase latency of read and write requests Very long collections will prevent the node from gossiping and other nodes will think it’s down
  166. Note the time on each GCInspector message. Even if the individual collections are fast, too many collections within a short timespan can hurt throughput
  167. This shows the number of individual collections reported included in the duration
  168. This shows the amount of heap in use before the collection occurred.
  169. This shows the maximum amount of heap available
  170. After a collection completes, warning is logged if the heap is still greater than 75% full If enough space can’t be reclaimed repeatedly, the node may enter a GC death spiral
  171. When the threshold is exceeded, it triggers a memtable flush to free up memory This threshold can be configured using flush_largest_memtables_at in cassandra.yaml Usually, you shouldn’t change this setting If you see this message, you should try to reduce heap usage or increase heap size Increasing heap size above 8GB is not recommended because GC will take longer
  172. Flapping is often caused by garbage collections
  173. The nodes go up-down-up-down, repeatedly That’s why it’s called flapping
  174. Notice the nodes that are flapping If a single node is flapping, check the logs on that node during the same timeframe and see if GC is occurring If multiple nodes are are reported up and down, the local node may be the problem If a node is doing GC it won’t be able to receive gossip messages from another node and may think they’re down Check for GC messages in the local log around the time that flapping occurs
  175. Note the time that flapping occurs If it happens infrequently, it may not be a problem If it happens multiple times a minute, it is a problem Check other node’s logs for GC events that occurred at the same time If you don’t see anything in the other node’s log, it may be a network issue You can reduce the failure detector’s sensitivity by increasing phi_convict_threshold in cassandra.yaml Default value is 8; maximum recommended value is 12 (useful on high latency networks such as AWS)
  176. If a write comes in for a down node, the coordinator will store hints for it When the node comes back up, the nodes that have hints for it will send them Hints are no longer stored after the node has been down over period of time specified in cassandra.yaml. This is to prevent a node that comes back up from being inundated with more hints than it can handle Any node that has been down longer than this period of time needs to run nodetool repair Flapping can cause excessive hint buildup, which adds extra burden for both the coordinator and the node that is flapping This can lead to cascading failures
  177. Repairs are initiated by running nodetool repair They do a full comparison of all the data for a particular token range with the other replicas for that range, then exchange any data that is out of sync system.log shows the process from start to finish
  178. Note the UUID for the repair session. This is your key to correlating the various messages.
  179. When a repair session begins, you will see a “new session” message
  180. It will report the nodes it’s going to sync with
  181. The token ranges it’s going to sync
  182. And the keyspace and column families it’s going to sync
  183. The first step is for the repair leader to request merkle trees from all the other replicas
  184. A message is logged reporting the receipt of each requested merkle tree Make sure you see a message that the merkle tree was received from each node that it was requested from
  185. After comparing the merkle trees, if the nodes are in sync, you’ll see a message like this
  186. If not, you’ll see a message like this, reporting how many ranges are out of sync
  187. The node will then begin a streaming repair with the out-of-sync replica
  188. Another message reports when the streaming task has succeeded
  189. A message will report when each table has been fully synced. This means either it was in sync to begin with, or all the streaming tasks necessary to sync it completed.
  190. Once all tables are synced, a message will report that the overall repair session completed successfully. If you see a “new session” message for a particular ID but not a “session completed successfully”, the repair is still running. If a repair doesn’t complete successfully after some time, you should look more closely at the other messages for that session to see where it might be stuck. Sometimes network issues can disrupt the streaming of data or a merkle tree, causing repair to hang Other times, there is simply a lot of data, and building merkle trees can take a long time, as can streaming data Increasing compaction throughput and streaming throughput will help speed the process, at the cost of using extra I/O and network bandwidth. Check the other nodes involved in the repair for messages using the same session ID Check for any errors that would have disrupted the repair
  191. Before we end, I just want to go back to the troubleshooting process I discussed at the beginning Next time you have a problem, think about the tools at your disposal and how they can help you with these steps