This article describes how Cassandra handles and processes requests. It will help you to get a better impression about Cassandra's internals and architecture. The path of a single read request as well as the path of a single write request will be described in detail.

1. Cassandra by example - the path of read and
write requests
Abstract
This article describes how Cassandra handles and processes requests. It will help you to get a better
impression about Cassandra's internals and architecture. The path of a single read request as well as
the path of a single write request will be described in detail. This description is based on a single data
center Cassandra V1.1.4 cluster (default store configuration).
Example data model
Please consider that this article is not an introduction to the Cassandra model. In the examples below
a column family hotel is used. In short, a column family is analogous to tables of the relational
database approach. Each hotel record or row is identified by a unique key. The columns of a hotel
row include the hotel name as well as the category of the hotel.
The column family hotel lives inside the keyspace book_a_hotel . A keyspace can be described by
analogy as a tablespace or database.
Thrift
The common way to access Cassandra is using Thrift. Thrift is a language-independent RPC protocol
originally developed at Facebook and contributed to Apache. Although Thrift is widely supported by
the most popular programming languages the Cassandra project suggests using higher level
Cassandra clients such as Hector or Astyanax instead the raw Thrift-based API. In general these high
level clients try to hide the underlying middleware protocol.
Gregor Roth Cassandra by example - the path of read and write requests 1

2. The listing below shows a simple query by using the Hector client library V1.1.
// [1] prepare the client (cluster)
Cluster cluster = HFactory.getOrCreateCluster("TestClstr", "172.39.126.14, 172.39.126.93, 172.39.126.52");
Keyspace keyspaceOperator = HFactory.createKeyspace("book_a_hotel", cluster);
// [2] create the query (fetching the column category)
SliceQuery<String, String, String> query = HFactory.createSliceQuery(keyspaceOperator,
AsciiSerializer.get(), StringSerializer.get(), StringSerializer.get());
query.setColumnFamily("hotel");
query.setKey("26813445");
query.setColumnNames("category");
// [3] perform the request
QueryResult<ColumnSlice<String, String>> result = query.execute();
ColumnSlice<String, String> row = result.get();
String category = row.getColumnByName("category").getValue();
//...
// [4] release the client (cluster)
cluster.getConnectionManager().shutdown();
In the first line of the listing a set of server IP addresses is passed over by creating a Hector Cluster
object. The server address identifies a single Cassandra node. A collection of independent Cassandra
nodes (the Cassandra cluster) represents the Cassandra database. Within this cluster all nodes are
peers. No master node or something like that exists.
The client is free to connect any Cassandra node to perform any request. In the listing above 3
addresses are configured. This does not mean that the Cassandra cluster consist of 3 nodes. It just
defines that the client will communicate with these nodes only.
The connected Cassandra node plays two roles, potentially. In each case the connected node is the
coordinator node which is responsible to handle the dedicated request. Furthermore the connected
node will be a replica store node, if the node is responsible to store a replica of the requested data.
For instance the requested Pavillon Nation hotel record of the example above does not have to be
stored on the connected node. Often the coordinator node has to send sub requests to other replica
nodes to be able to handle the request. As shown in the diagram below the notes 172.39.126.14,
172.39.126.93 and 172.39.126.52 would not able to serve a Pavillon Nation query in a direct way
without sub requesting other nodes.
Gregor Roth Cassandra by example - the path of read and write requests 2

3. Please consider that a coordinator node and a replica node is a role description of a Cassandra node
in context of a dedicated read or write operation. All Cassandra nodes can be a coordinator node as
well as a replica node.
Hector uses a round-robin strategy to select the node to use. By executing the example query Hector
first connects one of the configured nodes. The connect request will be handled on the server-side by
the CassandraServer .
By default the CassandraServer is bound to server port 9160 during the start sequence of a
Cassandra node. The CassandraServer implements Cassandra's Thrift interface which defines remote
procedure methods such as set_keyspace(…) or get_slice(…).This meansCassandra's Thrift interface is
stateful, implicitly. The Hector client has to call the remote method set_keyspace(..) first to assign the
keyspace book_a_hotel to the current connection session. After assigning the keyspace the
get_slice(..) can be called to request the columns of the Pavillon Nation hotel.
However, you are not forced to use Thrift to access Cassandra. Several alternative open-source
connectors such as REST-based connectors exist.
Determining the replica nodes
The CassandraServer is responsible to handle the client-server communication only. Internally, the
CassandraServer calls the local StorageProxy class to process the request. The StorageProxy
implements the coordinator logic. The coordinator logic includes determining the replica notes for
the request row key as well as requesting these replica nodes.
By default a RandomPartitioner is used to determine the replica nodes for the row key of the
request. The RandomPartinitoner spreads the data records (rows) evenly across the Cassandra nodes
which are arranged in a circular ring. Within this ring each node is assigned to a range of hash values
(tokens). To determine the first replica, the MD5 hash of the row key will be calculated and the node
will be selected where the key hash maps with the assigned token range.
Gregor Roth Cassandra by example - the path of read and write requests 3

4. For instance the token of the Pavillon Nation's row key 26813445 is
91851936251452796391746312281860607309. This token is within the token range of node
172.39.126.86 which means that node 172.39.126.86 is responsible to store a replica of the Pavillon
Nation record.
In most case a replica is stored by more than one node which depends on the key space's replication
factor. For instance a replication factor 2 means the clockwise next node of the ring will store the
replica, too. If replication level is 3, the next of the next will also store the replica and so forth.
Processing a read request
The handle a read request the StorageProxy (which is the coordinator of the request) determines the
replica nodes as described above. Additionally, the StorageProxy checks that enough replica nodes
are alive to handle the read request. If this is true, the replica nodes will be sorted by proximity
(closest node first) and the first replica node will be called to get the requested row data.
In contrast to the thrift-based client-server communication the Cassandra nodes interchange data by
using a message-oriented tcp-based protocol. This means the StorageProxy will get the requested
row data by using Cassandra's messaging protocol.
Calling other replica nodes depends on the consistency level. The consistency level is specified by the
client request. If consistency level ONE is required, no further replica nodes will be called. If
consistency level QUORUM is required, in total (replication_factor / 2) + 1 replica nodes
will be called.
In contrast to the first full-data read call all additional calls are digest calls. A digest call queries a
single MD5 hash of all column names, values and timestamps instead requesting the complete row
data. The hashes of all calls, including the first one will be compared together. If a hash does not
match, the replicas will be inconsistent and the out-of-date replicas will be auto-repaired during the
Gregor Roth Cassandra by example - the path of read and write requests 4

5. read process. To do this, a full-data read request will be sent to the additional nodes, the most recent
version of data will be computed and the diff will be sent to out-of-date replicas.
Occasionally all replica nodes for the row key will be called independent of the requested consistency
level. This depends on the column family's read_repair_chance parameter. This configuration
parameter specifies the probability with which read repairs should be invoked. The default value of
0.1 means that a read repair is performed 10%. However, the client response will always be
answered regarding to the requested consistency level. Additional work will be done in background.
A read_repair_chance parameter larger the 0 ensures that frequently read data remains consistent
even though only consistency level ONE is required. The row becomes consistent eventually.
Performing the local data query
As already mentioned above, a dedicated messaging protocol is used for inter-node communication.
Similar to the CassandraServer the MessagingService will be started during the start sequence of a
Cassandra node, too. By default the MessagingService in bound to server port 7000.
The replica node will receive the read call from the coordinator node through the replica node's
MessagingService. However, the MessagingService will not access the local store in a direct way. To
read and write data locally, the ColumnFamilyStore has to be used. Roughly speaking, the
ColumFamilyStore represents the underlying local store of a dedicated column family.
Please consider that a coordinator node can also be in role replica node. This will be true, if the client
calls node 172.39.126.52 to get the Mister bed city row instead of the Pavillon Nation row in the
example above. In this case the StorageProxy of the coordinator node will not call the
Gregor Roth Cassandra by example - the path of read and write requests 5

6. MessagingService of the same node. To avoid remote calls to the same node, the StorageProxy will
call the ColumnFamilyStore in the same way the MessagingServices does to access local data.
By processing a query the ColumnFamilyStore will try to read the requested row data through the
row cache, if the row cache is activated for the column family. The row cache holds the entire row
and is deactivated per default. If the row cache contains the requests row data, no disk IO will be
required. The query will be served very fast by performing in-memory operations only. However, an
activated row cache causes that the full row have to be fetched internally even though a sub set of
columns is requested. For this reasons the row cache is often less efficient for large rows and small
sub set queries.
If the request row isn't cached, the Memtables and the SSTables (sorted strings table) have to be
read. Memtables and SSTables are maintained per column family. SSTables are data files containing
row data fragments and only allow appending data. A Memtable is an in-memory table which buffers
writes. If the Memtable is full, it will be written to disk as a new SSTable file in background. For this
reason the columns of the requested Pavillon Nation row could be fragmented over several SSTables
and unflushed Memtables. For instance one SSTable book_a_hotel-hotel-he-1-Data.db could contain
the initial inserted columns ‘name’= ‘Pavillon Nation’ and ‘category’=’4’ of the Pavillon Nation row.
Another SSTable book_a_hotel-hotel-he-2-Data.db (or Memtable) could contain the updated
category column ‘category’=’5’.
If an SSTable exists for the requested column family, first the associated (key-scoped) Bloom filter of
the SSTable file will be read to avoid unnecessary disk IO. For each SSTable the ColumnFamilyStore
holds an in-memory structure called SSTableReader which contains metadata as wells as the Bloom
filter of the underlying SSTable file. The Bloom filter indicates that the dedicated SSTable could
contain a row data fragment (false positive are possible, false negative not). If this is true, the key
cache will be requested to get the seek position. If not found, the on-disk index will have to be
scanned. The fetched seek position will be added to the key cache in this case. Based on the seek
position the row data fragment will be read from the SSTable file. The data fragments of the SSTables
and Memtables will be merged together by using the column timestamp and the requested row data
will be returned to the caller.
Gregor Roth Cassandra by example - the path of read and write requests 6

7. Processing an write request
To insert, update or delete a row Cassandra's mutate method has to be called. The listing below
shows such a mutate call by using the Hector client.
//...
// [1.b] create and perform an update
Mutator<String> mutator = HFactory.createMutator(keyspaceOperator, AsciiSerializer.get());
mutator.addInsertion("26813445", "hotel",
HFactory.createColumn("category", "5", StringSerializer.get(), StringSerializer.get()));
MutationResult result = mutator.execute();
//...
The write path is very the same to the read path. Similar to the read request a write request also
includes the required consistency level. However, the coordinator node tries to send a write request
including the mutated columns to all replica nodes for the row key.
First, the StorageProxy of the coordinator node checks if enough replica notes for the row key are
alive regarding to the requested consistency level. If this is true, the write request will be sent to the
living replica nodes. If not, an error response will be returned. Write requests to temporarily failed
replica nodes will be scheduled as a hinted handoff. This means that a hint will be written locally
instead calling the failed node. Once the failed replica node is back the hint will be sent to this node
to perform the write operation. By sending the hints the failed nodes becomes consistent to the
other nodes. Please consider that hints will not longer store locally, if the failed node is dead longer
than 1 hour (config param max_hint_window_in_ms).
The coordinator node returns the response to the client as soon as the replica nodes conforming to
the consistency level have confirmed the update (a hinted write will not count towards the
requested consistency level). The updates of the other replica nodes will still be executed in
background. If an error occurs by updating the replica nodes conforming to the consistency level, an
error response will be returned. However, in this case the already updated nodes will not be
reverted. Cassandra does not support distributed transactions, and hence it does not support a
distributed rollback.
The write operation supports an additional consistency level ANY which means that the mutated
columns have to be written to at least one node regardless of whether this node is a replica node for
the key or not. In contrast to consistency level ONE the write will also succeed, if a hinted handoff is
written (by the coordinator node). However, in this case the mutated columns will not be readable
until the responsible replica nodes have recovered.
Gregor Roth Cassandra by example - the path of read and write requests 7

8. Performing the local update
Similar to the local data query a local update is triggered by handling a message through the
MessagingService or by the StorageProxy. However, in contrast to the read path, first a commit log
entry will be written for durability reasons. By default the commit log entry will be written in
background asynchronously.
The mutated columns will also be written into the in-memory Memtable of the column family. After
inserting the changes the local update is completed.
However, the memory size of a Memtable is limited. If the max size is exceeded, the Memtable will
be written to disk as a new SSTable. This is done by a background thread which checks the current
size of all unflushed Memtables of all ColumnFamilies, periodically. If a Memtable exceeds the max
size, the background thread replaces the current Memtable by a new one. The old Memtable will be
marked as pending flush and will be flushed by another thread. Under certain circumstances several
pending Memtables for a column family could exists. After writing the Memtable to disk a new
SSTableReader referring the written SSTable is created and added to the ColumnFamilyStore. Once
written, the SSTable file is immutable. By default the SSTable data will be compressed
(SnappyCompression).
Compacting
The SSTable file includes the modified columns of the row including their timestamps as well as
additional row meta data. For instance the meta data section includes a (column name-scoped)
Bloom Filter which is used to reduce disk IO by fetching columns by name.
To reduce fragmentation and save space, SSTable files will be merged into a new SSTable file,
occasionally. This compaction will be triggered by a background thread, if the compaction threshold
is exceeded. The compaction threshold can be set for each column family.
Gregor Roth Cassandra by example - the path of read and write requests 8

9. About the author
Gregor Roth works as a software architect at United Internet group, a leading European Internet
Service Provider to which GMX, 1&1, and Web.de belong. His areas of interest include software and
system architecture, enterprise architecture management, distributed computing, and development
methodologies.
Gregor Roth Cassandra by example - the path of read and write requests 9