Stratio Crossdata is a distributed data platform that allows for both batch and streaming queries across multiple data stores. It uses Spark to enable operations not natively supported and provides connectors to integrate different data sources. The platform aims to simplify deployment, administration and querying for clients through its metadata management and support for features like full text search, joins and streaming queries.

1. Stratio Meta
An efficient distributed datahub with batch and
streaming query capabilities
Daniel Higuero
Alvaro Agea
dhiguero@stratio.com
alvaro@stratio.com
#CassandraSummit-20141"

2. Stratio Crossdata
An efficient distributed datahub with batch and
streaming query capabilities
Daniel Higuero
Alvaro Agea
dhiguero@stratio.com
alvaro@stratio.com
#CassandraSummit-20142"

3. Who are we?
STRATIO
• Stra3o-is-a-Big-Data-Company
• Founded-in-2013
• Commercially-launched-in-2014
• 50+-employees-in-Madrid
• Office-in-San-Francisco
• Cer3fied-Spark-distribu3on
#CassandraSummit-2014
3"

4. We love…
Cassandra
• P2P-architecture
• Read/write-performance
• Fault-tolerance
• Easy-to-deploy
• CQL
#CassandraSummit-2014
4"

5. • Introduction
• Crossdata architecture
• Metadata management
• Streaming sources
• Full text search
• Spark and Crossdata
• ODBC
• The future
Agenda
5"

6. Introduction
o Big-Data-analysis-is-commonly-associated-with-batch-processing
• Users-aiming-to-combine-batch-and-stream-processing-have-to-
rely-on-tailorRmade-architectures
o Users-buy-Big-Data-plaSorms,-but
• How-do-I-start?
• What-is-my-entry-point-to-the-plaSorm?
#CassandraSummit-2014
6"

7. What our clients demand?
o Easy-deployment
o Easy-administra3on
o Read/write-performance
o EasyRtoRlearn-query-language-o
Integra3on-with-BI-Tools
o Join-opera3ons
o Support-for-streaming-sources
o Integra3on-with-other-data-stores
o Ability-to-query-data-without-thinking-about-the-schema-(nonRindexed-data)
#CassandraSummit-2014
7"

8. What our clients demand?
! Easy%deployment%
! Easy%administra0on%
! Read/write%performance%
! Easy6to6learn%query%language%
o Integra3on-with-BI-Tools
o Join-opera3ons
o Support-for-streaming-sources
o Integra3on-with-other-data-stores
o Ability-to-query-data-without-thinking-about-the-schema-(nonRindexed-data)
#CassandraSummit-2014
8"

9. What our clients demand?
! Easy"deployment"
! Easy"administra8on"
! Read/write"performance"
! Easy>to>learn"query"language"
! Integra3on-with-BI-Tools
! Join-opera3ons
! Support-for-streaming-sources
! Integra3on-with-other-data-stores
! Ability-to-query-data-without-thinking-about-the-schema-(nonRindexed-data)
#CassandraSummit-2014
9"

10. Crossdata
o A-new-technology-that:
• Is-not-limited-by-the-underlying-datastore-capabili3es
• Leverages-Spark-to-perform-nonRna3vely-supported-opera3ons
• Supports-batch-and-streaming-queries
• Supports-mul3ple-clusters-and-technologies
#CassandraSummit-2014
10"

11. Our architecture
#CassandraSummit-2014
11"

12. Connecting to the outside world
o Crossdata-defines-an-IConnector-extension-interface
o User-can-easily-add-new-connectors-to-support
• Different-datastores
• Different-processing-engines
• Different-versions
o Where-each-connector-defines-its-capabili3es
#CassandraSummit-2014
12"
Our planner will choose the best connector for each query

13. Query execution
#CassandraSummit-2014
13"
Parsing" Valida8on" Planning" Execu8on"
C*"
Connector1"
Connector2"
Connector3"
Our planner will choose the best connector for each query

14. Multi-cluster support
o Stra3o-Crossdata-offers-the-possibility-of-accessing-a-single-catalog-
across-a-set-of-datastores.-
• Mul3ple-clusters-can-coexist-to-op3mize-plaSorm-performance
" E.g.,-produc3on-cluster,-test-cluster,-writeRop3mized-cluster,-
readRop3mized-cluster,-etc.-
• A-table-is-saved-in-a-unique-datastore
#CassandraSummit-2014
14"

15. Logical and physical mapping
SELECT&*&FROM&app.users;&
Users"table" Test"table" old_users"table"
#CassandraSummit-2014
15"
App"catalog"
C*"produc8on" C*"development" Other"datastores"

16. Metadata
Management
16"

17. Metadata in the era of Schemaless NoSQL datastores
o Some-datastores-are-schemaless-but-our-applica3ons-are-not!-
• Flexible-schemas-vs-Schemaless
• Crossdata-provides-a-Metadata-manager-that-stores-schemas-
for-any-datasource
" Remember-ODBC-and-those-BI-tools
"
1010010101010
1010110101010
1111010001111
?" 001000"
#CassandraSummit-2014
17"

18. Metadata management
#CassandraSummit-2014
18"
Connector"
C*"produc8on"
Metadata"Store"
Infinispan"
Metadata"Manager"
2%
Updated"metadata"
informa8on"is"
maintained"among"
Crossdata"servers"
using"Infinispan"
If"the"connector"does"
not"support"metadata"
opera8ons"those"are"
skipped" 1% 2%

19. Streaming sources
19"

20. Managing streaming sources
o Nowadays-use-cases-expect-some-type-of-streaming-datasource
• Streaming-data-has-an-ephemeral-nature
• In-Stra3o-Crossdata-we-defined-the-ephemeral-table-abstrac3on-
#CassandraSummit-2014
to-work-with-streaming-sources-as-classical-
RDBMS-tables
20"
streaming"
source"
{schema:{col1:…},…}"
col1:text" col2:int" col3:int" col4:text"
Streaming_query0"
…"
Streaming_queryn"

21. Streaming queries
o Streaming-queries-are-infinite-by-defini3on
• A-3me-window-is-defined-to-create-a-batch-like-view-of-the-rows-
ingested-by-the-system-in-that-period
• The-user-launches-queries-specifying-a-processing-3me-window
" Crossdata-provides-methods-to-list-and-stop-running-streaming-
#CassandraSummit-2014
queries
21"

22. Streaming queries: windows syntax
#CassandraSummit-2014
22"
SELECT fieldGroup,avg(Field2)
FROM eph_table
WITH WINDOW 5 minutes
WHERE field1=100 AND field2>100
GROUP BY fieldGroup;

23. Joining batch and streaming
SELECT * FROM demo.temporal
WITH WINDOW 10 secs
INNER JOIN demo.users
#CassandraSummit-2014
ON users.name = temporal.name;
SELECT * FROM
demo.temporal
WITH WINDOW 10 secs
"
SELECT *
FROM demo.users
"
INNER JOIN ON
users.name =
temporal.name
"
23"

24. Full text search
24"

25. Full text search with
o Clients-request-the-ability-to-perform-full-text-searches
o We-have-developed-an-integra3on-between-Lucene-and-
Cassandra
o C*-users-can-now-enjoy-all-Lucene-features:
• Full-text-searches,-range-queries,-fuzzy-queries….
#CassandraSummit-2014
25"
https://github.com/Stratio/stratio-cassandra

26. Stratio Lucene 2i
#CassandraSummit-2014
26"
C*"
node"
C*"
node"
Lucene"
index"
C*"
node"
Lucene"
index"
C*"
node"
Lucene"
index"
C*"
node"
Lucene"
index"
Lucene"
index"

27. Full text search queries
o With-Crossdata,-we-simplify:
• The-crea3on-syntax-
• The-query-syntax-using-the-match-operator
#CassandraSummit-2014
27"
CREATE&FULLTEXT&INDEX&ON&app.users(name,email);&
SELECT&*&FROM&app.users&&
where&email&MATCH&‘*@stratio.com’;&

28. & Stratio Crossdata
28"

29. Why Spark?
o Stra3o-Crossdata-uses-Spark-to-perform-nonRna3vely-supported-opera3ons
o Spark-brings-several-benefits-over-Hadoop-o
InRMemory-processing
o RDD-abstrac3on
o Simpler-API-o
Increased-flexibility-(e.g.,-not-need-for-iden3ty-mapping)
#CassandraSummit-2014
29"

30. What about Spark SQL?
o Different-approach-to-query-execu3on
• We-only-use-Spark-when-it-speedups-queries
" Na3ve-drivers-are-faster-for-simple-queries
" Spark-SQL-has-limited-RDD-sources
• Avoid-some-Spark-limita3ons
• Several-batch-and-streaming-contexts-in-a-single-JVM-SPARKR2243
#CassandraSummit-2014
30"

31. Query approach
SparkSQL"approach" Crossdata"approach"
#CassandraSummit-2014
SparkSQL"
Spark"
Cassandra"
Spark" Na8ve"driver"
Cassandra"
31"
Stra8o"Crossdata"

32. Our Cassandra-Spark integration
o Project-started-in-June-2013
" With-the-objec3ve-of-providing-a-method-to-interact-with-
Cassandra-from-Spark
" Ini3al-approach-based-on-the-HadoopInputFormat-interface
" Current-version-uses-the-na3ve-Datastax-Java-driver
#CassandraSummit-2014
32"
https://github.com/Stratio/stratio-deep

33. Our Cassandra-Spark integration
o Benchmark-in-process-comparing-our-solu3on-with-the-
Datastax-Spark-driver
• Results-highly-influenced-by-the-split-size
• Ini3al-results-are-promising-for-Stra3o-Spark-Integra3on-using-
Datastax-default-values
• Group-by-–-up-to-40%-faster
• Join-–-up-to-17%-faster
• Stay-tuned-for-the-benchmark-publica3on!
#CassandraSummit-2014
33"

34. Spark vs Lucene 2i
#CassandraSummit-2014
34"
Time"
Spark"
Lucen"2i"
Records/node"

35. ODBC
35"

36. Stratio Crossdata ODBC
o WellRknown-interface-standard-(for-BI-tools,-external-apps,-…)
o We-have-implemented-for-Crossdata-using-Simba-SDK
o ODBC-opens-the-full-poten3al-of-Stra3o-Crossdata-to-the-external-
world
o Currently-tested-with-Tableau,-Qlikview-and-MS-Excel
#CassandraSummit-2014
36"
One ODBC for all datastores!

37. The future
37"

38. The future
o Security
o Query-op3mizer-and-smart-query-planner
o Leverage-system-sta3s3cs
o Support-for-UDFs
o Become-an-Apache-project
#CassandraSummit-2014
38"
https://github.com/Stratio/stratio-meta

39. We are looking for an Apache Champion
#CassandraSummit-2014
39"
Can"you"
help"us?"

40. A wish list for Cassandra
o Ability-to-stop-running-queries
o Interac3ve-users-are-unpredictable
o Some-excep3on-paths-are-not-clear-or-defined-(e.g.,-secondary-indexes)
o Distribute-some-of-the-opera3ons-currently-performed-on-the-coordinator
• E.g.,-aggrega3ons-like-count(*)
#CassandraSummit-2014
40"

41. Stratio Crossdata
An efficient distributed datahub with batch and
streaming query capabilities
Daniel Higuero
Alvaro Agea
dhiguero@stratio.com
alvaro@stratio.com
#CassandraSummit-201441"