SlideShare a Scribd company logo
1 of 36
Download to read offline
Using SPIDER for sharding in production
Kentoku SHIBA
Stephane Varoqui
Kayoko Goto
0. what is SPIDER?
1. why SPIDER? what SPIDER can do for you?
2. when SPIDER is right for you?
what cases should you use SPIDER?
3. how long is SPIDER used in the big environment?
4. SPIDER sharding architecture
5. how to get SPIDER working?
6. multi dimensional sharding technique with VP storage engine
7. roadmap of SPIDER
8. where to get SPIDER (with VP)
Agenda
What is Spider
What is the Spider Storage Engine?
Spider is a sharding solution and proxying
solution. Spider Storage Engine is a
plugin of MariaDB/MySQL. Spider tables
can be used to federate from other servers
MariaDB/MySQL/OracleDB tables as if they
stand on local server. And Spider can
create database sharding by using table
partitioning feature.
What is the Spider Storage Engine?
Spider is bundled in MariaDB
from 10.0.4.
Why SPIDER?
What SPIDER can do for you?
Why Spider? What Spider can do for you?
For federation
You can attach tables from other servers or
from local server by using Spider.
For sharding
You can divide huge tables and huge
traffics to multiple servers by using Spider.
SPIDER
(MySQL/MariaDB)
SPIDER
(MySQL/MariaDB)
Spider structure sample of database sharding
DB1
tbl_a1
1.Request
2.Just connect to spider
3.Response
DB2 DB3
AP
SPIDER
(MySQL/MariaDB)
An application can use all databases
by only connecting to one database.
APAP AP AP
tbl_a2 tbl_a3
Why Spider? What Spider can do for you?
Spider supports both cross shard join
and now join push down (direct join).
DEMO
1.cross shard join without batched key access
2.cross shard join with batched key access
3.join push down
When SPIDER is right for you?
What cases should you use SPIDER?
When SPIDER is right for you? What cases should you use SPIDER?
You should better use Spider
1.when you have 2 or more services and
the services needs to use data of
other services.
2.when you need scaling out for huge data
or write traffics.
When SPIDER is right for you? What cases should you use SPIDER?
You should better use Spider
3.Unless some big data solutions you can
benefit indexing on shards.
4.You need sharding and consistency.
How long is SPIDER used
in the big environment?
How long is SPIDER used in the big environment?
Galaxy Semiconductor
They handle 200 Billion records per
quarter on 3 Spider nodes and 4 data
nodes.
They use this cluster for data quality
analytics.
How long is SPIDER used in the big environment?
Tencent Games
They handle 100TB datas on 396 Spider
nodes and 2800 data nodes.
They use this cluster for produce online
games.
SPIDER sharding architecture
SPIDER sharding architecture
Spider stores partitions on different servers.
This sharding design is done using the
database native table partitioning.
You can federate multiple servers for the
same partition to bring HA and load
balancing per partition.
When writing multiple replicates or on
multiple servers, Spider use 2 phase
commit to preserve consistency.
How to get SPIDER working?
How to get SPIDER working? (1/5)
1. Install Spider bundled with
MariaDB/MySQL.
2. Login MariaDB/MySQL then
install Spider as a plugin.
(execute install_spider.sql)
3. Create Spider table.
How to get SPIDER working? (2/5)
Create one to one Spider table.
CREATE TABLE t1(
c1 int,
c2 varchar(100),
PRIMARY KEY(c1)
)ENGINE=spider DEFAULT CHARSET=utf8
COMMENT '
table "rt1", database "test", port "3306",
host "host name of data node",
user "user name for data node",
password "password for data node"
';
Set engine name to “Spider” and write connect
information (and parameter) in the comment.
How to get SPIDER working? (3/5)
You can create Spider tables without column definitions in
MariaDB. In this case Spider gets the column definition from
data node.
CREATE TABLE t1
ENGINE=spider DEFAULT CHARSET=utf8
COMMENT '
table "rt1", database "test", port "3306",
host "host name of data node",
user "user name for data node",
password "password for data node"
';
How to get SPIDER working? (4/5)
Create one to many (sharding) Spider table
CREATE TABLE t1(
c1 int,
c2 varchar(100),
PRIMARY KEY(c1)
)ENGINE=spider DEFAULT CHARSET=utf8
COMMENT 'table "rt1", database "test", port "3306",
user "user name for data node", password "password for data node"'
PARTITION BY RANGE(c1) (
PARTITION p0 VALUES LESS THAN (100000) COMMENT 'host "h1"',
PARTITION p1 VALUES LESS THAN (200000) COMMENT 'host "h2"',
PARTITION p2 VALUES LESS THAN (300000) COMMENT 'host "h3"',
PARTITION p3 VALUES LESS THAN MAXVALUE COMMENT 'host "h4"'
);
Write shared connect information to table comment,
shard specific connect information to partition comment.
How to get SPIDER working? (5/5)
You can use “CREATE SERVER” statement for defining
connection information.
CREATE SERVER srv1
FOREIGN DATA WRAPPER mysql
HOST 'host name of data node',
DATABASE 'test',
USER 'user name for data node',
PASSWORD 'password for data node',
PORT 3306
;
You can use create server definition by writing “server” parameter
into table/partition comment.
CREATE TABLE t1(
c1 int,
c2 varchar(100),
PRIMARY KEY(c1)
)ENGINE=spider DEFAULT CHARSET=utf8
COMMENT 'table "rt1", server "srv1"';
Multi dimensional sharding technique
with VP storage engine
Multi dimenstional sharding technique with VP storage engine
VP means Vertical Partitioning.
VP merges multiple child tables into a single
View. VP chooses efficiently child tables for
Each query. So, when you use sharded
Spider tables which have different
partitioning rules for VP child tables, VP
chooses sharded Spider tables efficiently.
Structure sample of using different sharding rules (1/2)
DB2
tbl_a
1.Request 3.Response
DB3
tbl_a
DB4
tbl_a
APAPAP AP AP
DB1
tbl_a (vp)
DB5
tbl_a
Partition
by col_a
tbl_a1(spider)
Partition
by col_b
tbl_a2(spider)
select … from tbl_a where col_a = 1
Structure sample of using different sharding rules (2/2)
DB2
tbl_a
1.Request 3.Response
DB3
tbl_a
DB4
tbl_a
APAPAP AP AP
DB1
tbl_a (vp)
DB5
tbl_a
Partition
by col_a
tbl_a1(spider)
Partition
by col_b
tbl_a2(spider)
select … from tbl_a where col_b = 1
Roadmap of SPIDER
Roadmap of SPIDER
Developed new fearures of Spider this year
- Recovering statistics information at startup tables.
(Improving startup phase of MariaDB)
- Direct join on data node part1. (not include partitioned
table and redundant table)
- Direct join on data node part2. (for redundant table)
binary
http://spiderformysql.com/downloads/spider-3.3/mariadb-10.2.0-spider-
3.3-vp-1.1-linux-x86_64-glibc25.tgz
source code
http://spiderformysql.com/downloads/spider-3.3/mariadb-10.2.0-spider-
3.3-vp-1.1.tgz
Roadmap of SPIDER
Autumn 2016
- Reducing number of threads for collecting statistics
information from data nodes. (Reduce memory usage)
Winter 2016
- Auto repair broken spider system table.
- Direct join on data node part3. (for partitioned table)
Roadmap of SPIDER
Spring 2017
- Auto XA recovery and commit/rollback on startup phase of
MariaDB.
- Direct join on data node part4. (for VP table)
Where to get SPIDER (with VP)
Where to get SPIDER (with VP)
Spider is merge with MariaDB on regular basis
since 10.0. But some changes needs
modification in the sql layer and partitioning
layer and VP is not bundled yet for that reason
we advice to use our MariaDB branch on
github. You get more features like parallel
query, BKA join on partition table, many push
down like aggregate query and joins.
https://github.com/Kentoku/Spider
We do provide binaries and packages on request.
support@spiderformysql.com
One more thing to
Announce!!
Just want to say…..
WE HAVE NEW
BRANCH OFFICE
in
AMSTERDAM!!

More Related Content

What's hot

Pig with Cassandra: Adventures in Analytics
Pig with Cassandra: Adventures in AnalyticsPig with Cassandra: Adventures in Analytics
Pig with Cassandra: Adventures in AnalyticsJeremy Hanna
 
Heuritech: Apache Spark REX
Heuritech: Apache Spark REXHeuritech: Apache Spark REX
Heuritech: Apache Spark REXdidmarin
 
Cassandra for Sysadmins
Cassandra for SysadminsCassandra for Sysadmins
Cassandra for SysadminsNathan Milford
 
Introduction to Real-Time Analytics with Cassandra and Hadoop
Introduction to Real-Time Analytics with Cassandra and HadoopIntroduction to Real-Time Analytics with Cassandra and Hadoop
Introduction to Real-Time Analytics with Cassandra and HadoopPatricia Gorla
 
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016DataStax
 
An introduction to Big-Data processing applying hadoop
An introduction to Big-Data processing applying hadoopAn introduction to Big-Data processing applying hadoop
An introduction to Big-Data processing applying hadoopAmir Sedighi
 
Powering a Graph Data System with Scylla + JanusGraph
Powering a Graph Data System with Scylla + JanusGraphPowering a Graph Data System with Scylla + JanusGraph
Powering a Graph Data System with Scylla + JanusGraphScyllaDB
 
Replicating in Real-time from MySQL to Amazon Redshift
Replicating in Real-time from MySQL to Amazon RedshiftReplicating in Real-time from MySQL to Amazon Redshift
Replicating in Real-time from MySQL to Amazon RedshiftContinuent
 
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)Gruter
 
Hadoop Tutorial
Hadoop TutorialHadoop Tutorial
Hadoop Tutorialawesomesos
 
Brisk hadoop june2011_sfjava
Brisk hadoop june2011_sfjavaBrisk hadoop june2011_sfjava
Brisk hadoop june2011_sfjavasrisatish ambati
 
OpenExpo: MySQL, Where are you going?
OpenExpo: MySQL, Where are you going?OpenExpo: MySQL, Where are you going?
OpenExpo: MySQL, Where are you going?FromDual GmbH
 
Apache Cassandra Interview Questions and Answers | Cassandra Tutorial | Cassa...
Apache Cassandra Interview Questions and Answers | Cassandra Tutorial | Cassa...Apache Cassandra Interview Questions and Answers | Cassandra Tutorial | Cassa...
Apache Cassandra Interview Questions and Answers | Cassandra Tutorial | Cassa...Edureka!
 
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax
 
Tobi Bosede - PyCassa Setting Up and Using Apache Cassandra with Python in Wi...
Tobi Bosede - PyCassa Setting Up and Using Apache Cassandra with Python in Wi...Tobi Bosede - PyCassa Setting Up and Using Apache Cassandra with Python in Wi...
Tobi Bosede - PyCassa Setting Up and Using Apache Cassandra with Python in Wi...PyData
 
Introduction to NoSQL & Apache Cassandra
Introduction to NoSQL & Apache CassandraIntroduction to NoSQL & Apache Cassandra
Introduction to NoSQL & Apache CassandraChetan Baheti
 

What's hot (20)

Pig with Cassandra: Adventures in Analytics
Pig with Cassandra: Adventures in AnalyticsPig with Cassandra: Adventures in Analytics
Pig with Cassandra: Adventures in Analytics
 
Heuritech: Apache Spark REX
Heuritech: Apache Spark REXHeuritech: Apache Spark REX
Heuritech: Apache Spark REX
 
Cassandra for Sysadmins
Cassandra for SysadminsCassandra for Sysadmins
Cassandra for Sysadmins
 
Introduction to Real-Time Analytics with Cassandra and Hadoop
Introduction to Real-Time Analytics with Cassandra and HadoopIntroduction to Real-Time Analytics with Cassandra and Hadoop
Introduction to Real-Time Analytics with Cassandra and Hadoop
 
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
 
An introduction to Big-Data processing applying hadoop
An introduction to Big-Data processing applying hadoopAn introduction to Big-Data processing applying hadoop
An introduction to Big-Data processing applying hadoop
 
Powering a Graph Data System with Scylla + JanusGraph
Powering a Graph Data System with Scylla + JanusGraphPowering a Graph Data System with Scylla + JanusGraph
Powering a Graph Data System with Scylla + JanusGraph
 
Replicating in Real-time from MySQL to Amazon Redshift
Replicating in Real-time from MySQL to Amazon RedshiftReplicating in Real-time from MySQL to Amazon Redshift
Replicating in Real-time from MySQL to Amazon Redshift
 
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
 
Hadoop description
Hadoop descriptionHadoop description
Hadoop description
 
Hadoop Tutorial
Hadoop TutorialHadoop Tutorial
Hadoop Tutorial
 
Hadoop sqoop
Hadoop sqoop Hadoop sqoop
Hadoop sqoop
 
Brisk hadoop june2011_sfjava
Brisk hadoop june2011_sfjavaBrisk hadoop june2011_sfjava
Brisk hadoop june2011_sfjava
 
Brisk hadoop june2011
Brisk hadoop june2011Brisk hadoop june2011
Brisk hadoop june2011
 
OpenExpo: MySQL, Where are you going?
OpenExpo: MySQL, Where are you going?OpenExpo: MySQL, Where are you going?
OpenExpo: MySQL, Where are you going?
 
Apache Cassandra Interview Questions and Answers | Cassandra Tutorial | Cassa...
Apache Cassandra Interview Questions and Answers | Cassandra Tutorial | Cassa...Apache Cassandra Interview Questions and Answers | Cassandra Tutorial | Cassa...
Apache Cassandra Interview Questions and Answers | Cassandra Tutorial | Cassa...
 
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
 
OpenStack Manila 紹介
OpenStack Manila 紹介OpenStack Manila 紹介
OpenStack Manila 紹介
 
Tobi Bosede - PyCassa Setting Up and Using Apache Cassandra with Python in Wi...
Tobi Bosede - PyCassa Setting Up and Using Apache Cassandra with Python in Wi...Tobi Bosede - PyCassa Setting Up and Using Apache Cassandra with Python in Wi...
Tobi Bosede - PyCassa Setting Up and Using Apache Cassandra with Python in Wi...
 
Introduction to NoSQL & Apache Cassandra
Introduction to NoSQL & Apache CassandraIntroduction to NoSQL & Apache Cassandra
Introduction to NoSQL & Apache Cassandra
 

Viewers also liked

Spider storage engine (dec212016)
Spider storage engine (dec212016)Spider storage engine (dec212016)
Spider storage engine (dec212016)Kentoku
 
Spiderストレージエンジンのご紹介
Spiderストレージエンジンのご紹介Spiderストレージエンジンのご紹介
Spiderストレージエンジンのご紹介Kentoku
 
MariaDB ColumnStore 20160721
MariaDB ColumnStore 20160721MariaDB ColumnStore 20160721
MariaDB ColumnStore 20160721Kentoku
 
Sharding with spider solutions 20160721
Sharding with spider solutions 20160721Sharding with spider solutions 20160721
Sharding with spider solutions 20160721Kentoku
 
Spider HA 20100922(DTT#7)
Spider HA 20100922(DTT#7)Spider HA 20100922(DTT#7)
Spider HA 20100922(DTT#7)Kentoku
 
Advanced Sharding Techniques with Spider (MUC2010)
Advanced Sharding Techniques with Spider (MUC2010)Advanced Sharding Techniques with Spider (MUC2010)
Advanced Sharding Techniques with Spider (MUC2010)Kentoku
 
The New MariaDB Offering: MariaDB 10, MaxScale and More
The New MariaDB Offering: MariaDB 10, MaxScale and MoreThe New MariaDB Offering: MariaDB 10, MaxScale and More
The New MariaDB Offering: MariaDB 10, MaxScale and MoreMariaDB Corporation
 
Droonga - 分散Groongaで快適レプリケーション生活
Droonga - 分散Groongaで快適レプリケーション生活Droonga - 分散Groongaで快適レプリケーション生活
Droonga - 分散Groongaで快適レプリケーション生活Hiroshi Yuki
 
Postgres on OpenStack
Postgres on OpenStackPostgres on OpenStack
Postgres on OpenStackEDB
 
Clash of clans data structures
Clash of clans   data structuresClash of clans   data structures
Clash of clans data structuresRan Silberman
 
From a kafkaesque story to The Promised Land
From a kafkaesque story to The Promised LandFrom a kafkaesque story to The Promised Land
From a kafkaesque story to The Promised LandRan Silberman
 
Get More Out of MySQL with TokuDB
Get More Out of MySQL with TokuDBGet More Out of MySQL with TokuDB
Get More Out of MySQL with TokuDBTim Callaghan
 
Spiderの最新動向 20130419
Spiderの最新動向 20130419Spiderの最新動向 20130419
Spiderの最新動向 20130419Kentoku
 
PHP on IBM i Tutorial
PHP on IBM i TutorialPHP on IBM i Tutorial
PHP on IBM i TutorialZendCon
 
Oracle Compute Cloud Service快速实践
Oracle Compute Cloud Service快速实践Oracle Compute Cloud Service快速实践
Oracle Compute Cloud Service快速实践Zhaoyang Wang
 
MySQL Manchester TT - Replication Features
MySQL Manchester TT  - Replication FeaturesMySQL Manchester TT  - Replication Features
MySQL Manchester TT - Replication FeaturesMark Swarbrick
 
A Storage Story #ChefConf2013
A Storage Story #ChefConf2013A Storage Story #ChefConf2013
A Storage Story #ChefConf2013Kyle Bader
 
Tiery Eyed
Tiery EyedTiery Eyed
Tiery EyedZendCon
 
Framework Shootout
Framework ShootoutFramework Shootout
Framework ShootoutZendCon
 
MySQL Manchester TT - Security
MySQL Manchester TT  - SecurityMySQL Manchester TT  - Security
MySQL Manchester TT - SecurityMark Swarbrick
 

Viewers also liked (20)

Spider storage engine (dec212016)
Spider storage engine (dec212016)Spider storage engine (dec212016)
Spider storage engine (dec212016)
 
Spiderストレージエンジンのご紹介
Spiderストレージエンジンのご紹介Spiderストレージエンジンのご紹介
Spiderストレージエンジンのご紹介
 
MariaDB ColumnStore 20160721
MariaDB ColumnStore 20160721MariaDB ColumnStore 20160721
MariaDB ColumnStore 20160721
 
Sharding with spider solutions 20160721
Sharding with spider solutions 20160721Sharding with spider solutions 20160721
Sharding with spider solutions 20160721
 
Spider HA 20100922(DTT#7)
Spider HA 20100922(DTT#7)Spider HA 20100922(DTT#7)
Spider HA 20100922(DTT#7)
 
Advanced Sharding Techniques with Spider (MUC2010)
Advanced Sharding Techniques with Spider (MUC2010)Advanced Sharding Techniques with Spider (MUC2010)
Advanced Sharding Techniques with Spider (MUC2010)
 
The New MariaDB Offering: MariaDB 10, MaxScale and More
The New MariaDB Offering: MariaDB 10, MaxScale and MoreThe New MariaDB Offering: MariaDB 10, MaxScale and More
The New MariaDB Offering: MariaDB 10, MaxScale and More
 
Droonga - 分散Groongaで快適レプリケーション生活
Droonga - 分散Groongaで快適レプリケーション生活Droonga - 分散Groongaで快適レプリケーション生活
Droonga - 分散Groongaで快適レプリケーション生活
 
Postgres on OpenStack
Postgres on OpenStackPostgres on OpenStack
Postgres on OpenStack
 
Clash of clans data structures
Clash of clans   data structuresClash of clans   data structures
Clash of clans data structures
 
From a kafkaesque story to The Promised Land
From a kafkaesque story to The Promised LandFrom a kafkaesque story to The Promised Land
From a kafkaesque story to The Promised Land
 
Get More Out of MySQL with TokuDB
Get More Out of MySQL with TokuDBGet More Out of MySQL with TokuDB
Get More Out of MySQL with TokuDB
 
Spiderの最新動向 20130419
Spiderの最新動向 20130419Spiderの最新動向 20130419
Spiderの最新動向 20130419
 
PHP on IBM i Tutorial
PHP on IBM i TutorialPHP on IBM i Tutorial
PHP on IBM i Tutorial
 
Oracle Compute Cloud Service快速实践
Oracle Compute Cloud Service快速实践Oracle Compute Cloud Service快速实践
Oracle Compute Cloud Service快速实践
 
MySQL Manchester TT - Replication Features
MySQL Manchester TT  - Replication FeaturesMySQL Manchester TT  - Replication Features
MySQL Manchester TT - Replication Features
 
A Storage Story #ChefConf2013
A Storage Story #ChefConf2013A Storage Story #ChefConf2013
A Storage Story #ChefConf2013
 
Tiery Eyed
Tiery EyedTiery Eyed
Tiery Eyed
 
Framework Shootout
Framework ShootoutFramework Shootout
Framework Shootout
 
MySQL Manchester TT - Security
MySQL Manchester TT  - SecurityMySQL Manchester TT  - Security
MySQL Manchester TT - Security
 

Similar to Using spider for sharding in production

Newest topic of spider 20131016 in Buenos Aires Argentina
Newest topic of spider 20131016 in Buenos Aires ArgentinaNewest topic of spider 20131016 in Buenos Aires Argentina
Newest topic of spider 20131016 in Buenos Aires ArgentinaKentoku
 
Apache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupApache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupNed Shawa
 
Spark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production usersSpark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production usersDatabricks
 
How to migrate_to_sharding_with_spider
How to migrate_to_sharding_with_spiderHow to migrate_to_sharding_with_spider
How to migrate_to_sharding_with_spiderKentoku
 
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...Josef A. Habdank
 
M|18 How MariaDB Server Scales with Spider
M|18 How MariaDB Server Scales with SpiderM|18 How MariaDB Server Scales with Spider
M|18 How MariaDB Server Scales with SpiderMariaDB plc
 
Transparent sharding with Spider: what's new and getting started
Transparent sharding with Spider: what's new and getting startedTransparent sharding with Spider: what's new and getting started
Transparent sharding with Spider: what's new and getting startedMariaDB plc
 
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...Alex Levenson
 
Cloudera Impala technical deep dive
Cloudera Impala technical deep diveCloudera Impala technical deep dive
Cloudera Impala technical deep divehuguk
 
Orcl siebel-sun-s282213-oow2006
Orcl siebel-sun-s282213-oow2006Orcl siebel-sun-s282213-oow2006
Orcl siebel-sun-s282213-oow2006Sal Marcus
 
15 Ways to Kill Your Mysql Application Performance
15 Ways to Kill Your Mysql Application Performance15 Ways to Kill Your Mysql Application Performance
15 Ways to Kill Your Mysql Application Performanceguest9912e5
 
Hadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at TwitterHadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at TwitterDataWorks Summit
 
O13 024-sparc-t5-architecture-1920540
O13 024-sparc-t5-architecture-1920540O13 024-sparc-t5-architecture-1920540
O13 024-sparc-t5-architecture-1920540Ory Chhean
 
Collaborate07kmohiuddin
Collaborate07kmohiuddinCollaborate07kmohiuddin
Collaborate07kmohiuddinSal Marcus
 
Apache Spark Performance tuning and Best Practise
Apache Spark Performance tuning and Best PractiseApache Spark Performance tuning and Best Practise
Apache Spark Performance tuning and Best PractiseKnoldus Inc.
 
Spark cassandra connector.API, Best Practices and Use-Cases
Spark cassandra connector.API, Best Practices and Use-CasesSpark cassandra connector.API, Best Practices and Use-Cases
Spark cassandra connector.API, Best Practices and Use-CasesDuyhai Doan
 
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and DatabricksJump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and DatabricksDatabricks
 
Spark Performance Tuning .pdf
Spark Performance Tuning .pdfSpark Performance Tuning .pdf
Spark Performance Tuning .pdfAmit Raj
 
Introduction to Apache Spark
Introduction to Apache Spark Introduction to Apache Spark
Introduction to Apache Spark Hubert Fan Chiang
 

Similar to Using spider for sharding in production (20)

Newest topic of spider 20131016 in Buenos Aires Argentina
Newest topic of spider 20131016 in Buenos Aires ArgentinaNewest topic of spider 20131016 in Buenos Aires Argentina
Newest topic of spider 20131016 in Buenos Aires Argentina
 
Apache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetupApache spark-melbourne-april-2015-meetup
Apache spark-melbourne-april-2015-meetup
 
Spark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production usersSpark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production users
 
How to migrate_to_sharding_with_spider
How to migrate_to_sharding_with_spiderHow to migrate_to_sharding_with_spider
How to migrate_to_sharding_with_spider
 
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
 
M|18 How MariaDB Server Scales with Spider
M|18 How MariaDB Server Scales with SpiderM|18 How MariaDB Server Scales with Spider
M|18 How MariaDB Server Scales with Spider
 
Transparent sharding with Spider: what's new and getting started
Transparent sharding with Spider: what's new and getting startedTransparent sharding with Spider: what's new and getting started
Transparent sharding with Spider: what's new and getting started
 
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...
 
Cloudera Impala technical deep dive
Cloudera Impala technical deep diveCloudera Impala technical deep dive
Cloudera Impala technical deep dive
 
Orcl siebel-sun-s282213-oow2006
Orcl siebel-sun-s282213-oow2006Orcl siebel-sun-s282213-oow2006
Orcl siebel-sun-s282213-oow2006
 
15 Ways to Kill Your Mysql Application Performance
15 Ways to Kill Your Mysql Application Performance15 Ways to Kill Your Mysql Application Performance
15 Ways to Kill Your Mysql Application Performance
 
Hadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at TwitterHadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at Twitter
 
O13 024-sparc-t5-architecture-1920540
O13 024-sparc-t5-architecture-1920540O13 024-sparc-t5-architecture-1920540
O13 024-sparc-t5-architecture-1920540
 
Collaborate07kmohiuddin
Collaborate07kmohiuddinCollaborate07kmohiuddin
Collaborate07kmohiuddin
 
Apache Spark Performance tuning and Best Practise
Apache Spark Performance tuning and Best PractiseApache Spark Performance tuning and Best Practise
Apache Spark Performance tuning and Best Practise
 
Spark cassandra connector.API, Best Practices and Use-Cases
Spark cassandra connector.API, Best Practices and Use-CasesSpark cassandra connector.API, Best Practices and Use-Cases
Spark cassandra connector.API, Best Practices and Use-Cases
 
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and DatabricksJump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and Databricks
 
Spark Performance Tuning .pdf
Spark Performance Tuning .pdfSpark Performance Tuning .pdf
Spark Performance Tuning .pdf
 
Introduction to Apache Spark
Introduction to Apache Spark Introduction to Apache Spark
Introduction to Apache Spark
 
Spark1
Spark1Spark1
Spark1
 

More from Kentoku

An issue of all slaves stop replication
An issue of all slaves stop replicationAn issue of all slaves stop replication
An issue of all slaves stop replicationKentoku
 
MariaDB 10.3から利用できるSpider関連の性能向上機能・便利機能ほか
MariaDB 10.3から利用できるSpider関連の性能向上機能・便利機能ほかMariaDB 10.3から利用できるSpider関連の性能向上機能・便利機能ほか
MariaDB 10.3から利用できるSpider関連の性能向上機能・便利機能ほかKentoku
 
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介Kentoku
 
Mroonga 20141129
Mroonga 20141129Mroonga 20141129
Mroonga 20141129Kentoku
 
MariaDB Spider Mroonga 20140218
MariaDB Spider Mroonga 20140218MariaDB Spider Mroonga 20140218
MariaDB Spider Mroonga 20140218Kentoku
 
Mroonga 20131129
Mroonga 20131129Mroonga 20131129
Mroonga 20131129Kentoku
 
Spiderの最新動向 20131009
Spiderの最新動向 20131009Spiderの最新動向 20131009
Spiderの最新動向 20131009Kentoku
 
Mroonga 20121129
Mroonga 20121129Mroonga 20121129
Mroonga 20121129Kentoku
 
Mroonga unsupported feature_20111129
Mroonga unsupported feature_20111129Mroonga unsupported feature_20111129
Mroonga unsupported feature_20111129Kentoku
 
Introducing mroonga 20111129
Introducing mroonga 20111129Introducing mroonga 20111129
Introducing mroonga 20111129Kentoku
 
hs_spider_hs_something_20110906
hs_spider_hs_something_20110906hs_spider_hs_something_20110906
hs_spider_hs_something_20110906Kentoku
 
Charms of MySQL 20101206(DTT#7)
Charms of MySQL 20101206(DTT#7)Charms of MySQL 20101206(DTT#7)
Charms of MySQL 20101206(DTT#7)Kentoku
 
Introducing Spider 20101206(DTT#7)
Introducing Spider 20101206(DTT#7)Introducing Spider 20101206(DTT#7)
Introducing Spider 20101206(DTT#7)Kentoku
 
Spider DeNA Technology Seminar #2
Spider DeNA Technology Seminar #2Spider DeNA Technology Seminar #2
Spider DeNA Technology Seminar #2Kentoku
 
Spider Performance Test(Bench Mark04242009)
Spider Performance Test(Bench Mark04242009)Spider Performance Test(Bench Mark04242009)
Spider Performance Test(Bench Mark04242009)Kentoku
 
Spider Shibuya.pm #12
Spider Shibuya.pm #12Spider Shibuya.pm #12
Spider Shibuya.pm #12Kentoku
 

More from Kentoku (16)

An issue of all slaves stop replication
An issue of all slaves stop replicationAn issue of all slaves stop replication
An issue of all slaves stop replication
 
MariaDB 10.3から利用できるSpider関連の性能向上機能・便利機能ほか
MariaDB 10.3から利用できるSpider関連の性能向上機能・便利機能ほかMariaDB 10.3から利用できるSpider関連の性能向上機能・便利機能ほか
MariaDB 10.3から利用できるSpider関連の性能向上機能・便利機能ほか
 
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
Spiderストレージエンジンの使い方と利用事例 他ストレージエンジンの紹介
 
Mroonga 20141129
Mroonga 20141129Mroonga 20141129
Mroonga 20141129
 
MariaDB Spider Mroonga 20140218
MariaDB Spider Mroonga 20140218MariaDB Spider Mroonga 20140218
MariaDB Spider Mroonga 20140218
 
Mroonga 20131129
Mroonga 20131129Mroonga 20131129
Mroonga 20131129
 
Spiderの最新動向 20131009
Spiderの最新動向 20131009Spiderの最新動向 20131009
Spiderの最新動向 20131009
 
Mroonga 20121129
Mroonga 20121129Mroonga 20121129
Mroonga 20121129
 
Mroonga unsupported feature_20111129
Mroonga unsupported feature_20111129Mroonga unsupported feature_20111129
Mroonga unsupported feature_20111129
 
Introducing mroonga 20111129
Introducing mroonga 20111129Introducing mroonga 20111129
Introducing mroonga 20111129
 
hs_spider_hs_something_20110906
hs_spider_hs_something_20110906hs_spider_hs_something_20110906
hs_spider_hs_something_20110906
 
Charms of MySQL 20101206(DTT#7)
Charms of MySQL 20101206(DTT#7)Charms of MySQL 20101206(DTT#7)
Charms of MySQL 20101206(DTT#7)
 
Introducing Spider 20101206(DTT#7)
Introducing Spider 20101206(DTT#7)Introducing Spider 20101206(DTT#7)
Introducing Spider 20101206(DTT#7)
 
Spider DeNA Technology Seminar #2
Spider DeNA Technology Seminar #2Spider DeNA Technology Seminar #2
Spider DeNA Technology Seminar #2
 
Spider Performance Test(Bench Mark04242009)
Spider Performance Test(Bench Mark04242009)Spider Performance Test(Bench Mark04242009)
Spider Performance Test(Bench Mark04242009)
 
Spider Shibuya.pm #12
Spider Shibuya.pm #12Spider Shibuya.pm #12
Spider Shibuya.pm #12
 

Recently uploaded

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 

Recently uploaded (20)

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 

Using spider for sharding in production

  • 1. Using SPIDER for sharding in production Kentoku SHIBA Stephane Varoqui Kayoko Goto
  • 2. 0. what is SPIDER? 1. why SPIDER? what SPIDER can do for you? 2. when SPIDER is right for you? what cases should you use SPIDER? 3. how long is SPIDER used in the big environment? 4. SPIDER sharding architecture 5. how to get SPIDER working? 6. multi dimensional sharding technique with VP storage engine 7. roadmap of SPIDER 8. where to get SPIDER (with VP) Agenda
  • 4. What is the Spider Storage Engine? Spider is a sharding solution and proxying solution. Spider Storage Engine is a plugin of MariaDB/MySQL. Spider tables can be used to federate from other servers MariaDB/MySQL/OracleDB tables as if they stand on local server. And Spider can create database sharding by using table partitioning feature.
  • 5. What is the Spider Storage Engine? Spider is bundled in MariaDB from 10.0.4.
  • 6. Why SPIDER? What SPIDER can do for you?
  • 7. Why Spider? What Spider can do for you? For federation You can attach tables from other servers or from local server by using Spider. For sharding You can divide huge tables and huge traffics to multiple servers by using Spider.
  • 8. SPIDER (MySQL/MariaDB) SPIDER (MySQL/MariaDB) Spider structure sample of database sharding DB1 tbl_a1 1.Request 2.Just connect to spider 3.Response DB2 DB3 AP SPIDER (MySQL/MariaDB) An application can use all databases by only connecting to one database. APAP AP AP tbl_a2 tbl_a3
  • 9. Why Spider? What Spider can do for you? Spider supports both cross shard join and now join push down (direct join). DEMO 1.cross shard join without batched key access 2.cross shard join with batched key access 3.join push down
  • 10. When SPIDER is right for you? What cases should you use SPIDER?
  • 11. When SPIDER is right for you? What cases should you use SPIDER? You should better use Spider 1.when you have 2 or more services and the services needs to use data of other services. 2.when you need scaling out for huge data or write traffics.
  • 12. When SPIDER is right for you? What cases should you use SPIDER? You should better use Spider 3.Unless some big data solutions you can benefit indexing on shards. 4.You need sharding and consistency.
  • 13. How long is SPIDER used in the big environment?
  • 14. How long is SPIDER used in the big environment? Galaxy Semiconductor They handle 200 Billion records per quarter on 3 Spider nodes and 4 data nodes. They use this cluster for data quality analytics.
  • 15. How long is SPIDER used in the big environment? Tencent Games They handle 100TB datas on 396 Spider nodes and 2800 data nodes. They use this cluster for produce online games.
  • 17. SPIDER sharding architecture Spider stores partitions on different servers. This sharding design is done using the database native table partitioning. You can federate multiple servers for the same partition to bring HA and load balancing per partition. When writing multiple replicates or on multiple servers, Spider use 2 phase commit to preserve consistency.
  • 18. How to get SPIDER working?
  • 19. How to get SPIDER working? (1/5) 1. Install Spider bundled with MariaDB/MySQL. 2. Login MariaDB/MySQL then install Spider as a plugin. (execute install_spider.sql) 3. Create Spider table.
  • 20. How to get SPIDER working? (2/5) Create one to one Spider table. CREATE TABLE t1( c1 int, c2 varchar(100), PRIMARY KEY(c1) )ENGINE=spider DEFAULT CHARSET=utf8 COMMENT ' table "rt1", database "test", port "3306", host "host name of data node", user "user name for data node", password "password for data node" '; Set engine name to “Spider” and write connect information (and parameter) in the comment.
  • 21. How to get SPIDER working? (3/5) You can create Spider tables without column definitions in MariaDB. In this case Spider gets the column definition from data node. CREATE TABLE t1 ENGINE=spider DEFAULT CHARSET=utf8 COMMENT ' table "rt1", database "test", port "3306", host "host name of data node", user "user name for data node", password "password for data node" ';
  • 22. How to get SPIDER working? (4/5) Create one to many (sharding) Spider table CREATE TABLE t1( c1 int, c2 varchar(100), PRIMARY KEY(c1) )ENGINE=spider DEFAULT CHARSET=utf8 COMMENT 'table "rt1", database "test", port "3306", user "user name for data node", password "password for data node"' PARTITION BY RANGE(c1) ( PARTITION p0 VALUES LESS THAN (100000) COMMENT 'host "h1"', PARTITION p1 VALUES LESS THAN (200000) COMMENT 'host "h2"', PARTITION p2 VALUES LESS THAN (300000) COMMENT 'host "h3"', PARTITION p3 VALUES LESS THAN MAXVALUE COMMENT 'host "h4"' ); Write shared connect information to table comment, shard specific connect information to partition comment.
  • 23. How to get SPIDER working? (5/5) You can use “CREATE SERVER” statement for defining connection information. CREATE SERVER srv1 FOREIGN DATA WRAPPER mysql HOST 'host name of data node', DATABASE 'test', USER 'user name for data node', PASSWORD 'password for data node', PORT 3306 ; You can use create server definition by writing “server” parameter into table/partition comment. CREATE TABLE t1( c1 int, c2 varchar(100), PRIMARY KEY(c1) )ENGINE=spider DEFAULT CHARSET=utf8 COMMENT 'table "rt1", server "srv1"';
  • 24. Multi dimensional sharding technique with VP storage engine
  • 25. Multi dimenstional sharding technique with VP storage engine VP means Vertical Partitioning. VP merges multiple child tables into a single View. VP chooses efficiently child tables for Each query. So, when you use sharded Spider tables which have different partitioning rules for VP child tables, VP chooses sharded Spider tables efficiently.
  • 26. Structure sample of using different sharding rules (1/2) DB2 tbl_a 1.Request 3.Response DB3 tbl_a DB4 tbl_a APAPAP AP AP DB1 tbl_a (vp) DB5 tbl_a Partition by col_a tbl_a1(spider) Partition by col_b tbl_a2(spider) select … from tbl_a where col_a = 1
  • 27. Structure sample of using different sharding rules (2/2) DB2 tbl_a 1.Request 3.Response DB3 tbl_a DB4 tbl_a APAPAP AP AP DB1 tbl_a (vp) DB5 tbl_a Partition by col_a tbl_a1(spider) Partition by col_b tbl_a2(spider) select … from tbl_a where col_b = 1
  • 29. Roadmap of SPIDER Developed new fearures of Spider this year - Recovering statistics information at startup tables. (Improving startup phase of MariaDB) - Direct join on data node part1. (not include partitioned table and redundant table) - Direct join on data node part2. (for redundant table) binary http://spiderformysql.com/downloads/spider-3.3/mariadb-10.2.0-spider- 3.3-vp-1.1-linux-x86_64-glibc25.tgz source code http://spiderformysql.com/downloads/spider-3.3/mariadb-10.2.0-spider- 3.3-vp-1.1.tgz
  • 30. Roadmap of SPIDER Autumn 2016 - Reducing number of threads for collecting statistics information from data nodes. (Reduce memory usage) Winter 2016 - Auto repair broken spider system table. - Direct join on data node part3. (for partitioned table)
  • 31. Roadmap of SPIDER Spring 2017 - Auto XA recovery and commit/rollback on startup phase of MariaDB. - Direct join on data node part4. (for VP table)
  • 32. Where to get SPIDER (with VP)
  • 33. Where to get SPIDER (with VP) Spider is merge with MariaDB on regular basis since 10.0. But some changes needs modification in the sql layer and partitioning layer and VP is not bundled yet for that reason we advice to use our MariaDB branch on github. You get more features like parallel query, BKA join on partition table, many push down like aggregate query and joins. https://github.com/Kentoku/Spider We do provide binaries and packages on request. support@spiderformysql.com
  • 34. One more thing to Announce!!
  • 35. Just want to say…..
  • 36. WE HAVE NEW BRANCH OFFICE in AMSTERDAM!!