SlideShare a Scribd company logo
1 of 33
Download to read offline
S3 Compatible Storage
      “LeoFS”


    Rakuten. Inc, RIT Yosuke Hara 25/05/2012   1
Table of Contents
  1. Motivation

  2. Overview

  3. Inside of LeoFS

  4. WEB Console



                       2
1. Motivation


                3
Why NFS?

 Which is suitable storage 
    for storing the media files?




                                         ?
Low ROI
Possibility of SPOF
Storage Expansion is difficult during increasing data
                                                        4
5
Object Storage Farm for IaaS




                               6
2. Overview


              7
Overview




           Storage

           Gateway

           Manager




                     8
System Layout

         Request from Web Application(s)
                            Load Balancer
                                                                                 LeoFS-Manager

                                                                 S3-API
                                  REST over HTTP
LeoFS-Gateway
                                                              RPC
w/Cache Server
                                                                                                SNMP
                                       RPC
LeoFS-Storage



       Storage Engine/Router
         Storage Engine/Router
      Storage Engine/Router
     WEB Console



        META
     Object Store
        META
     Object Store
    META
      Object Store


                                                                                                       9
System Layout

         Request from Web Application(s)
                            Load Balancer
                   Gateway
                                               Manager
                                                                          LeoFS-Manager
                HTTP
                       Cluster
   Request/Response Handling
   S3-API
                                       Management
                 REST over HTTP
LeoFS-Gateway
                 +
                    RPC
                                                
           w/Object Cache
w/Cache Server
                                           Ring Watcher
                       (AWS S3-API)
                                         Node Watcher
                                                                                         SNMP
                                   RPC
LeoFS-Storage

                                       Storage
                                                                                        GUI Console
           Object Storage, Meta data Storage
       Storage Engine/Router
     Storage Engine/Router
     Storage Engine/Router


                           +
        META
    Replicator/Recoverer
                  Object Store
                     META
       META
       Object Store
              Object Store


                                                                                                 10
3. Inside of LeoFS


                     11
Architecture

                                   HTTP
  Gateway (stateless)


                                   Erlang RPC

               Erlang RPC



                        Storage Cluster
                        (multi master)
                                                  Erlang RPC
                                                Process Monitor



                                           Manager Cluster
                                                                  12
Architecture

Architecture - Gateway/Storage


       LeoFS Gateway                                                       Cacher
                                   REST over HTTP (S3-API)

                             get        put          delete     head
                   redundant-manager        membership (fault-detection)

                                              RPC
       LeoFS-Storage

               redundant-manager        replicator     read-repairer
RPC                                                                          RPC
                   membership (fault-detection)         queue

                                 Storage Engine

                       Object Storage      Metadata Storage

                                                                                13
Architecture

Architecture - Manager Cluster

                                   Erlang Mnesia




                                       RING
                                Member / Cluster State
                                    Auth / ACL
                                 Process Monitor
    Gateway / Storage Cluster



                                                         14
Architecture




               15
Three “HIGH”


       High Cost Performance
          Monolithic Storage System

          Storage Engine For Unstructured Files

          Traffic Restrain Mechanism

                > File Cache System (Gateway Plugin)




                                                       16
Three “HIGH”


       High Reliability
          NO SPOF

          Split Brain Measure

          “Erlang OTP” > Nine Nines (99.9999999%)




                                                    17
Three “HIGH”


       High Scalability
          Elastic Storage System

                > Able to dynamic attach/detach nodes

                > Able to over 100-nodes

                          > NOT Mesh-connected Mutual Servers




                                                                18
Cache Mechanism


                  19
Cache Mechanism

Gateway Buffer Pool

                   Slab Alloc
                    Skiplist


                                 {$filename, $etag}
                                       request

   from Client


                                       response


                 Gateway            match: {ok, match}
                                NOT match: {ok, $metadata, $body,}


                   High I/O efficiency
                   Low Latency
                                                                     20
Storage Engine


                 21
Storage Engine

Metadata + Object Storage

            LeoFS-Storage



                  Storage Engine / Replicator / Recoverer




           Object’s Attribute Storage
                  (metadata)
                                         Object Storage




    Metadata : Keeps an in-memory index of all data.
    Object Storage : Log structured (append-only) object store.
                                                                   22
Storage Engine

Retrieve an object from Storage

                                        Storage Engine



                          < META DATA >	
                          ID	
                          Filename	
                          Offset	
                          Size	
                          Checksum	
          Data



                              Header
                Metadata

                               File

                              Footer


                    Object Container
                                                                23
Storage Engine

Insert an object into Storage
                                 Storage Engine

                                       Add a Metadata




                                         Meta Data Server

           Data




       Append a File                                        24
Storage Engine

Reduce unnecessary files




                  Compaction




                                25
4. WEB Console


                 26
Web Console


               File Manager

 WEB Console

               Node Stats




               Log Search


                              27
Web Console



                          Node State Monitor




Log Analyzer / Searcher


                                          28
Web Console
Web Console System Layout



                                    Manager   Storage        Gateway



                                                        or

                   GUI Console	

   Producers and
      Admins	

                                                               Logstatsh




                                                                           29
Wrap Up


          30
Wrap Up




          High Cost Performance
          High Reliability
          High Scalability
                                  31
32
Thank you for your time



                          33

More Related Content

What's hot

All Oracle DBAs have to know about Unix Memory Monitoring
All Oracle DBAs have to know about Unix Memory MonitoringAll Oracle DBAs have to know about Unix Memory Monitoring
All Oracle DBAs have to know about Unix Memory MonitoringYury Velikanov
 
Keep your hadoop cluster at its best! v4
Keep your hadoop cluster at its best! v4Keep your hadoop cluster at its best! v4
Keep your hadoop cluster at its best! v4Chris Nauroth
 
Hadoop 3 @ Hadoop Summit San Jose 2017
Hadoop 3 @ Hadoop Summit San Jose 2017Hadoop 3 @ Hadoop Summit San Jose 2017
Hadoop 3 @ Hadoop Summit San Jose 2017Junping Du
 
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsCeph Community
 
HBase Coprocessor Introduction
HBase Coprocessor IntroductionHBase Coprocessor Introduction
HBase Coprocessor IntroductionSchubert Zhang
 
LLAP: Locality is dead (in the cloud)
LLAP: Locality is dead (in the cloud)  LLAP: Locality is dead (in the cloud)
LLAP: Locality is dead (in the cloud) Future of Data Meetup
 
Apache Big Data EU 2015 - HBase
Apache Big Data EU 2015 - HBaseApache Big Data EU 2015 - HBase
Apache Big Data EU 2015 - HBaseNick Dimiduk
 
Unlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu Yong
Unlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu YongUnlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu Yong
Unlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu YongCeph Community
 
HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014Nick Dimiduk
 
Hadoop operations-2015-hadoop-summit-san-jose-v5
Hadoop operations-2015-hadoop-summit-san-jose-v5Hadoop operations-2015-hadoop-summit-san-jose-v5
Hadoop operations-2015-hadoop-summit-san-jose-v5Chris Nauroth
 
Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of OzoneHadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of OzoneErik Krogen
 
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache RatisFloating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache RatisDataWorks Summit
 
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...Odinot Stanislas
 
Difference between hadoop 2 vs hadoop 3
Difference between hadoop 2 vs hadoop 3Difference between hadoop 2 vs hadoop 3
Difference between hadoop 2 vs hadoop 3Manish Chopra
 

What's hot (20)

All Oracle DBAs have to know about Unix Memory Monitoring
All Oracle DBAs have to know about Unix Memory MonitoringAll Oracle DBAs have to know about Unix Memory Monitoring
All Oracle DBAs have to know about Unix Memory Monitoring
 
HBase Accelerated: In-Memory Flush and Compaction
HBase Accelerated: In-Memory Flush and CompactionHBase Accelerated: In-Memory Flush and Compaction
HBase Accelerated: In-Memory Flush and Compaction
 
Keep your hadoop cluster at its best! v4
Keep your hadoop cluster at its best! v4Keep your hadoop cluster at its best! v4
Keep your hadoop cluster at its best! v4
 
Achieving 100k Queries per Hour on Hive on Tez
Achieving 100k Queries per Hour on Hive on TezAchieving 100k Queries per Hour on Hive on Tez
Achieving 100k Queries per Hour on Hive on Tez
 
Hadoop 3 @ Hadoop Summit San Jose 2017
Hadoop 3 @ Hadoop Summit San Jose 2017Hadoop 3 @ Hadoop Summit San Jose 2017
Hadoop 3 @ Hadoop Summit San Jose 2017
 
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
 
ORC 2015: Faster, Better, Smaller
ORC 2015: Faster, Better, SmallerORC 2015: Faster, Better, Smaller
ORC 2015: Faster, Better, Smaller
 
HBase Coprocessor Introduction
HBase Coprocessor IntroductionHBase Coprocessor Introduction
HBase Coprocessor Introduction
 
Fastest Servlets in the West
Fastest Servlets in the WestFastest Servlets in the West
Fastest Servlets in the West
 
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in HiveLLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
 
LLAP: Locality is dead (in the cloud)
LLAP: Locality is dead (in the cloud)  LLAP: Locality is dead (in the cloud)
LLAP: Locality is dead (in the cloud)
 
Stress your DUT
Stress your DUTStress your DUT
Stress your DUT
 
Apache Big Data EU 2015 - HBase
Apache Big Data EU 2015 - HBaseApache Big Data EU 2015 - HBase
Apache Big Data EU 2015 - HBase
 
Unlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu Yong
Unlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu YongUnlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu Yong
Unlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu Yong
 
HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014
 
Hadoop operations-2015-hadoop-summit-san-jose-v5
Hadoop operations-2015-hadoop-summit-san-jose-v5Hadoop operations-2015-hadoop-summit-san-jose-v5
Hadoop operations-2015-hadoop-summit-san-jose-v5
 
Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of OzoneHadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of Ozone
 
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache RatisFloating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
 
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
 
Difference between hadoop 2 vs hadoop 3
Difference between hadoop 2 vs hadoop 3Difference between hadoop 2 vs hadoop 3
Difference between hadoop 2 vs hadoop 3
 

Similar to Rakuten LeoFs - distributed file system

LAPP/SELinux - A secure web application stack powered by SELinux
LAPP/SELinux - A secure web application stack powered by SELinuxLAPP/SELinux - A secure web application stack powered by SELinux
LAPP/SELinux - A secure web application stack powered by SELinuxKohei KaiGai
 
Gemfire Sqlfire - La Marmite NoSql
Gemfire Sqlfire - La Marmite NoSqlGemfire Sqlfire - La Marmite NoSql
Gemfire Sqlfire - La Marmite NoSqlDuchess France
 
Containerized Storage for Containers Meetup #3
Containerized Storage for Containers Meetup #3Containerized Storage for Containers Meetup #3
Containerized Storage for Containers Meetup #3OpenEBS
 
Containerized Storage for Containers - Meetup #3
Containerized Storage for Containers - Meetup #3Containerized Storage for Containers - Meetup #3
Containerized Storage for Containers - Meetup #3kiran mova
 
Make easier Integration of your services with Fuse Solutions - RedHat 2013
Make easier Integration of your services with Fuse Solutions - RedHat 2013Make easier Integration of your services with Fuse Solutions - RedHat 2013
Make easier Integration of your services with Fuse Solutions - RedHat 2013Charles Moulliard
 
openATTIC Technology Overview - Ceph Management
openATTIC Technology Overview - Ceph ManagementopenATTIC Technology Overview - Ceph Management
openATTIC Technology Overview - Ceph Managementit-novum
 
Confoo2013 make your java-app rest enabled
Confoo2013 make your java-app rest enabledConfoo2013 make your java-app rest enabled
Confoo2013 make your java-app rest enabledAnthony Dahanne
 
OSGi Service Platform 4.2
OSGi Service Platform 4.2OSGi Service Platform 4.2
OSGi Service Platform 4.2Ilya Katsov
 
LAPP/SELinux - A secure web application stack using SE-PostgreSQL
LAPP/SELinux - A secure web application stack using SE-PostgreSQLLAPP/SELinux - A secure web application stack using SE-PostgreSQL
LAPP/SELinux - A secure web application stack using SE-PostgreSQLKohei KaiGai
 
Java Web Programming [1/9] : Introduction to Web Application
Java Web Programming [1/9] : Introduction to Web ApplicationJava Web Programming [1/9] : Introduction to Web Application
Java Web Programming [1/9] : Introduction to Web ApplicationIMC Institute
 
Learning from ZFS to Scale Storage on and under Containers
Learning from ZFS to Scale Storage on and under ContainersLearning from ZFS to Scale Storage on and under Containers
Learning from ZFS to Scale Storage on and under Containersinside-BigData.com
 
Open stack in sina
Open stack in sinaOpen stack in sina
Open stack in sinaHui Cheng
 
Common Sense Performance Indicators in the Cloud
Common Sense Performance Indicators in the CloudCommon Sense Performance Indicators in the Cloud
Common Sense Performance Indicators in the CloudNick Gerner
 
Triangle Node.js DevOps
Triangle Node.js DevOpsTriangle Node.js DevOps
Triangle Node.js DevOpsShubhra Kar
 
03.eGovFrame Runtime Environment Training Book Supplement
03.eGovFrame Runtime Environment Training Book Supplement03.eGovFrame Runtime Environment Training Book Supplement
03.eGovFrame Runtime Environment Training Book SupplementChuong Nguyen
 
Introduction to Apache Camel
Introduction to Apache CamelIntroduction to Apache Camel
Introduction to Apache CamelFuseSource.com
 
LAPP/SELinux - A secure web application platform powered by SELinux
LAPP/SELinux - A secure web application platform powered by SELinuxLAPP/SELinux - A secure web application platform powered by SELinux
LAPP/SELinux - A secure web application platform powered by SELinuxKohei KaiGai
 

Similar to Rakuten LeoFs - distributed file system (20)

LAPP/SELinux - A secure web application stack powered by SELinux
LAPP/SELinux - A secure web application stack powered by SELinuxLAPP/SELinux - A secure web application stack powered by SELinux
LAPP/SELinux - A secure web application stack powered by SELinux
 
Gemfire Sqlfire - La Marmite NoSql
Gemfire Sqlfire - La Marmite NoSqlGemfire Sqlfire - La Marmite NoSql
Gemfire Sqlfire - La Marmite NoSql
 
Containerized Storage for Containers Meetup #3
Containerized Storage for Containers Meetup #3Containerized Storage for Containers Meetup #3
Containerized Storage for Containers Meetup #3
 
Containerized Storage for Containers - Meetup #3
Containerized Storage for Containers - Meetup #3Containerized Storage for Containers - Meetup #3
Containerized Storage for Containers - Meetup #3
 
Make easier Integration of your services with Fuse Solutions - RedHat 2013
Make easier Integration of your services with Fuse Solutions - RedHat 2013Make easier Integration of your services with Fuse Solutions - RedHat 2013
Make easier Integration of your services with Fuse Solutions - RedHat 2013
 
openATTIC Technology Overview - Ceph Management
openATTIC Technology Overview - Ceph ManagementopenATTIC Technology Overview - Ceph Management
openATTIC Technology Overview - Ceph Management
 
Confoo2013 make your java-app rest enabled
Confoo2013 make your java-app rest enabledConfoo2013 make your java-app rest enabled
Confoo2013 make your java-app rest enabled
 
OSGi Service Platform 4.2
OSGi Service Platform 4.2OSGi Service Platform 4.2
OSGi Service Platform 4.2
 
LAPP/SELinux - A secure web application stack using SE-PostgreSQL
LAPP/SELinux - A secure web application stack using SE-PostgreSQLLAPP/SELinux - A secure web application stack using SE-PostgreSQL
LAPP/SELinux - A secure web application stack using SE-PostgreSQL
 
Java Web Programming [1/9] : Introduction to Web Application
Java Web Programming [1/9] : Introduction to Web ApplicationJava Web Programming [1/9] : Introduction to Web Application
Java Web Programming [1/9] : Introduction to Web Application
 
Bca1931 final
Bca1931 finalBca1931 final
Bca1931 final
 
Learning from ZFS to Scale Storage on and under Containers
Learning from ZFS to Scale Storage on and under ContainersLearning from ZFS to Scale Storage on and under Containers
Learning from ZFS to Scale Storage on and under Containers
 
Open stack in sina
Open stack in sinaOpen stack in sina
Open stack in sina
 
Crx 2.2 Deep-Dive
Crx 2.2 Deep-DiveCrx 2.2 Deep-Dive
Crx 2.2 Deep-Dive
 
Common Sense Performance Indicators in the Cloud
Common Sense Performance Indicators in the CloudCommon Sense Performance Indicators in the Cloud
Common Sense Performance Indicators in the Cloud
 
Triangle Node.js DevOps
Triangle Node.js DevOpsTriangle Node.js DevOps
Triangle Node.js DevOps
 
03.eGovFrame Runtime Environment Training Book Supplement
03.eGovFrame Runtime Environment Training Book Supplement03.eGovFrame Runtime Environment Training Book Supplement
03.eGovFrame Runtime Environment Training Book Supplement
 
Introduction to Apache Camel
Introduction to Apache CamelIntroduction to Apache Camel
Introduction to Apache Camel
 
LAPP/SELinux - A secure web application platform powered by SELinux
LAPP/SELinux - A secure web application platform powered by SELinuxLAPP/SELinux - A secure web application platform powered by SELinux
LAPP/SELinux - A secure web application platform powered by SELinux
 
Caching On Zos
Caching On ZosCaching On Zos
Caching On Zos
 

More from Rakuten Group, Inc.

コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話Rakuten Group, Inc.
 
楽天における安全な秘匿情報管理への道のり
楽天における安全な秘匿情報管理への道のり楽天における安全な秘匿情報管理への道のり
楽天における安全な秘匿情報管理への道のりRakuten Group, Inc.
 
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...Rakuten Group, Inc.
 
大規模なリアルタイム監視の導入と展開
大規模なリアルタイム監視の導入と展開大規模なリアルタイム監視の導入と展開
大規模なリアルタイム監視の導入と展開Rakuten Group, Inc.
 
楽天における大規模データベースの運用
楽天における大規模データベースの運用楽天における大規模データベースの運用
楽天における大規模データベースの運用Rakuten Group, Inc.
 
楽天サービスを支えるネットワークインフラストラクチャー
楽天サービスを支えるネットワークインフラストラクチャー楽天サービスを支えるネットワークインフラストラクチャー
楽天サービスを支えるネットワークインフラストラクチャーRakuten Group, Inc.
 
楽天の規模とクラウドプラットフォーム統括部の役割
楽天の規模とクラウドプラットフォーム統括部の役割楽天の規模とクラウドプラットフォーム統括部の役割
楽天の規模とクラウドプラットフォーム統括部の役割Rakuten Group, Inc.
 
Rakuten Services and Infrastructure Team.pdf
Rakuten Services and Infrastructure Team.pdfRakuten Services and Infrastructure Team.pdf
Rakuten Services and Infrastructure Team.pdfRakuten Group, Inc.
 
The Data Platform Administration Handling the 100 PB.pdf
The Data Platform Administration Handling the 100 PB.pdfThe Data Platform Administration Handling the 100 PB.pdf
The Data Platform Administration Handling the 100 PB.pdfRakuten Group, Inc.
 
Supporting Internal Customers as Technical Account Managers.pdf
Supporting Internal Customers as Technical Account Managers.pdfSupporting Internal Customers as Technical Account Managers.pdf
Supporting Internal Customers as Technical Account Managers.pdfRakuten Group, Inc.
 
Making Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfMaking Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfRakuten Group, Inc.
 
How We Defined Our Own Cloud.pdf
How We Defined Our Own Cloud.pdfHow We Defined Our Own Cloud.pdf
How We Defined Our Own Cloud.pdfRakuten Group, Inc.
 
Travel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoTravel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoRakuten Group, Inc.
 
Travel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoTravel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoRakuten Group, Inc.
 
Introduction of GORA API Group technology
Introduction of GORA API Group technologyIntroduction of GORA API Group technology
Introduction of GORA API Group technologyRakuten Group, Inc.
 
100PBを越えるデータプラットフォームの実情
100PBを越えるデータプラットフォームの実情100PBを越えるデータプラットフォームの実情
100PBを越えるデータプラットフォームの実情Rakuten Group, Inc.
 
社内エンジニアを支えるテクニカルアカウントマネージャー
社内エンジニアを支えるテクニカルアカウントマネージャー社内エンジニアを支えるテクニカルアカウントマネージャー
社内エンジニアを支えるテクニカルアカウントマネージャーRakuten Group, Inc.
 
モニタリングプラットフォーム開発の裏側
モニタリングプラットフォーム開発の裏側モニタリングプラットフォーム開発の裏側
モニタリングプラットフォーム開発の裏側Rakuten Group, Inc.
 

More from Rakuten Group, Inc. (20)

コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
 
楽天における安全な秘匿情報管理への道のり
楽天における安全な秘匿情報管理への道のり楽天における安全な秘匿情報管理への道のり
楽天における安全な秘匿情報管理への道のり
 
What Makes Software Green?
What Makes Software Green?What Makes Software Green?
What Makes Software Green?
 
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
 
大規模なリアルタイム監視の導入と展開
大規模なリアルタイム監視の導入と展開大規模なリアルタイム監視の導入と展開
大規模なリアルタイム監視の導入と展開
 
楽天における大規模データベースの運用
楽天における大規模データベースの運用楽天における大規模データベースの運用
楽天における大規模データベースの運用
 
楽天サービスを支えるネットワークインフラストラクチャー
楽天サービスを支えるネットワークインフラストラクチャー楽天サービスを支えるネットワークインフラストラクチャー
楽天サービスを支えるネットワークインフラストラクチャー
 
楽天の規模とクラウドプラットフォーム統括部の役割
楽天の規模とクラウドプラットフォーム統括部の役割楽天の規模とクラウドプラットフォーム統括部の役割
楽天の規模とクラウドプラットフォーム統括部の役割
 
Rakuten Services and Infrastructure Team.pdf
Rakuten Services and Infrastructure Team.pdfRakuten Services and Infrastructure Team.pdf
Rakuten Services and Infrastructure Team.pdf
 
The Data Platform Administration Handling the 100 PB.pdf
The Data Platform Administration Handling the 100 PB.pdfThe Data Platform Administration Handling the 100 PB.pdf
The Data Platform Administration Handling the 100 PB.pdf
 
Supporting Internal Customers as Technical Account Managers.pdf
Supporting Internal Customers as Technical Account Managers.pdfSupporting Internal Customers as Technical Account Managers.pdf
Supporting Internal Customers as Technical Account Managers.pdf
 
Making Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfMaking Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdf
 
How We Defined Our Own Cloud.pdf
How We Defined Our Own Cloud.pdfHow We Defined Our Own Cloud.pdf
How We Defined Our Own Cloud.pdf
 
Travel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoTravel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech info
 
Travel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoTravel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech info
 
OWASPTop10_Introduction
OWASPTop10_IntroductionOWASPTop10_Introduction
OWASPTop10_Introduction
 
Introduction of GORA API Group technology
Introduction of GORA API Group technologyIntroduction of GORA API Group technology
Introduction of GORA API Group technology
 
100PBを越えるデータプラットフォームの実情
100PBを越えるデータプラットフォームの実情100PBを越えるデータプラットフォームの実情
100PBを越えるデータプラットフォームの実情
 
社内エンジニアを支えるテクニカルアカウントマネージャー
社内エンジニアを支えるテクニカルアカウントマネージャー社内エンジニアを支えるテクニカルアカウントマネージャー
社内エンジニアを支えるテクニカルアカウントマネージャー
 
モニタリングプラットフォーム開発の裏側
モニタリングプラットフォーム開発の裏側モニタリングプラットフォーム開発の裏側
モニタリングプラットフォーム開発の裏側
 

Recently uploaded

KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 

Recently uploaded (20)

KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 

Rakuten LeoFs - distributed file system

  • 1. S3 Compatible Storage “LeoFS” Rakuten. Inc, RIT Yosuke Hara 25/05/2012 1
  • 2. Table of Contents 1. Motivation 2. Overview 3. Inside of LeoFS 4. WEB Console 2
  • 4. Why NFS? Which is suitable storage for storing the media files? ? Low ROI Possibility of SPOF Storage Expansion is difficult during increasing data 4
  • 5. 5
  • 6. Object Storage Farm for IaaS 6
  • 8. Overview Storage Gateway Manager 8
  • 9. System Layout Request from Web Application(s) Load Balancer LeoFS-Manager S3-API REST over HTTP LeoFS-Gateway RPC w/Cache Server SNMP RPC LeoFS-Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router WEB Console META Object Store META Object Store META Object Store 9
  • 10. System Layout Request from Web Application(s) Load Balancer Gateway Manager LeoFS-Manager HTTP Cluster Request/Response Handling S3-API Management REST over HTTP LeoFS-Gateway + RPC w/Object Cache w/Cache Server Ring Watcher (AWS S3-API) Node Watcher SNMP RPC LeoFS-Storage Storage GUI Console Object Storage, Meta data Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router + META Replicator/Recoverer Object Store META META Object Store Object Store 10
  • 11. 3. Inside of LeoFS 11
  • 12. Architecture HTTP Gateway (stateless) Erlang RPC Erlang RPC Storage Cluster (multi master) Erlang RPC Process Monitor Manager Cluster 12
  • 13. Architecture Architecture - Gateway/Storage LeoFS Gateway Cacher REST over HTTP (S3-API) get put delete head redundant-manager membership (fault-detection) RPC LeoFS-Storage redundant-manager replicator read-repairer RPC RPC membership (fault-detection) queue Storage Engine Object Storage Metadata Storage 13
  • 14. Architecture Architecture - Manager Cluster Erlang Mnesia RING Member / Cluster State Auth / ACL Process Monitor Gateway / Storage Cluster 14
  • 16. Three “HIGH” High Cost Performance Monolithic Storage System Storage Engine For Unstructured Files Traffic Restrain Mechanism > File Cache System (Gateway Plugin) 16
  • 17. Three “HIGH” High Reliability NO SPOF Split Brain Measure “Erlang OTP” > Nine Nines (99.9999999%) 17
  • 18. Three “HIGH” High Scalability Elastic Storage System > Able to dynamic attach/detach nodes > Able to over 100-nodes > NOT Mesh-connected Mutual Servers 18
  • 20. Cache Mechanism Gateway Buffer Pool Slab Alloc Skiplist {$filename, $etag} request from Client response Gateway match: {ok, match} NOT match: {ok, $metadata, $body,} High I/O efficiency Low Latency 20
  • 22. Storage Engine Metadata + Object Storage LeoFS-Storage Storage Engine / Replicator / Recoverer Object’s Attribute Storage (metadata) Object Storage Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store. 22
  • 23. Storage Engine Retrieve an object from Storage Storage Engine < META DATA > ID Filename Offset Size Checksum Data Header Metadata File Footer Object Container 23
  • 24. Storage Engine Insert an object into Storage Storage Engine Add a Metadata Meta Data Server Data Append a File 24
  • 25. Storage Engine Reduce unnecessary files Compaction 25
  • 27. Web Console File Manager WEB Console Node Stats Log Search 27
  • 28. Web Console Node State Monitor Log Analyzer / Searcher 28
  • 29. Web Console Web Console System Layout Manager Storage Gateway or GUI Console Producers and Admins Logstatsh 29
  • 30. Wrap Up 30
  • 31. Wrap Up High Cost Performance High Reliability High Scalability 31
  • 32. 32
  • 33. Thank you for your time 33