Inkscope is a GUI for the great software defined storage system CEPH.
It is an open source initiave that can help to operate a ceph cluster.
It gives at a glance a comprehensive status of the system and helps to manage osds, pools, rados gateway...
Those slides were presented at Ceph Day Paris in September 2014.
2. 2
Agenda
Ceph at Orange Labs
Inkscope project
Demo
Parallel with Calamari
Next steps
Q&A
CephDay Paris – September18,2014
3. 3
Since 6 years, research on innovative concepts to enrich the cloud storage
functionalities
– Data growth control
– Storage costs reduction
– Bottleneck avoidance
– Data durability insurance
– Ease of data accessibility (ATAWADAC)
– Development of an internal cloud storage solution
Since June 2013
– CEPH is identified as a real challenger
– Acquire expertise on Ceph
Study how the innovative concepts can be
implemented into Ceph (erasure codes,
multi-tiering, multi-siting, store-edge, indexation)
Be an actor in CEPH community
Be a CEPH tamer
CEPH at Orange Labs
source : Crypto – Kraken - Menvielle/Martin
CephDay Paris – September18,2014
4. 4
Federation
– Management of heterogeneous storage services
– Storage location regarding costs and performance
Storage contracts
– content = f(data, contract, metadata)
Smart data placement
– Storage orchestration (best storage infrastructure)
– Data robustness (replication/erasure code, sharding)
Multi-tiering
– Right Data, Right Tier
– Implementation in Ceph
– Crushmap management
– Dynamic tiering
Multi-siting
– Right Data, Right Site
– Implementation in Ceph
– Rados Gateway Federation
Ceph Monitoring and Administration
– Inkscope initiative
R&D Works
CephDay Paris – September18,2014
5. 5
Ceph visualization and operation through command lines
No open-source Ceph GUI available in late 2013
Breizh Meetup helps us to launch the project
Our choices:
– Open source
– Web app
– Use of Ceph rest API
– Modularity, simplicity
Contributors:
– A.Dechorgnat, Ph.Raipin (Orange labs), A.Sall (trainee)
– E. Mourgaya (Arkea)
Why Inkscope ?
CephDay Paris – September18,2014
6. 6
cluster
inkscopeViz
– Web client
inkscopeCtrl
– Server part
– Provides an advanced REST API
inkscopeProbe
– Collects system and ceph infos
– Feeds a mongoDB database
inkscopeMonitor (not developed)
– Monitoring of Ceph metrics stored in db
– Feeds monitoring tools like Nagios
Inkscope modules
Supervision
framework
(Nagios…)inkscopeProbe
inkscopeCtrl inkscopeMonitor
inkscopeViz
inkscope
server
ceph APIs
inkscope
client
CephDay Paris – September18,2014
8. 8
Dashboard
Ceph cluster status at a glance
OSD visualization and details
– Free and used capacity
– Functional or not
Pools details
– Used capacity
– Parameters
– PG states
Object storage (S3/swift)
– Users management
– Buckets management
Crushmap visualization
Inkscope functionalities summary
CephDay Paris – September18,2014
9. 9
Tiering
– Wring the neck of the crushmap to fulfill tiering requirements
Work in progress on Ceph
From a basic topology…through a hw-based
topology…
to an application-based topology…
CephDay Paris – September18,2014
let’s see it in
inkscope…
10. 10
Inktank dev
open source since May 2014
mailing-list :
ceph-calamari
presented as diagnostic tool
4 modules
– calamari-web
– cthulhu
– salt-master
– rest-api
Calamari is “in the wild”
CephDay Paris – September18,2014
11. 11
Parallel Calamari/Inkscope : functionalities
Calamari Inkscope
Visualization Dashboard Yes Yes
OSD Yes Yes
Host Yes Yes
PG Yes Yes
Crushmap No Yes
Object No Yes
Graphs Cluster Yes No
Pools IOPS Yes No
Hosts Yes No
Management Console No Yes
OSD Yes (operation only) Partial (stop, in/out)
Pools Yes (limited parameters) Yes
Cluster settings (flags) Yes No
S3/swift users No Yes
S3/swift buckets No Yes
S3/swift zones/regions No No
Monitoring Alerts No No
Logs Yes No
CephDay Paris – September18,2014
12. 12
Parallel Calamari/Inkscope : technologies
CephDay Paris – September18,2014
Client HTML5 HTML5/AngularJs
Javascript libraries:
•AngularJS
•IndexedDb
•BackboneJS
Javascript libraries:
•AngularJs addons (ngTable, dialog, ui Bootstrap)
•Visualization with D3.js, NVD3, Rickshaw
Compass, Sass CSS (needs gem and Ruby),
Bootstrap 3 CSS
Bootstrap CSS
and also Grunt, NodeJs, make to build the
client
Server - Apache with Django (python scripts) Apache with Flask (python scripts)
- packages: django, flask, python-dev, libcairo,
liqpq5, psutil, gevent, sqlalchemy, pytz,
alembic, zwig, zeromq, m2crypto …
Python packages: flask, pymongo, simples3, python-
dev, psutil
PosgreSql MongoDB
Carbon, Graphite, Diamond Shellinabox (optional)
Saltstack
- Supervisor
- Vagrant
and all their dependancies…
Calamari Inkscope
13. 13
Planned in Inkscope
– Extend or replace Ceph Rest API request by command lines
powered by Salt
– Improve S3/swift features : zones and regions management
– Improve probes operations
– Implement objects visualization
– Plug the monitoring module to feed Nagios/Shinken
Other ideas
– Simulation : impact calculation in case of crushmap update
(storage capacity, bandwidth,..)
“Fusion” with Calamari?
– Contact taken with Calamari team
Next steps for inkscope
CephDay Paris – September18,2014