We present several use-cases for Swift Storage Policies, and how they are implemented in the Scality RING Swift back-end.
#vBrownBag session at the Vancouver OpenStack Summit, May 2015
4. Copyright Scality 2015
Scality & OpenStack
●
Dedicated engineering team
●
Integrating with OpenStack Storage projects
– Cinder (data volumes), in-tree
– Glance (VM images), currently out-of-tree
– Swift (object storage), out-of-tree
– 2015 H2: Manila (shared filesystem)
●
Single system to manage, unify OpenStack storage requirements
– Eliminates multiple storage silos
– Consolidates 80% of OpenStack data storage
– Scalable to petabytes and beyond
●
Team encouraged to work on any OpenStack project(s) as part of job
6. Copyright Scality 2015
Scality Ring Scality Sproxyd
Swift Accounts
Service + Storage
Swift Containers
Service + Storage
Customer
Application
Scality Sproxyd
Swift Object Service
swift-scality-
backend
Swift Object Service
swift-scality-
backend
Swift Object Service
swift-scality-
backend
Swift Proxy ServiceSwift Proxy ServiceSwift Proxy Service
scality-sproxyd-client
7. Copyright Scality 2015
Swift Storage Policies
●
Allow admin to define policies/classes of storage
– With different performance, pricing models, SLA's,...
●
Allow tenant to assign a policy to a container (only at creation time)
●
Objects in containers stored according to container policy
– Swift upstream: replication count, SSD/spindle, EC (beta in Kilo)
– Scality: map to sets of Sproxyd connectors in different
configurations
8. Copyright Scality 2015
Use Case #1: Performance vs. Cost
●
Replication vs. Erasure Coding
– Storage consumption: *N vs *((n + k)/ n)
– Low time-to-first-byte vs. higher latency
– Efficient random access vs. mostly-streaming
●
Spindles vs. SSD
●
Depending on application, transfer data between containers/policies
over time
– Keep costs under control
9. Copyright Scality 2015
Use Case #2: Geo-Distributed Storage
●
Geo-distributed datacenters add network latency impact
●
Keep operations as local as possible
– Configuration of SPs takes location of endpoints into account
– Sort on distance
●
Regulations impose locality constraints
– E.g. privacy laws
– Storage Policies used to ensure data placement complies
10. Copyright Scality 2015
Use Case #2A: Multi-datacenter, Single Cluster
●
Active/Active, managed as a single Ring
●
Location-aware allocation: both Replication and ARC/EC can spread
data across disparate domains for site failure tolerance
●
Let Swift object servers talk to 'closeby' connectors
– Reduce number of WAN latency hits
11. Copyright Scality 2015
Use Case #2B: Multi-datacenter, Multiple Clusters
●
Active/Passive, source Ring asynchronously synced
to target
– Site-level Disaster Recovery (within time window)
without latency hit
– Ensure data is replicated to 'hot' access points
– Works really well for immutable objects
– E.g. movie production teams around the world, working
locally, data synced to LA HQ continuously (or at night,
or ...)
●
Ensure Swift talks to 'source' Rings for CUD
operations, local Rings for R operations (taking stale
reads into account, fallback to source Ring on HTTP
404)
