The OpenStack Object Storage system, aka "Swift", was built to power Rackspace's public cloud storage service, Cloud Files. Subsequently, Swift was released into open source as part of the OpenStack project. While Swift is increasingly being utilized to run many large scale public storage clouds outside of Rackspace, Swift is also showing great promise for use in private storage clouds - which look and feel like the public object storage systems from Amazon/Rackspace, yet meeting the needs of the individual company. In this talk, I will present a step-by-step plan on getting OpenStack Swift up and running in your organization. I will cover: - when to use Swift (and when not to) - convincing your boss (the economics) - selecting appropriate hardware (benchmarking) - achieving high durability and availability (cluster design) - knowing how it's working (management & monitoring) - what interfaces exist (compatible clients) - how to build killer apps (features for developers) About Joe Arnold: Joe Arnold is the CEO of SwiftStack. Joe managed the first public OpenStack launch of Swift independent of Rackspace, and has subsequently deployed multiple large-scale cloud storage systems. He is currently building Swift deployment and management tools with his firm, SwiftStack. Prior to SwiftStack, Joe held leadership positions at Cloudscaling, Engine Yard and Yahoo!.

1. 7 STEPS
to Roll-out a Private
OpenStack Swift Cluster
Joe Arnold
OpenStack Conference
April 19, 2012

3. OpenStack Swift
Powers Rackspace Cloud Files

4. Data is
Sticky
http://www.flickr.com/photos/josefgrunig/

5. App App App
Storage Storage Storage
Compute Compute Compute

6. Rackspace Openstack
Cloud Files Launch
59 developers
contributing
to Swift
’09 ’10

7. When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)

8. When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)

9. Swi ft is good for
- Application data
- Filesharing
- Backups
- Unstructured data

10. Swift is laser focused
Swift Attributes
Scalable

11. Swift Attributes
Durable

12. Swift Attributes
Concurrency

13. Not good at . . .
Databases - transactional workloads
No File Locking

14. Not good at . . .
Single
Stream
I/O

15. When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)

16. Data Durability - Zones

17. Data Durability - Replication & Consistency

18. High Availability
Proxy
+
Zones

19. Example: 2 Nodes
- Whole Files
- 3 Replicas
- Withstand multiple
disk failures

20. Example: 5 Nodes
- Withstand multiple
Node Failures
- Consider Separate
Proxy Tier

21. Example: 50 Nodes
- Withstand Rack Failures
- Requires Separate
Proxy Tier

22. When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)

23. Swi ft is economical
$0.40/GB Capex
Example: $42,520 for 105TB usable
1 x Dell r815 ($4,500)
2x Supermicro JBODs ($5,400)
144 x 3TB hard drives ($33,120)
Thailand flooding drive prices

24. Swi ft is economical
Size Configuration Cost $ / GB
2 ToR Switch
2 Proxy/Auth Servers
145 TB 5 Object Stores $95k $0.65
...and cables, rack, etc
2 Agg Switches
6 Proxy/Auth Servers
1.3 PB 5 ToR Switches $750k $0.52
50 Object Stores
...and cables, racks, etc

25. When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)

26. High volume web /
mobile application
Various Enterprise Filesharing
Work loads
Backup /
Active Archive

27. Measure and Test
Frequency
File Sizes Concurrency of upload vs
download

28. RAM - 12GB
PROXY NODES Networking - 2 x 10GbE
CPU - quad-core
RAM - 1GB of RAM / 2TB
Drives - 2-3 TB Drives
Controller Cards - No RAID
Using SSDs - User Accounts &
Container Listings
STORAGE NODES

29. When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)

30. People are
Expensive

31. Added StatsD to Watch EVERYTHING
Swift Process
Swift Process
Swift Process
Swift Process Lightweight
Swift Process
Swift Process UDP StatsD
StatsD
Time-Series Backend
Ganglia / Graphite

32. Integrating Components around Swift
Load Balancing
SSL Termination Authentication
OpenStack Object Storage
OS, Node & Drive Swift Runtime
Monitoring Monitoring
Operating System + Kernel

33. Managing a distributed system
Controller

34. Capacity adjustments
Capacity
time

35. When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)

36. S3
Cloud Files
’10
’06 ’08

38. When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)

39. HTTP API
Data Storage

40. Media Delivery
Direct to Device
Integrates to CDN
Streaming
Static Documents
Direct from Device

42. TRY http://swiftstack.com
joe@swiftstack.com
@joearnold - @swiftstack
rkshop ๏Install
Swi ft wo ๏Build an Application
tom orrow at
1:3 0 - 4 : 00