These slides supported a presentation at the 2013 Red Hat Summit. It covers: ✦ Introduction to OpenStack ✦ OpenStack Architecture ✦ Understanding the Elastic Cloud ✦ OpenStack in the Real World

1. 1

2. Introduction & Overview of
OpenStack for IaaS Clouds
Keith Basil
Principal Product Manager, OpenStack
Red Hat
June 12, 2013
2

3. Your presenter..
personal
Virginia hare scrambler, plays chess..
professional
Red Hat
Cloudscaling, Time Warner Cable,
FederalCloud.com, Cisco and
a couple of startups
blended
skype/twitter/github/irc, life: noslzzp
3

4. Agenda
✦ Introduction to OpenStack
✦ OpenStack Architecture
✦ Understanding the Elastic Cloud
✦ OpenStack in the Real World
4

5. What Is OpenStack?
OpenStack is both Software and a Community
5

6. What Problem Does OpenStack Solve?
OpenStack provides a framework for building
elastic cloud infrastructure at massive scale.
✦Facilitates management of
✦COMPUTE,
✦NETWORK, and
✦STORAGE resources
✦Provides natural infrastructure for elastic applications
We will cover the “elastic cloud” later in more detail..
6

7. OpenStack Components Provide..
✦Compute
✦Virtual machine management
✦Comprehensive hypervisor support
✦Networks
✦IP address management
✦Security services
✦Storage
✦Volumes (block storage)
✦Object storage for VM images and files
7

8. OpenStack as Community
Technical Committee - defines and stewards technical direction
Board of Directors - provides strategic and financial oversight of
Foundation Resources
User Committee - created to represent enterprise, academic and
service provider users
8

9. OpenStack History and Cadence
Havana
(Oct 2013)
Grizzly
(Apr 2013)
Folsom
(Oct 2012)
Essex
(Apr 2012)
Diablo
(Sep 2011)
Cactus
(Apr 2011)
Bexar
(Feb 2011)
Austin
(Oct 2010)
Ceilometer & Heat integrated
(Oct 2013)
Maturation of Quantum and Cinder, focus on upgrade support
(Apr 2013)
Quantum (Networking) full inclusion, Volume Service added
(Oct 2012)
Dashboard and Identity service released, Quantum incubated
(Apr 2012)
First “Production Ready” release
(Sep 2011)
Scaling enhancements, support for many hypervisors
(Apr 2011)
OpenStack Compute ready, initial release of Image Service
(Feb 2011)
Initial release, Object Storage Production Ready, Compute in testing
6-month
cadence
9

10. OpenStack Trends, Growth & Milestones
Havana
(Oct 2013)
Grizzly
(Apr 2013)
Folsom
(Oct 2012)
Essex
(Apr 2012)
Diablo
(Sep 2011)
Cactus
(Apr 2011)
Bexar
(Feb 2011)
Austin
(Oct 2010)
First OpenStack
release with Red Hat code
OpenStack Foundation formed
(Sep 2012)
47 committers across
the top ten companies
Red Hat assigns its first developer to the OpenStack community
(Aug 2011)
71 committers
230
committers
Largest OpenStack Summit to date,
enterprise customer keynotes
Data extracted from Google Trends (keyword: OpenStack) and Bitergia reports.
“Committers” shown above indicate the number of individual committers across the top ten contributing companies.
10

11. #3
OVERALL CODE
CONTRIBUTOR TO
ESSEX
(Apr 2012)
11

12. #2
OVERALL CODE
CONTRIBUTOR TO
FOLSOM
(Oct 2012)
12

13. #1
OVERALL CODE
CONTRIBUTOR TO
GRIZZLY
(Apr 2013)
13

14. Red Hat Contributors
Many contributors. One community.
No coincidence.
14

15. Agenda
✦ Introduction to OpenStack
✦ OpenStack Architecture
✦ Understanding the Elastic Cloud
✦ OpenStack in the Real World
15

16. OpenStack Design and Architecture
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
Modular architecture
Based on a (growing) set of core services
Designed for Scalability and Elasticity
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
16

17. OpenStack Dashboard (Horizon)
Provides simple self service UI for end-users
Basic cloud administrator functions
(No infrastructure management)
Define users, tenants and quotas
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
17

18. OpenStack Compute (Nova)
Supports multiple hypervisors (KVM, Xen, LXC, Hyper-V, ESX)
Native OpenStack API and Amazon EC2 API support
Distributed controller services handle scheduling, API calls, etc.
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
18

19. OpenStack Block Storage (Cinder)
Provides block storage for virtual machines (persistent disks)
Similar to Amazon EBS service
Plugin architecture for vendor extensions
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
19

20. OpenStack Networking (Quantum)
Network Service
Provides framework for Software Defined Networking (SDN)
Allows integration of hardware and software based network
solutions
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
20

21. OpenStack Image Service (Glance)
Stores and retrieves disk images (virtual machine templates)
Supports Raw, QCOW, VMDK, VHD, ISO, OVF & AMI/AKI
Backend storage : Filesystem, Swift, Amazon S3
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
21

22. OpenStack Object Storage (Swift)
Modeled after Amazon's S3 service
Provides simple service for storing and retrieving arbitrary data
Native API and S3 compatible API
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
22

23. OpenStack Identity Service (Keystone)
Common authorization framework
Manages users, tenants and roles
Pluggable backends (SQL, PAM, LDAP, etc)
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
23

24. Connecting the Components
DASHBOARD
(Horizon)
COMPUTE
(Nova)
IDENTITY
SERVICE
(Keystone)
OpenStack uses message queues for communication
between components
Supported queueing backends: RabbitMQ, Qpid and ZeroMQ
BLOCKSTORAGE
(Cinder)
OBJECTSTORE
(Swift)
NETWORKING
(Quantum)
IMAGESERVICE
(Glance)
Message Queue
24

25. Incubation Project: Ceilometer
Metering & Monitoring
(Ceilometer)
- Collects meter data (CPU, network, etc)
- Designed for integration and extensibility
- Data collected is made available via REST API
- Message signing provides non-repudiation
Graduated from Incubation to Integrated
status for the Havana release
25

26. Incubation Project: Heat
Application Orchestration
(Heat)
- Provides template driven cloud application
orchestration
- Modeled after AWS CloudFormation
- Targeted to provide advanced functionality
such as high availability and autoscaling
- Introduced by !
Graduated from Incubation to Integrated
status for the Havana release
26

27. Spin me
up a VM!
And make it LARGE!
Umm, Do I
know you? I need to
see some papers!!
Keystone
Ok, we
need to find a
place to build
this VM.
Nova
Tag -
you’re it!
VM
capacity capacity
capacity Papers are
good. Time to
get to work!Nova
Node
Quantum, I
need a network with all
the trimmings!
Quantum
Here’s your IP,
default route and
FW settings.
Cinder, have
that volume
ready for me?
Node
Indeed I do.
Don’t forget to
mount it!
Swift
Glance
Hey Glance, can I
get the RHEL 6.4
image?
Node
8)
Let’s Follow a Request..
Thank you
OpenStack!!
8)
It’s
rendering
time!
27

28. Agenda
✦ Introduction to OpenStack
✦ OpenStack Architecture
✦ Understanding the Elastic Cloud
✦ OpenStack in the Real World
28

29. Elastic Cloud != Enterprise Virtualization
On-demand self-service
Broad network access
Resource pooling
Rapid elasticity
Measured service
Many applications on each server
Maximum server utilization
Minimum server count
29

30. Workload Evolution
CLOUD WORKLOADS
✦Smaller stateless VMs
✦Lifecycle measured in hours to months
✦Applications scale out horizontally
with new VMs
✦Applications expect failure
TRADITIONAL WORKLOADS
✦Larger stateful VMs
✦Lifecycle measured in years
✦Applications scale up (more vCPU, vRAM)
✦Applications NOT designed to
tolerate failure
30

31. “Pets vs Cattle” (Scale Up vs Scale Out)
The above adapted from Tim Bell, CERN
http://www.slideshare.net/noggin143/20121017-openstack-cern-accelerating-cience
Original “Pets vs. Cattle” is attributed to Bill Baker, Microsoft Distinguished Engineer.
http://mem-pass.org/summits/2011/pdfs/DBA-302-HD.pdf
“Future application architectures should use Cattle but Pets with
strong configuration management are viable and still needed”
- Tim Bell, CERN
“
Scale Up
- Servers are like pets.
Pets are given names, are unique,
lovingly hand raised and cared for.
When they get ill, you nurse them back
to health
Scale Out
- Servers are like cattle.
Cattle are given numbers and are
almost identical to each other.
When they get ill, you get another
one.
31

32. Why the Elastic Cloud Is Needed
Our Data is too large
✦We are past the point of single computers being able to
efficiently handle our data
Service Requests are too large
✦Client devices are more plentiful than ever
Or, BOTH..
32

33. Why the Elastic Cloud Is Needed
Servers fail - Deal with it!1
✦Assume you could start with super reliable servers
(MTBF of 30 years!)
✦If you build a system with 10,000 of those servers
✦You will watch one fail every day
Fault-tolerant software is inevitable!
[1] Adapted from Jeff Dean’s presentation on Designs, Lessons and Advice from Building Large Distributed Systems
http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf
33

34. “Fault Tolerance” (survival) in Nature
elastic, agile, scaled with small failure domains
34

35. Pets in the Water
35

36. but.. Unnatural Acts Can Be Impressive
36

37. Agenda
✦ Introduction to OpenStack
✦ OpenStack Architecture
✦ Understanding the Elastic Cloud
✦ OpenStack in the Real World
37

38. OpenStack’s Increasing Maturity
We are seeing organizations struggle with
elastic cloud adoption
✦Unfortunately, Frankenclouds are
being white boarded daily!
✦“Pet friendly” strategies are underway
✦Green field deployments work well
Knowledge capture is early but evolving
✦OpenStack Operations Guide
✦OpenStack Security Guide
38

39. OpenStack Deployment Considerations
OpenStack’s two most important deployment questions:
✦What does the NETWORK look like?
✦“The 90’s called. They want their network architecture back.”
✦ What does the cloud HARDWARE look like?
✦Amazon and Google have figured it out: embrace and extend!
39

40. Question: The Network?
40

41. Network Elasticity is Required..
NODE NODE NODE NODE NODE NODE NODE NODE
NODE NODE
NODE NODE
NODE NODE NODE NODE NODE NODE NODE NODE
NODE NODE
NODE NODE
NODE NODE NODE NODE NODE NODE NODE NODE
NODE NODE
NODE NODE
NODE NODE NODE NODE NODE NODE NODE NODE NODE
BLOCK
STORE
BLOCK
STORE
NODE
NODE NODE NODE NODE NODE NODE NODE
BLOCK
STORE
BLOCK
STORE
NODE
NODE NODE NODE NODE NODE NODE NODE
NODENODE
NODE
BLOCK
STORE
BLOCK
STORE
BLOCK
STORE
BLOCK
STORE
Elastic Cloud Resource Map
NODE
NODE
41

42. Because your cloud will grow..
Each unit here could be a server, or a rack of servers.
42

43. And continue to grow..
1,152 servers shown? Or 1,152 racks with 16 servers each (18,432 servers)?
hint: it shouldn’t matter
43

44. Spine and Leaf Topology
Ask your friendly network vendor for guidance
✦Cisco, ARISTA, Brocade, Juniper, Force10, etc.
http://bradhedlund.com/2012/01/25/construct-a-leaf-spine-design-with-40g-or-10g-an-observation-in-scaling-the-fabric/
44

45. Question: The Hardware?
45

46. 1/1
1/2
1/4
1/8
n1-standard-8-d
n1-standard-4-d
n1-standard-2-d
n1-standard-1-d
m1.xlarge
m1.large
m1.medium
m1.small
m1.class
n1-standard.class
xlarge
large
medium
small
Public Cloud VM Instances Exposed!
46

47. Deployment: Sizing a Compute Node
xlarge
large medium
small
Solve for the
biggest VM in
the class
Smaller VMs are fractional proportions of the largest.
This facilitates efficient hardware use and scheduling.
Compute Hardware Node (m1.class)
128GB memory, (16) 1TB disks, (2) E5-2670 CPU
1/1 1/2 1/4 1/8
47

48. Deployment: Sizing a Compute Node
xlarge
Compute Hardware Node (m1.class)
128GB memory, (16) 1TB disks, (2) E5-2670 CPU
xlarge
small
small
small
small
small
small
small
small
medium medium
medium medium
large
xlarge xlarge
large
small
small
small
small
small
small
small
small
Given the machine config above, it would support:
(4) n1-standard-8-d, (8) n1-standard-4-d, (16) n1-standard-2-d, (32) n1-standard-1-d
(8) m1.xlarge, (16) m1.large, (32) m1.medium, (64) m1.small
48

49. Plan for the Resource Service Level
Compute/StorageNetwork Fabric
Cloud Controller
Resource
Service
Level
49

50. OpenStack Deployment Profiles
50

51. PayPal
Profile Highlights:
✦113 million registered accounts; PCs or mobile devices
in 190 different markets with 25 different currencies
✦Targeting 90 percent coverage for several thousand
nodes in nine to 12 months
✦DIY team leveraging OpenStack community support
51

52. OpenStack Accelerating Science
Profile Highlights:
✦Using OpenStack to support particle
physics research
✦Currently running 500 nodes and 2000 VMs
✦Immediate plans:
✦Deploying production OpenStack running Grid software
✦Intends to use Ceilometer, bare metal for tenants and LBaaS
✦Ramping to 15,000 hypervisors with 100k - 300k VMs
by 2015
52

53. National Security Agency
Profile Highlights:
✦One of the NSA's largest hosting platforms
✦Number of users, systems, servers, storage, applications
users: [REDACTED]
✦Agility, flexibility and scalability providing better
support for mission systems
✦Big Data is truly big
✦Has been working with OpenStack since Cactus
53

54. In Summary
54

55. OpenStack ...
✦Is open source software and vibrant community
✦Provides a framework for an elastic cloud.
✦Requires fresh thinking for deployments
✦Is being deployed successfully at scale today
55

56. And finally..
56

57. 57

58. Thank You!
Red Hat IaaS Overview & Roadmap
Andrew Cathrow — Sr. Virt. Product Manager, Red Hat
Simon Grinberg — Principal Product Manager, Red Hat
Wednesday, June 12
3:40 pm - 4:40 pm
Red Hat OpenStack Performance & Scale
Mark Wagner — Senior Principal Engineer, Red Hat
Wednesday, June 12
4:50 pm - 5:50 pm
OpenStack Architecture
Russell Bryant — Principal Software Engineer, Red Hat
Thursday, June 13
3:40 pm - 4:40 pm
Check out these sessions!
58