According to OpenStack users survey, Cloud Foundry is the 2nd most popular workload on OpenStack. You want to deploy Cloud Foundry on OpenStack or already have. What's next? Cloud Foundry continues to evolve with revolutionary changes, e.g move from bosh-micro to bosh-init, using the new eCPI, move to Diego etc. Same with OpenStack, e.g changes from Keystone v2 to v3, from Liberty to Mitaka, network plugins changes etc. Both IaaS and PaaS layers are changing frequently. How do you do in-place updates/upgrades/operational tasks without impacting user experience at both the layers? In this talk will discuss our lessons learnt operating hybrid Cloud Foundry deployments on top of OpenStack over the last two years and how we used underlying technologies to seamlessly operate them

1. As a Service: Cloud Foundry on OpenStack - Lessons Learnt
Apps
@AnimeshSingh
OpenStack Summit
Barcelona
October 2016
@blueboxjesse

2. A polyglot “platform for the people”
• The de facto open PaaS platform
• Foundation established Dec. 2014, under Linux Foundation umbrella
Cloud Foundry

3. Cloud Foundry Community
60+ member companies driving Cloud Foundry.

4. PaaS
Cloud Foundry on OpenStack
IaaS
UAA
Router
DEA Pool
Service Gateway Apps
Service Connector
Health Manager
Messaging
Cloud Controller
Build Packs
Service Nodes
Cloud Foundry BOSH
Cloud Provider Interface

5. BOSH Deployment Process
Deployment Manifest
• Release name/version
• # VMs, job params
• Stemcells to use
Stemcell
• Base OS
• BOSH agent
Release
• Name
• Software packages
• Config templates
• Scripts
BOSH
Cloud Foundry
Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software packages

6. © IBM
Corporation
6
Problems:
Faced, by us
and by the
community!

7. Problems we faced
As CF and OpenStack deployers we experienced various issues deploying CF and OpenStack, and CF on OpenStack
§ Instability: Instability of OpenStack environments from various distros
§ APIs: Lack of compliance in APIs and new releases. Some of the API behavior also differs based on the plugins
used
§ Capacity: Right from cpu/mem/disk etc. to HA, persistent disk, floating ips etc. – sizing needs to be precise
§ Network: Should CF co reside with other management components in a network?
§ OpenStack for Enterprise Software: Enterprise software lags, with most variations still available only for VMware.
§ Generic Deployment: Consistency when deploying CF on OpenStack, VMware or any other IaaS
§ Combined OpenStack and Cloud Foundry Usage: How to allow seamless usage and consumption of OpenStack
services along with CF services at the same time?
§ CF HA: What works, what doesn’t?
§ Proxy: Environments behind customer firewalls with restricted outbound access
§ Constant Release Cycles: Both CF and OpenStack have frequent releases, patches, updates etc. How do we
reconcile?7

8. 50% experience significant
issues deploying CF onto OS
45% are intermediate OS users
Problems Community Faces: Survey Results

9. Close to 50% of users had to
customize their environment
for CF
Problems Community Faces: Survey Results

10. Most users on Juno/Kilo
Close to 50% of users run
their own local OpenStack
Problems Community Faces: Survey Results

11. © IBM
Corporation
11
Our OpenStack
and Cloud
Foundry
Offerings

12. APACHEHAPROXY APACHEHAPROXY
HA MYSQL
ARBITER
(PERCONA)
HA NEUTRON!
HA KEYSTONE!
HA GLANCE!
HA HORIZON!
NOVA!
CINDER API & ISCSI!
HA NEUTRON!
HA KEYSTONE!
HA GLANCE!
HA HORIZON!
NOVA!
CINDER API & ISCSI!
HA MYSQL
(PERCONA)
RABBITMQ
HA MYSQL
(PERCONA)
RABBITMQ
Our OpenStack Offering - Bluemix Private Cloud (IaaS)
NOVA
CINDER API & ISCSI
NOVA
CINDER API & ISCSI

13. § IBM Platform as a Services offering
• Three deployments styles: public, dedicated, private
• 1M+ registered users (20K+/month)
• 100K+ running apps and 500+ services
Bluemix Bare Metal (public and dedicated) and OpenStack/VMware (private)
Our Cloud Foundry Offering - Bluemix (PaaS)
Services
Lifecycle
Management
IDS
Application
Runtime
Runtimes &
Frameworks
Middleware Application Operational Mobile ExternalData
Node Java Ruby Worklight
WebSphere
Liberty
Eclipse IDE
Application
Composition
Environment
Create & Manage Services
Test/Run Test/Run
Explore
Services
Explore
Services
IBM Bluemix
Check In Code Check In Code
Web IDE
(Eclipse Orion)

14. © IBM
Corporation
14
Lessons Learnt
deploying
Cloud Foundry
on OpenStack

15. Test if your OpenStack is fit for Cloud Foundry deployment
Manually, you can run the validations here to see if your OpenStack is a good fit:
https://docs.cloudfoundry.org/deploying/openstack/validate_openstack.html
§ Can you access the OpenStack APIs for your instance of OpenStack?
§ Can you access OpenStack metadata service from a virtual machine?
§ Can you ping one virtual machine from another?
§ Can you invoke large numbers of API calls?
§ Can you create and mount a large volume?
§ Can you upload and deploy an Ubuntu Server Cloud Image?
§ Can networking be configured for both external and internal IPs?
§ Can you access the Internet from within instances?

16. Test if your OpenStack is fit for Cloud Foundry deployment
To check in automated fashion if your OpenStack is ready to run BOSH and Cloud Foundry,
you can run this CF Incubator project :
https://github.com/cloudfoundry-incubator/cf-openstack-validator
16
Testing the CPI API
Upload stemcell
Create VM
Find VM
Create disk
Find disk
Attach disk to VM
Detach disk from VM
Create disk snapshot
Delete disk snapshot
Delete disk
Delete VM
Delete stemcell
Other OpenStack tests
Check API rate limit
Check required versions of OpenStack projects
CPI requires API version 1 for glance and cinder
Security group settings
Check if security group rules allow necessary incoming/outgoing
ports
Outbound internet access
Timeservers can be reached
Attach a floating IP
Set VM metadata tags
Access a VM over ssh from the outside
Access one VM from another VM
Create a large volume

17. Get the right sizing for Cloud Foundry
• Sample sizing are available online as references e.g
https://docs.cloudfoundry.org/deploying/openstack/required-flavors.html
https://www.mirantis.com/blog/full-stack-devops-with-pivotal-cloud-foundry-on-mirantis-openstack
https://docs.pivotal.io/pivotalcf/1-7/customizing/openstack.html#versions
• Increase default quota of the tenant to meet minimum requirement
e.g number of ports always gets hit first
• Recommended ‘OpenStack flavors’ disk space should be go in ephemeral disk.
Keep root disk to 10GB minimum

18. Optimize OpenStack scheduling and communication
Optimize Internal Communication:
• Configure OpenStack for high volume API calls
e.g Increase OpenStack API rate limits (/etc/nova/api-paste.ini)
• Avoid name based security groups with nova-network. Name
based security groups require message bus activity and database
updates proportional to the number of existing VMs
• If Neutron is configured with VXLAN via the Open vSwitch
mechanism, the MTU should be 1400. For GRE, the recommended
number is 1460.
Use the right Scheduler
Use an OpenStack scheduler which evenly distributes load
e.g compute_scheduler_driver =
nova.scheduler.filter_scheduler.FilterScheduler

19. Account for different OpenStack configurations
• If your OpenStack setup requires you to use disks from block
storage instead, that will work with Cloud Foundry as well.
properties. openstack.boot_from_volume: true
• By default, the VMs created try to receive data from OpenStack's
HTTP metadata service. If your OpenStack installation doesn't
provide metadata and userdata over HTTP, but requires you to use
config-drive instead of metadata, you need to specify this in the
property
properties.openstack.config_drive: cdrom

20. Optimize Cloud Foundry / BOSH bandwidth and communication
Optimize Internal Communication:
• Increase BOSH NATS time out for high concurrency
e.g Configure NATS messaging bus to increase ping interval
Optimized routing and bandwidth allocation
• Isolate Cloud Foundry components using multiple networks
e.g DEAs in their own network, brokered services in their own
network
Optimize log forwarding
• Supporting services like logging, report generation etc should
go in their own tenant network. Also any communication
between the VM(s) sending logs, and log receiving
component(s) should go over the private network. Don’t
use floating ips as destination, else you are paying double the
cost.

21. Optimize for Security and Network
• Only open ports which are needed. Use the most limited
permissions required to complete the job.
• If OpenStack is using a self-signed certificate, configure
properties.openstack.connection_options to include the
property ca_cert
• Use tenant credentials: Do not use full admin credentials in your
BOSH manifest
• Minimize floating ips: Except for the incoming Gateway
device(HA Proxy, Datapower, F5 etc), none of the fabric VMs
should be on public network or need a floating ip

22. Account for customer boundary firewall or proxy
• BOSH will retrieve CF release components via the
URL provided in deployment manifest. Also services
and CF apps may need outbound access. In
environments behind firewall or proxy with restricted
access, this could be a major issue.
• If the firewall or proxy requires both destination
and source ips to be enlisted, prepare a list of
destination IPs/URLs you need to reach out and
hand it to datacenter admin
• If your OpenStack VMs don’t have a floating ip from
external network, the source ip presented to
firewall will be Neutron gateway ip
• Cloud Foundry doesn’t support out of the box
SSL packet inspection where the SSL certificate
for a given sight is replaced by your own self signed
certificate. Inspection is only supported using an
Internet Authority signed certificate.

23. Work on seamless OpenStack update/upgrade
OpenStack separates the availability of workloads (Data Place) from the availability of the API services (Control Plane)
OpenStack Code Major Upgrade Maintenance: Moving from one OpenStack release to another, e.g. from Kilo to Mitaka with
data migration
• Infrequent, twice a year at most. New OpenStack code is put in place for all of the services that make up a cloud (e.g. Nova,
Neutron, Glance, Heat, Cinder, Ceilometer, Aodh, Swift, etc..).
• ~10 minutes for each service
OpenStack Code Minor Upgrade Maintenance: Upgrading OpenStack service, without data migration
• Typically only bug fixes. New OpenStack code is put in place and the services associated with that code are simply restarted.
• ~5 seconds for each service
OpenStack Config Maintenance: Changing the configuration for one or more of the OpenStack services, needed to correct bugs in
the service, or to tune the service due to consumer requests.
• Require the restart of a particular service. The disruption to the API control plane is isolated to the services receiving configuration
updates.
• ~ seconds for each service

24. © IBM
Corporation
24
Lessons Learnt:
Automate
everything

25. OpenStack Installation:
• Leverage the open source Ursula Ansible and Rally Cloud infrastructure Automation framework
• Requires information about hardware, network environment and software repositories.
Bluemix Private Cloud Install Automation
Setup Storage
Setup OpenStack
Setup Network
Run validation
Setup Hardware
Ursula, Rally-OpenStack

26. OpenStack Discovery:
• Leverage the open source Fog gem to discover OpenStack artifacts in an automated manner
• Require OpenStack credentials and discover OpenStack compute and network information.
Cloud Foundry on OpenStack deployment Automation
Discover VM
Configuration Sizes
Discover Network
Subnets
Discover Network
Security Rules
Discover DHCP , DNS
Gateway and floating IPs
Discover Security
Credentials

27. Cloud Foundry Pre-req setup on OpenStack:
• Leverage the open source Fog gem to setup Cloud Foundry requirements in an automated manner
• Setup according to best practices and guidelines – still giving users the flexibility to change if desired
Create Security
Credentials
Create VM configs for
Router, DEAs, Cloud
Controller, Service
Nodes
Create network
Security Rules
Setup tenant quota
Cloud Foundry on OpenStack deployment Automation

28. Cloud Foundry Deployment Automation
• Automate stemcell modification
• Automate Cloud Foundry deployment manifest file genration using Ruby ERB
• Automate upload of Cloud Foundry core release, services and runtime frameworks, followed by Cloud
Foundry deployment
Stemcell Creation and
Upload
Generate BOSH and
Cloud Foundry
Manifest
Upload Cloud
Foundry core,
Services and runtime
Deploy Cloud Foundry
Deploy bosh director
RUBY BOSH
Cloud Foundry on OpenStack deployment Automation

29. © IBM
Corporation
29
As a Service:
As a provider,
look into offering
Cloud Foundry
and OpenStack
as a Service

30. Why as a Service?
Cloud Foundry:
– New release every 2-3 weeks
– Bluemix PaaS is a combination of CF and 150+ Services
– Older versions will lead to huge version mismatches, and lead
to version sprawl
– Keeps Public/Dedicated and Local Bluemix in sync
OpenStack:
– Twice annual releases that touch the entire code base.
– Upgrading sequentially is important: stay up to date!
– OpenStack’s complexity requires expertise in many operational
areas
– Focus on higher business value. Work with OpenStack, not on
OpenSTack

31. Private
Cloud
Hardware
Bluemix Private Cloud: IaaS Relay: Box Panel
Box Panel
Site Controller (Software)
Bluemix
Private
Cloud
OpenStack
Box Panel
Formations
Central Authentication
Customer Relationship Management
Service Catalog and Metering
Billing and Invoicing
Object
Storage
Block
Storage
Core
Networking
Inventory Management
Network Management
Reporting and Analytics
Support Ticking, Chat and Email

32. IaaS Relay: Box Panel

33. IaaS Relay: Site Controller
Box Panel
Site Controller
Customer Cloud
Resides on-premises adjacent to customer clouds, providing
real-time administrative control of cloud environments.
– Network Automation
– Power Distribution Unit Automation
– System and Network:
• Monitoring
• Telemetry
• Logging
– Secure Remote Control and Access
– Bare Metal Provisioning
– Package and Container Repo

34. SDN router
The Internet
Customer Network
IBM Urban Code
Deploy
Softlayer Server
Bluemix Platform
Stemcells
Releases
Manifests
BOSH CLI
Automated
Management
Processes
(Deploy, Upgrade etc.)
IBM Urban
Code Deploy
Relay
Customer Hardware & Infrastructure
Bluemix Core Services
• Monitoring & Logging
• Cache
• Cloudant (Data Store)
• Qradar EPS
• IEM Relay
Configuration
Store
(per customer)
Bluemix
Code &
Automation
Repository
Opensource Code
IBM Test & Staging
Validation
IBM Production
Deployment &Validation
Bluemix Local
Inception VM
UCD Agent
• Secure connection
• Connection originated from
customer premise
• Restricted access (agent-
only)
Cloud Foundry
ACE UI
Enterprise ITSM
Customer Services
Customer
Premises
Premises
On-premise data store for
logs, monitoring data etc.
Bluemix Ops Console
IBM Cloud Security Services
Qradar Console
IEM
Server
Bluemix Ops
Directory
Server
Privileged ID
Governance
IPSec Tunnel
DataPower
• Customer’s Service
traffic
• Syndicated & 3rd party
service traffic
• App staging artifacts
• Inbound & outbound
user to app traffic
• LDAP
• Enterprise services
• Other SaaS services
• vCenter
Network
IsolationCustomer Services
Bluemix Private Cloud PaaS Relay IBMPremises

35. PaaS Relay: Operations Console

36. PaaS Relay: Operations Console

37. VPN Tunnel
Inception VM
Stemcells
Releases
Manifests
BOSH CLI
DataPower
ACE UI
Metering
Admin UI
NATS
BM DB
Backup
Login server
UAA CC
Blobsto
re
HM
CCDB
Loggre
gator
Go router
DEAs
UAADB
Logging
UCD AgentOps Center
Agent
VPN Tunnel
Bluemix Private Cloud: PaaS and IaaS As a Service!
Bluemix Relay
Bluemix
Private Cloud Relay
Site Controller
Repo / Deploy Server
Monitoring Server
(Sensu)
Logging (ELK)
Bastion (Access)

38. Cloud Foundry on OpenStack – A Great Fit!
• 100% Open PaaS and IaaS solutions – No vendor lock-ins
• Strong and growing community of contributors and sponsors on both sides
• Power of Open Source community can be leveraged to automate the deployment and
lifecycle management of Cloud Foundry on OpenStack
• OpenStack meets Cloud Foundry integration requirements, and is totally configurable
and adaptable to handle the scale of a PaaS solution like Cloud Foundry
• Bottom Line: It’s a match made in Heaven !