KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
Container Conf 2017: Rancher Kubernetes
1. DEPLOY, MANAGE & SCALE
KUBERNETES WITH
RANCHER
BANGALORE CONTAINER CONFERENCE
7TH APRIL 2017
VISHAL BIYANI
RANCHER & INFRACLOUD
2. VISHAL BIYANI
CTO & Founder at
infraCloud technologies (www.infracloud.io )
2004
Java, PLM, JSP,
Servlets
2004-2009
eMatrix PLM, J2EE,
Database, architecture, Shell
and what not
2010 - 2013: Spring,
Maven, Jenkins,
ElasticSearch, CloudFoundry,
Google App Engine, APIs, CI
2013: Puppet, Chef, Ansible,
CD/CI, DevOps Coach, Docker,
API Mgmt, Microservices, Infra
as code
Now:
Containers,
Kubernetes, Mesos,
Salt, Scale, Distributed
https://twitter.com/vishal_biyani
https://www.vishalbiyani.com
3. infraCloud is a Rancher consulting partner
http://rancher.com/partners-index/
Rancher has published a FREE eBook on
“Scaling and deploying Kubernetes”
http://info.rancher.com/deploying-scaling-kubernetes-ebook
4. The average
company
QUINTUPLES its
Docker usage
within 9 MONTHS1
There are 460K
Dockerized apps, a
3100% GROWTH
over 2 years2
Docker containers
have been
downloaded more
than 4 BILLION
times3
THE MOMENTUM OF CONTAINER ADOPTION IS UNDENIABLE…
4
1 Datadog, June 2016
2 Coscale, July 2016
3 Docker, November 2016
6. A COMPLETE CONTAINER MANAGEMENT
PLATFORM THAT MAKES IT EASY TO…
6
INNOVATE WITH CONTAINERS
by empowering developers with fast access to the latest tools
SIMPLIFY APPLICATION DEVELOPMENT
with a powerful, yet easy to use interface and application catalog
RUN CONTAINERS
with the most complete set of container and infrastructure management capabilities
Enterprise ready
✔ Open platform for
innovating
✔ Easy to use
interface
✔ Multi-tenancy
✔ Role based access
✔ 24X7 support
✔ And more….
9. CHALLENGES : KUBERNETES ONLY IMPLEMENTATIONS
• Creating a Kubernetes environment that is customized to DevOps needs
• Automating the deployment of multiple Kubernetes clusters
• Managing the health of Kubernetes clusters
• Automating the upgrade of Kubernetes clusters
• Deploying multiple clusters on premises or across disparate cloud providers
• Ensuring enterprise readiness, including access to 24×7 support
• Customizing then repeatedly deploying multiple combinations of infrastructure
services (e.g. storage, networking, DNS, load balancer)
• Deploying and automating upgrades for Kubernetes add-ons such as Dashboard,
Helm and Heapster
10. RUNNING CONTAINERS IN PRODUCTION IS HARD,
RANCHER MAKES IT EASY
10
Develop Build Package Test Deploy/Upgrade Operate
Docker Hub
12. WORKSHOP AGENDA
• Infrastructure Side
• How to modify and maintain
multiple Kubernetes
configurations easily
• Configure separate data, cluster &
worker nodes
• Configure Kubernetes cloud
providers
• NFS & EBS configuration
• Configuring Network types: IPSec
& VXLan
• Application Side
• Deploy applications with Helm
chart
• Auto creation of disks and ELB in
action
• Custom Registry
• Auto Scaling of hosts
• Hosts upgrades
13. SETUP WITH DIGICAL OCEAN
• Use the promo code DOBCC. It will give you $15 worth of credits on DigitalOcean
platform. Please note the following:
• a) You can sign up for an account
@ https://cloud.digitalocean.com/registrations/new. The above promo code will
add credits only to new DigitalOcean accounts.
• b) Adding a payment option (credit/debit card or Paypal) is part of the sign up
workflow. To verify the authenticity of the card, sometimes the payment gateway
does an authorization charge of around $1 but this charge gets reversed
immediately after the card has been verified.
• c) Once the above promo code is applied, $15 in credits will be added to your
account which can be used for anything on the
14. ENVIRONMENT TEMPLATES
• Creating and customizing templates for different requirements in an
organization
• You can have different storage, networking and other requirements in
different units/projects
• You might want a true HA setup for Pre-prod/prod where as a simple
setup for Development environment
• Rancher enables this with template stacks - official as well as
community supported.
• You can create multiple environment templates and can launch
environments based on template
16. RESILIENCY PLANES
• Objective: Achieve separation between data,
Orchestration and compute nodes.
• Data - Used by Etcd to store all data
• Recommended minimum 3
• Orchestrate - for Kubernetes
• Recommended minimum 2 (For HA)
• Compute - for actual workload
• 1 or more
• You can not change a node type from one
resiliency plane to other
etcd=true
orchestrate=true
compute=true
1 2 3
1 2
1 N
17. CLOUD PROVIDER CONFIGURATION
• Kubernetes cloud providers: interface to underlying cloud
provider
• Useful for things such as: Load balancer, Node management,
Networks etc.
• Rancher comes built with two cloud providers: Rancher & AWS
• AWS provider can be used for ELB, EBS and Node management
• Rancher provider is useful for Nodes & HAProxy based load
balancers
18. DNS - USING DIGITAL OCEAN
• Enables quick and easy integration with DNS (AWS Route53, Digital Ocean DNS etc.)
• Each service of type Load Balancer - gets the load balancer auto provisioned and DNS
record created.
• DNS record is customizable
19. RANCHER NETWORK SERVICES
VXLan (Overlay)
• Unencrypted traffic
between hosts
• Good if underlying network
is secure
• Faster Configurable MTU
IPSec (Overlay)
• Encrypted traffic between
hosts, MTU configurable
• Good for public clouds
• Relatively slow due to
encryption overhead
More plugins coming for
• Calico
• Weave etc.
Network Manager
• Interface to CNI plugin &
responds to add/remove
container events
• Takes care of part mapping
(Initial CNI did not have it)
Rancher DNS
• DNS Service within cluster,
communicates with
upstream DNS
• Provides service discovery
in cluster
Rancher - Metadata
• Metadata agent runs on all
hosts
• Provides Service Discovery
locally
Networking Under the hood
All three components are open source
20. RANCHER HEALTH CHECK
• Health check stack is one of infrastructure stacks
• Launched as a set of containers and utilized HAProxy internally to
validate health of containers
• Containers are checked for health from multiple health check
containers
• If even one of health check containers respond positive on a service -
then it is good
• If all of health check containers respond negative on a service, then it
is assumed down
21. PORTAINER
• We deployed Portainer as part of the stack, which is a simple UI for
containers.
• The Dashboard is reachable at http://rancher-
server:8080/r/projects/1a5/portainer/ (Just open the Kubernetes
dashboard UI and change the URL)
• Portainer is simple utility and shows containers on a host
• This shows how easy it can be to deploy custom
utilities stacks on top of Rancher
22. POWERFUL COMPOSITION
• Every stack is a rancher-compose + docker-compose
• You can custom create complete stack, upload and have a new
environment template
23. AWS CLOUD PROVIDER BASED ENVIRONMENT
• Create a AWS cloud provider based Kubernetes environment
template and an environment
• Create Roles for instance profiles for the Kubernetes master &
agent - this enables the instances to attach disks or create ELB
and so on
• Create 4 hosts - one master & 3 nodes and install docker on
them
• Add the hosts manually to the Rancher environment
• See the environment build up
25. WALKTHROUGH OF INFRA STACKS
• Health check stack for health
checks
• IPSec networking for
encrypted overlay traffic
• Ingress controller for LB and
Ingress management
• Supporting Network services
- NW manager and metadata
• Portainer as a utility
• Scheduler framework for
additionally scheduling
26. WALKTHROUGH OF INFRA STACKS
• Kubernetes stack for all
core components
• Controller manager
control nodes,
endpoints etc.
• Kubernetes - API Server
• Ingress controller for
ingress & LB
management
• Core Scheduler
27. SAMPLE APPLICATION DEPLOYMENT
• We will use a Helm chart to deploy WordPress stack - which
contains the WordPress app & MySQL DB
• MySQL DB needs a persistent disk - which be auto provisioned
for us.
• We also need a LoadBalancer - which will be auto created.
• We won’t use DNS like we did in last example, but that is
possible too.
28. HELM ON MY MACHINE
• Configure ~/.kube/config file - verify with kubectl
• ‘helm init’ - initializes all directories and standard repo
• helm search WordPress
• helm install --name bcc-release stable/wordpress
29. OH, WAIT, WHAT IS HELM?
• Helm is a package manager for Kubernetes
• Tiller - Repo Server
• Chart - a package
• Helm is the client for Tiller
• Charts are in a repo (Typically some Git repo)
• A chart - is set of manifests
• The values can be defaulted to or overridden as input from user
• A chart is released as a release so that it can be tracked.
30. IS WORDPRESS DEPLOYED?
• Deployments for WordPress
created
• Services created
• Volumes auto created
• ELB auto created
31. MORE VALIDATIONS
• PV & PVC created using the
default storage class
• And we can reach our blog:
33. HOST EVACUATION
• You want to upgrade a host for some security patches or
some change
• But without disrupting normal operations
• Evacuation helps you reschedule pods to other hosts,
gracefully!
34. CUSTOM REGISTRY ADDITION
• You can use Docker hub or any private registry
• Host dockercfg is auto populated - so images can be pulled
from those registries
35. RECEIVER HOOKS
• Like webhooks - can be used to
invoke actions in Rancher
• Can be tied to let’s say monitoring
system
• Possible to achieve auto - host
scaling & service upgrade as of
today.
• More actions & “Kind” of hooks
coming soon
36. AND IT COMES WITH AN API
• Rancher has a comprehensive
API - and all actions can be
done via API
• API is well documented, has in
browser accessibility and is
exhaustive
• Rancher also comes with a CLI
38. INGRESS: LOAD BALANCERS
• For an ingress you need a load balancer.
• Rancher creates/updates/manages Rancher load balancers based on ingress lifecycle, using rancher ingress
controller.
• This also makes usage of ingress easier outside a cloud provider.
• Rancher load balancers support
• Host/path based routing
• TLS
• Advanced targeting and scheduling of load balancers.
Editor's Notes
The momentum of container adoption is undeniable:
The average company QUINTUPLES its Docker usage within 9 MONTHS
There are 460K Dockerized apps, a 3100% GROWTH over 2 years
Docker containers have been downloaded more than 4 BILLION times
But running containers in production still isn’t easy…
Growing number of tools with a high rate of change is significantly increasingly the complexity of building, deploying and updating containerized environments
Increased density combined with decreased lifespans (containers lifespan 1/6 of a VM) significantly increases the volatility of containerized environment compared to a traditional VM environment - there is an order-of-magnitude increase in the number of things that need to be individually managed and monitored, which significantly increases the volatility of the environment.
Rancher is a complete container management platform that makes it easy to…
Run containers in production with the most complete set of container and infrastructure management capabilities
Manage applications by simplifying day to day application lifecycle management
Innovate with containers without compromising flexibility by empowering developers with fast access to the latest tools
And Rancher is production ready
Enterprise-class features such as role-based access control, integration with LDAP and Active Directories, detailed audit logs, high-availability management servers and encrypted networking are available out of the box.
With over 2.4 million downloads and optional 24x7x365 support, Rancher has quickly become the platform of choice for DevOps and IT teams who are serious about running containers in production
More than 30 customers are using Rancher in production including large enterprise such as US Bank, IBM, Cisco, Invisalign and more.
There’s a slew of technologies that are in the ecosystem and they all serve useful purposes but it’s paralyzing having to deal w/all of the individually. Very flexible working w/all of these open source projects but it’s challenging updating and keeping in sync w/all of them.
Rancher provides a turnkey container service and looked at all of the the technologies and taking ownership of them as it relates to being deployed at customer site.
Rancher ties all of this together and makes it Easy to deploy, easy to support, easy to scale.
Challenge is integrating it into your org and make use of the investments you’ve made
Environment templates are ways to create blueprints for different environments you need. You can customize the blueprint for various aspects such as storage, networking, DNS etc.
In some environments you might want a fully HA setup where as in another environment you might be ok with a non HA setup. The way to achieve this is to create separate environment templates for different needs and then launch environments from templates. Of course you can launch more than one environment from same template.
Some of stacks available in environment customization might be community supported and some are officially supported by Rancher - be sure to choose the right one.
Now let’s quickly create an environments which we will use for the demo today (We will create another one in a bit).
In the environment - we will configure the digitalocean-dns, healthcheck, kubernetes, network-services, portainer, ipsec, scheduler. Also we choose not to use resiliency planes in Kubernetes for simplicity, but we will discuss about it.
This slide will eventually be removed in the actual version of presentationa and a live demo will be given instead. This slide only demonstrates the rough flow to be demoed.
Resiliency Plaines allow you to build system that are highly available and fault tolerant. This is specially important when you have to build systems for production grade workloads. In Rancher, for a given environment template you can enable or disable resiliency planes. If you enable them you have to use nodes labels to identify the hosts belonging to certain plane.
At a minimum it is good idea to have 3 node for etcd so that the data plane can tolerate failure of one node. If you need higher fault tolerance then you can go for 5 etcd nodes - which provides a tolerance of upto 2 hosts failing.
For the Kubernetes or orchestrate layer, you need at least two nodes. On compute side you can have minimum 1 but in real world you might have many more hosts in compute plane.
Cloud provider configuration enables you with native integration with the cloud provider so you can seamlessly use some of underlying resources. At the moment rancher supports two cloud providers - one is rancher and other is AWS. There might be more in near future. With Rancher cloud provider you can get load balancing even without a cloud provider. The rancher provided load balacing is based on haProxy.
With AWS you get native integration with EBS, ELB & EC2 instances. For a service type Load Balancer, an ELB is provsioned, although you can create a Rancher load balancer using Ingress.
We are using digital ocean demo for this one but you could also use the stack for Route53. The basic idea is that foe every service you want to expose outside of cluster, you simply create LoadBalncer type service and the DNS records are managed automatically based one semantics. And the format that the service name should put in DNS is customizable.
So here is how it works - you of course need to have a domain name. Then you need to create a hosted zone(AWS)/cloud DNS (Google Cloud)/DOMAIN (Digital Ocean) for that domain name. Ensure that you update the name servers in your domain name registrar to point to correct name servers of cloud provider. Once this is done, then in the template we simply provide DO access key and the name of domain name that needs to be managed.
Now as you create services - we will see two things:
How when you create a service - the Load Balancer using HA Proxy is created (Which is what we discussed in previous slide)
How a DNS entry is made in Digital Ocean Domain and updated with IP.
Rancher provides
In second environment, create a AWS cloud provider based Kubernetes environment. Then create hosts manually and apply them Instance Roles - separate once for master & agent machines. Then custom add them to Rancher.
This slide is temporary - and actual demo will be shown here.