Technical Track presented by Yudai Iwasaki, Lead Engineer, Software Innovation Center, NTT. Yudai Iwasaki is a research engineer at NTT Software Innovation Center. He is a core member of the development team of Cloudn PaaS, which is a public PaaS solution provided by NTT Communications, and is leading the development of their deployment system for Cloud Foundry. He is also the leader of community relationships at NTT and is a member of the Japan Cloud Foundry Group, in which capacity he gives lectures on the structure of Cloud Foundry for Japanese Cloud Foundry developers. He is a developer of Nise BOSH, which is a light weight BOSH emulator, BOSH CloudStack CPI, and BOSH AutoScaler.

1. © 2014 Nippon Telegraph and Telephone Corporation
AUTOSCALING
CLOUD FOUNDRY WITH BOSH
Yudai Iwasaki
NTT Service Innovation Laboratory Group
CF Summit 2014

2. © 2014 Nippon Telegraph and Telephone Corporation
• Core developer of Cloudn PaaS
• Working on Cloud Foundry since 2012
– Nise BOSH, BOSH CloudStack CPI and BOSH AutoScaler
• Twitter: @i_yudai

3. © 2014 Nippon Telegraph and Telephone Corporation
Agenda
• Problem
– System load is not fixed
• Solution
– BOSH AutoScaler
• Getting Started

4. © 2014 Nippon Telegraph and Telephone Corporation
Problem:
System Load is
Not Fixed

5. © 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks

6. © 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks

7. © 2014 Nippon Telegraph and Telephone Corporation
Manually Scaling?
Always monitor load?

8. © 2014 Nippon Telegraph and Telephone Corporation
Impossible
No one wants to do such a boring job

9. © 2014 Nippon Telegraph and Telephone Corporation
Solution:
BOSH AutoScaler

10. © 2014 Nippon Telegraph and Telephone Corporation
BOSH AutoScaler
• Extension for BOSH
• Flexible scaling policies in deployment
manifest files
• Special support for Cloud Foundry

11. © 2014 Nippon Telegraph and Telephone Corporation
Architecture
Overview

12. © 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler
(with CF Plugin)
Heartbeat
Heartbeat
(subscribe)
process
(BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy
(PUT /deployments)
matching rules against log
periodically
Logging metrics（@buffers）run()
Agent
Agent
Agent
Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component
CF Component
CF Component
Collector
TSDB
Varz
Load policies
(GET /deployments)

13. © 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler
(with CF Plugin)
Heartbeat
Heartbeat
(subscribe)
process
(BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy
(PUT /deployments)
matching rules against log
periodically
Logging metrics（@buffers）run()
Agent
Agent
Agent
Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component
CF Component
CF Component
Collector
TSDB
Varz
Load policies
(GET /deployments)

14. © 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler
(with CF Plugin)
Heartbeat
(subscribe)
process
(BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy
(PUT /deployments)
matching rules against
log periodically Logging metrics（@buffers）run()
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
Collector
TSDBLoad policies
(GET /deployments)

15. © 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler
(with CF Plugin)
Heartbeat
(subscribe)
process
(BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy
(PUT /deployments)
matching rules against
log periodically Logging metrics（@buffers）run()
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
Collector
TSDBLoad policies
(GET /deployments)

16. © 2014 Nippon Telegraph and Telephone Corporation
Policy Definitions

17. © 2014 Nippon Telegraph and Telephone Corporation
Defining policies
• Adding “scale” section
• Defining conditions for
each job
• Various condition classes
to define flexible policies
-­‐-­‐-­‐
name:
cf
....
scale:
jobs:
-­‐
name:
router
cooldown:
300
out:
limit:
10
unit:
2
conditions:
-­‐
class:
CpuAverage
larger_than:
80
duration:
300
-­‐
class:
MemoryAverage
larger_than:
90
duration:
300
in:
limit:
3
conditions:
-­‐
class:
CpuAverage
smaller_than:
10
duration:
300
-­‐
class:
MemoryAverage
smaller_than:
20
duration:
300

18. © 2014 Nippon Telegraph and Telephone Corporation
Conditions: BOSH Heartbeat
• CpuAverage
– Average CPU percentage for duration
• MemoryAverage
– Average memory percentage for duration
• LoadAverage1
– Latest Load Average in 1 minute
• LoadAverage5
– Latest Load Average in 5 minutes
• LoadAverage15
– Latest Load Average in 15 minutes

19. © 2014 Nippon Telegraph and Telephone Corporation
Conditions: CF Plugin
• CFVarzAverage
– Average Varz value for duration
DEA
available_memory_ratio
available_disk_ratio
Router
total_routes
latency.1m
HM9000
NumberOfRunningInstances
NumberOfCrashedInstances
etcd
SendingRequestRate
ReceivingRequestRate
Loggregator Server
receivedMessageCount
numberOfWebsocketSinks
Cloud Controller
connection_count
threadqueue.num_waiting

20. © 2014 Nippon Telegraph and Telephone Corporation
Web UI

21. © 2014 Nippon Telegraph and Telephone Corporation

22. © 2014 Nippon Telegraph and Telephone Corporation

23. © 2014 Nippon Telegraph and Telephone Corporation

24. © 2014 Nippon Telegraph and Telephone Corporation
Getting Started
It’s easy to plug in

25. © 2014 Nippon Telegraph and Telephone Corporation
https://github.com/nttlabs/bosh-scaler

26. © 2014 Nippon Telegraph and Telephone Corporation
-­‐-­‐-­‐
#
BOSH
NATS
settings
nats:
&nats
uri:
mbus://192.168.50.4:21084
user:
nats
pass:
nats
#
BOSH
Director
REST
API
settings
rest:
&rest
endpoint_uri:
https://192.168.50.4:25555
user:
scaler
password:
scaler
disable_verify_certification:
true
collectors:
-­‐
class:
BoshNatsCollector
bosh_nats:
*nats
bosh_rest:
*rest
-­‐
class:
CfVarzTsdbCollector
port:
4567
listeners:
-­‐
class:
BoshScaler
bosh_rest:
*rest
interval:
60
#
seconds
buffer_size:
1000
ui:
enable:
true
port:
8888
Configuration

27. © 2014 Nippon Telegraph and Telephone Corporation
...
properties:
collector:
use_tsdb:
true
deployment_name:
cf
opentsdb:
#
your
AutoScaler
address
address:
192.168.15.139
port:
4567
Cloud Foundry Manifest

28. © 2014 Nippon Telegraph and Telephone Corporation
Optimize
Running Costs with
BOSH AutoScaler