Today, there are several trends that are forcing application architectures to evolve. Users expect a rich, interactive and dynamic user experience on a wide variety of clients including mobile devices. Applications must be highly scalable, highly available and run on cloud environments. Organizations often want to frequently roll out updates, even multiple times a day. Consequently, it’s no longer adequate to develop simple, monolithic web applications that serve up HTML to desktop browsers. In this talk we describe the limitations of a monolithic architecture. You will learn how to use the scale cube to decompose your application into a set of narrowly focused, independently deployable back-end services and an HTML 5 client. We will also discuss the role of technologies such as NodeJS and AMQP brokers. You will learn how a modern PaaS such as Cloud Foundry simplifies the development and deployment of this style of application.
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Decomposing Applications for Scalability and Deployability (April 2012)
1. Decomposing applications for
deployability and scalability
Chris Richardson
Author of POJOs in Action
Founder of the original CloudFoundry.com
@crichardson
crichardson@vmware.com
1
11. Traditional web application architecture
WAR
StoreFrontUI
Accounting
Service
MySQL
Browser Apache Database
InventoryService
Shipping
Service
Simple to Tomcat
develop
test
deploy
scale
11
13. Users expect a rich, dynamic and
interactive experience on mobile
devices and desktop
h
oug
d en
oo
’t g
HTTP Request
e isn
r
ctu
Java Web
Browser
e
hit
HTML/Javascript Application
c
I ar
ty le U
s
Old
Real-time web ≅ NodeJS
13
14. One dimensional scalability
§ Scales to handle transaction volumes
§ Higher transaction rate required run more instances
§ Some clouds will do this automatically, others easily
§ No code changes required
BUT
§ Does not scale with increasing data volumes
§ Caching is less effective
§ Requires more memory
§ Requires more I/O
14
15. Components that don’t cluster
§ Most application components are easily clustered:
• Most services are stateless
• Server-side session state: sticky session routing/state
server
BUT there are exceptions
§ Singletons that maintain global application state:
• e.g. CEP engine analyzing event stream over HTTP
§ Commercial software with a per-server license fee
§ If one application component can’t be clustered you can’t
cluster the application!
15
16. Obstacle to frequent deployments
§ Need to redeploy everything to change one component
§ Interrupts long running background (e.g. Quartz) jobs
§ Increases risk of failure
Fear of change
§ Updates will happen less often
§ e.g. Makes A/B testing UI really difficult 16
18. Scaling development
WAR
StoreFrontUI
Accounting
Scalable
!=
InventoryService development
Shipping
§ Forces teams to synchronize development efforts
§ Teams need to coordinate updates
18
19. Long-term commitment to a single technology stack
§ Let’s suppose you go with the JVM
• Some polyglot support
• Many (but not all) JVM languages interoperate easily
• e.g. Scala, Groovy modules within a Java application
• But you can’t use non-JVM languages
§ Application depends on a particular:
• Framework - e.g. Spring or Java EE
• Container - e.g. Tomcat
§ Switching technology stack touches entire application
• Painful
• Rarely done
19
20. Agenda
§ The (sometimes evil) monolith
§ Decomposing applications into services
§ How do services communicate?
§ Presentation layer design
§ Decomposing a monolithic application
§ How Cloud Foundry helps
20
21. 3 dimensions to scaling
Y axis -
functional
decomposition
Scale by
s
ng
splitting
ila g
n
hi
g s oni
rt
different things
iti
rt
im
pa
ta
in
da
ittl
s-
sp
i
ax
by
Z
ale
X axis - horizontal duplication
Sc
Scale by cloning
21
22. X axis scaling - application level
Load WAR
Balancer
Tomcat
22
23. X axis scaling - database level
Writes Consistent reads
MySQL
Master
Inconsistent reads
MySQL MySQL MySQL
Slave 1 Slave 2 Slave N
Slaves are clones
23
24. Benefits and drawbacks of X axis splits
§ Benefits
• Easy and fast to implement
• Handles increasing transaction volumes
• Simple to manage
§ Drawbacks
• Doesn’t scale with increasing data volumes
• Doesn’t address application/development complexity
• Couples entire application to a particular technology stack
24
25. Z axis scaling - data partitioning
§ Each server runs the identical code base
§ Different servers responsible for different data
§ Route request based on
• Attribute of the request
• Identity/attribute of the requestor/customer
§ For example:
• Sharding - partition data by primary key
• Free customer vs. paying customer
Server 1
Request:
... Some attribute ...
“Router”
Server 2
...
25
26. Z axis scaling - database sharding
Partition column
RESTAURANT table
ID Name …
1 Ajanta
2 Eggshop
…
X axis scaling too!
Database server 1 Database server 2
ID Name … ID Name …
1 Ajanta 2 Eggshop
… …
26
27. Z axis scaling - application example
Content Query
Update Router Query Aggregator
Search Service Search Service
Search Service Search Service
Cache Cache
Database server 1 Database server 2
Partition 1 Partition 2 27
28. Benefits and drawbacks of Z axis splits
§ Benefits
• Each server only deals with a subset of the data
• Improved caching
• Reduced memory usage and reduced I/O
• Improved transaction scalability
• Improved fault isolation - failure only makes part of the data
inaccessible
§ Drawbacks
• Doesn’t scale with increasing development and application
complexity
• Increased management/deployment complexity
28
29. Y axis scaling - functional partitioning
§ Splits monolithic application into a set of services
§ Each service implements related set of functionality
§ Partitioning schemes:
• Partition functionality by noun or by verb
• Single Responsibility Principle
• e.g. Unix utilities - do one focussed thing well
29
30. Y axis scaling - application level
billing web application
Accounting
Service
Store front web application inventory web application
Store Front InventoryService
shipping web application
ShippingService
Apply X axis cloning and/or Z axis partitioning to each service
30
31. Y axis scaling - databases
Application
MySQL MySQL
MySQL MySQL
Order Customer
table table
.... ....
Apply X axis cloning and/or Z axis partitioning to each database
31
32. Real world examples
http://techblog.netflix.com/
Between 100-150 services are accessed to
build a page.
http://highscalability.com/amazon-architecture
http://www.addsimplicity.com/downloads/
eBaySDForum2006-11-29.pdf
http://queue.acm.org/detail.cfm?id=1394128
32
33. Benefits and drawbacks of Y axis splits
§ Benefits
• Scales development: focussed two pizza devops teams
• Deploy services independently
• Scale services independently
• Improves fault isolation
• Eliminates long-term commitment to a single technology
stack
• Enforces well defined interfaces between components
§ Drawbacks
• Implementation complexity
• Deployment complexity
33
34. Two levels of architecture
§ System level
• Defines the inter-service glue: interfaces and
communication mechanisms
• Slow changing
Versus
§ Service level
• Defines the internal architecture of each service
• Far fewer constraints on technology
• Each service could use a different technology stack
• Rapid evolving
34
35. If services are small...
§ Regularly rewrite using a better technology stack
§ Pick the best developers rather than best <pick a
language> developers
§ Adapt system to changing requirements and better
technology without a total rewrite
Fred George
“Developer Anarchy”
35
36. Moreover: you are not the same you ...
§ Cell lifetimes: Can we build software
• hours - some white blood cells systems with these
characteristics?
• days - stomach lining cells
• years - bone cells Too much technical debt
• lifetime - brain cells component death?
§ 50 to 70 billion of your cells die each day
§ Yet you (the system) remains intact
http://dreamsongs.com/Files/
DesignBeyondHumanAbilitiesSimp.pdf
http://dreamsongs.com/Files/WhitherSoftware.pdf
36
37. Agenda
§ The (sometimes evil) monolith
§ Decomposing applications into services
§ How do services communicate?
§ Presentation layer design
§ Decomposing a monolithic application
§ How Cloud Foundry helps
37
38. Inter-service communication options
§ Multiple collaborating services need a communication
mechanism
§ Many choices:
• Synchronous asynchronous
• Transports: HTTP, AMQP, ...
• Formats: JSON, XML, Protocol Buffers, Thrift, ...
• Even via the database
§ Distributed application error handling strategies
38
39. Synchronous communication
wgrus-billing.war
Accounting
Service
REST
wgrus-store.war
wgrus-shipping.war
SOAP
StoreFrontUI Shipping MySQL
Service
Thrift
... wgrus-inventory.war
InventoryService
Lots of choices
39
40. Service discovery options
§ Clients need to know the coordinates of their dependencies
§ Option #1: Caller is configured with dependencies’s URL
• e.g. Environment variables, system properties
§ Option #2: Services announce their location
• Relies on ‘broadcast’ protocol or Gossip protocol to
announce location
• e.g. RabbitMQ message broker
§ Option #3: configuration server
• Maintains all the configuration information
• e.g. Zookeeper
40
41. Benefits and drawbacks
§ Benefits
• Simple to implement and use
§ Drawbacks
• Poor fault isolation: caller is blocked if server is down/slow
• Caller needs to know server’s coordinates (URL, ...)
• Less scalable than asynchronous protocols
41
42. Asynchronous message-based communication
wgrus-billing.war
Accounting
Service
wgrus-store.war
RabbitMQ wgrus-inventory.war
StoreFrontUI (Message
MySQL
Broker) Widget
InventoryService
wgrus-inventory.war
InventoryService
42
43. Benefits and drawbacks
§ Benefits
• Decouples caller from server
• Caller unaware of server’s coordinates (URL)
• Message broker buffers message when server is down/slow
§ Drawbacks
• Additional complexity of message broker
• RPC using messaging is more complex
43
45. Benefits and drawbacks
§ Benefits
• Simple
• Single (consistent) source of data
§ Drawbacks
• Defeats the purpose of having multiple services
• Less scalable
• More coupling
45
46. Better: separate databases
Customer Update Event
Order Management Customer Management
Database
Database
Orders
Customers
Customers (copy)
46
47. Maintaining consistency without 2PC
begin transaction
update order tables
save intent to update customer tables
commit transaction
for each saved intent
begin transaction
delete intent
queue message to update customer Customer management
commit transaction dequeue message
Order management begin transaction
if (update has not been applied) {
record update as applied
update customer tables
BASE: An Acid Alternative by Dan Pritchett
}
http://queue.acm.org/detail.cfm?id=1394128 commit transaction
acknowledge message 47
49. Composable futures
§ Problem:
• Service A needs to call services B and C and then D
• Makes sense to call B and C parallel
• Yet most concurrency APIs are low-level, error-prone etc
§ Solution:
• Use Akka composable futures = really nice abstraction
val futureB = callB() Two calls execute in parallel
val futureC = callC()
val futureD = for {
b <- futureB.mapTo[SomeType]
c <- futureC.mapTo[SomeType]
d <- callD(b, c) And then invokes D
} yield d
val result = Await.result(futureD, 1 second).
asInstanceOf[SomeOtherType]
Get the result of D
http://doc.akka.io/docs/akka/2.0.1/scala/futures.html
http://en.wikipedia.org/wiki/Futures_and_promises
49
50. Spring Integration
§ Builds on Spring framework
§ High-level of abstraction for building message
based applications
§ Implements EAI patterns
§ Provides plumbing for exchanging messages
between application components
§ Promotes loosely coupled components
§ Integrates with external messaging infrastructure:
JMS, AMQP, HTTP, Email, File transfer
50
51. Spring Integration concepts
§ Message channel
• Virtual pipe connecting producer and consumer
§ Message endpoints
• The filter of a pipes-and-filter architecture
• Read from and/or write to channel
§ Endpoint types:
• Transformer
• Filter
• Router
• Splitter
• Aggregator
• ServiceActivator
• Inbound channel adapter - read from external source, writes to channel
• Outbound channel adapter - read from channel write to external destination
51
53. Example of reconfigurability - local
@Service
public class OrderServiceImpl { @Service
public class ShippingServiceImpl {
@Autowired
private ShippingService shippingService; public void shipOrder(String orderId) {
System.out.println("shipped order: " + orderId);
public void placeOrder() { }
String orderId = generateOrderId();
… }
shippingService.shipOrder(orderId);
}
}
Order Shipping
Service service
Messaging
Gateway
Channel Service
Activator
53
54. Example of reconfigurability - distributed
Code unchanged in new deployment
Order Shipping
Service service
Messaging
Gateway RabbitMQ
Channel AMQP AMQP Channel Service
Activator
54
55. Handling failure
§ Errors happen (especially in distributed systems)
§ Use timeouts and retries
• Never wait forever
• Some errors are transient so retry
§ Use per-dependency bounded thread pool with bounded queue
• Limits number of outstanding requests
• Fails fast if service is slow or down
§ Use circuit breaker
• High rate of errors stop calling temporarily
• Avoids calling service that has issues
§ On failure
• Returned cached or default data
• Invoke custom error handler
http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
55
56. Agenda
§ The (sometimes evil) monolith
§ Decomposing applications into services
§ How do services communicate?
§ Presentation layer design
§ Decomposing a monolithic application
§ How Cloud Foundry helps
56
58. NodeJS is the fashionable technology
Many JavaScript client frameworks have a NodeJS counterpart
e.g. socket.io
58
59. NodeJS isn’t the only game in town
JVM-based http://vertx.io/
59
60. A modern web application
Service 1
RESTful WS Node JS
Browser
HTML 5 Server Application
Application Service 2
Events
Socket.io Socket.io
client server
...
60
61. Alternatively: multiple front-end servers
Node JS Service 1
Browser
HTML 5 Node JS Service 2
Application
Node JS ...
Needs single-sign-on
61
62. NodeJS - using RESTful WS and AMQP
REST
Service
Requests
Node JS
Events
socket.io
AMQP AMQP
RabbitMQ Service
62
63. Updating the UI is easy
§ Update the UI independently of rest of system
§ Easily run A/B tests
§ Enables fast iteration of the UI
http://theleanstartup.com/principles
63
64. But coordination with backend changes required
§ Let’s imagine that you are deploying an advanced search
feature:
• Enhancements to search service
• Enhancements to UI
§ Before
• Deploy new war
§ Now:
• Some coordination required
• Deploy updated backend service
• Deploy updated NodeJS and browser code
• Enable feature using feature switch
64
65. Agenda
§ The (sometimes evil) monolith
§ Decomposing applications into services
§ How do services communicate?
§ Presentation layer design
§ Decomposing a monolithic application
§ How Cloud Foundry helps
65
66. The challenge
§ In the beginning:
• It would be easy to use Y axis/functional decomposition
• But you don’t need it and it would slow you down
§ Later on:
• You do need it
• But your application logic is tangled and decomposition is
difficult
66
67. Lower cost way to start with Y axis decomposition
§ Enforce boundaries between services, e.g. DTOs
§ But run in a single JVM with a single database
§ Use Spring Integration
• Course-grained message-based communication
• Via in-memory channels
§ Downside:
• Implementing the glue layers are extra work
67
68. Special case: Components that don’t cluster
Application
Monitoring Rest of domain classes
A
C
B
D
Esper
Tangled dependencies prevent Y axis scaling
Doesn’t cluster No X axis scaling/cloning 68
69. Don’t let this happen!
Use well-defined interfaces
Ensure loose coupling
69
71. Extracting a service = untangling dependencies
Application Service
Service C
Service A Service B
Trouble!
Domain model 1 Domain model 2
X A C
Y B
Z
71
72. Untangling dependencies
§ Domain object is part of an association class A {
• Read only - replicate private B other;
• Update only - notification interface }
• Update then read - replicate
§ Domain object is a request parameter class A {
• Read-only - replace with DTO void m(B other) {
...
• Update only - notification interface }
• Update then read - tricky }
§ Note - Need acyclic dependencies
• If used object isn’t part of a service = assumption is that it
belongs in calling service
72
73. Bounded context is a useful idea
§ Different parts of a system different domain models
§ Different services have a different view of some domain
objects, e.g.
• User Management = complex view of user
• Rest of application = PK + ACL + Name
§ Different services can have a different domain model
§ Services exchange messages to synchronize data
73
74. Untangling dependencies 1
Service A Service B Service C
Domain model 1 Domain model 2
X A C
doSomething(Y)
doSomethingElse()
Y B
getSomething()
Z
74
75. Untangling dependencies 2
Service A Service B Service C
Domain model 1 Domain model 2
doSomething(YDTO) C
X A
Facade A
doSomethingElse()
Y B
Facade B
Z
75
76. Untangling dependencies 3
Service A Service B Service C
Domain model 1 Domain model 2
C
doSomething(YDTO)
X Facade A A
Message
Broker Listener
Y
Listener B
doSomethingElse()
Z Facade B
76
77. Agenda
§ The (sometimes evil) monolith
§ Decomposing applications into services
§ How do services communicate?
§ Presentation layer design
§ Decomposing a monolithic application
§ How Cloud Foundry helps
77
79. Developing service-oriented apps is harder
§ Many more moving parts to manage
• Infrastructure services: SQL, NoSQL, RabbitMQ
• Application services
§ Who is going to setup the environments:
• the developer sandbox?
• ...
• QA environments?
79
80. Cloud Foundry makes it easier
§ Easy to deploy applications
§ Easy to provision services
§ Manifest describes the structure of a set of services
§ vmc env-add for binding 'app services' together
§ Micro Cloud Foundry is your personal cloud/sandbox
§ Caldecott exposes services for use by integration tests
80
82. Creating a service instance
$ vmc create-service mysql --name mysql1
Creating Service: OK
$ vmc services
......
=========== Provisioned Services ============
+-------------+---------+
| Name | Service |
+-------------+---------+
| mysql1 | mysql |
+-------------+---------+
84. Multi-application manifest - part 2
store/target: Path to application
name: store
url: cer-store.chrisr.cloudfoundry.me
framework:
name: spring
info:
mem: 512M
description: Java SpringSource Spring Application
exec:
mem: 512M
instances: 1
services: Required services
si-mongo:
type: :mongodb
si-rabbit:
type: :rabbitmq
84
85. One command to create services and deploy application
$ vmc push
Would you like to deploy from the current directory? [Yn]:
Pushing application 'inventory'...
Creating Application: OK
Creating Service [si-rabbit]: OK
Binding Service [si-rabbit]: OK
Creating Service [si-mongo]: OK
Binding Service [si-mongo]: OK
Creating Service [si-redis]: OK
Binding Service [si-redis]: OK
Uploading Application:
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (12K): OK
vmc push:
Push Status: OK •Reads the manifest file
•Creates the required services
Staging Application 'inventory': OK
Starting Application 'inventory': OK
Pushing application 'store'...
Creating Application: OK •Deploys all the applications
Binding Service [si-mongo]: OK
Binding Service [si-rabbit]: OK
Uploading Application:
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (5K): OK
Push Status: OK
Staging Application 'store': OK
85
Starting Application 'store': ...
86. Application configuration via environment variables
$ vmc env-add cf1
PAYMENT_SVC=http://...
Adding Environment Variable
[PAYMENT_SVC=http://...]: OK
Stopping Application: OK
Staging Application: OK
String value = System.getenv("PAYMENT_SVC")
@Value("#{systemEnvironment['PAYMENT_SVC']}")
private String envVariable;
87. Micro Cloud Foundry: new developer sandbox
App Instances Services
Open source Platform as a Service project
10.04
A PaaS packaged as a VMware Virtual Machine
Use as a developer sandbox
• Use the services from Junit integration tests
• Deploy your application for functional testing
• Remote debugging from STS
87
88. Using Caldecott…
$ vmc tunnel
1: mysql-135e0
2: mysql1
Which service to tunnel to?: 2
Password: ********
Stopping Application: OK
Redeploying tunnel application 'caldecott'.
Uploading Application:
Checking for available resources: OK
Packing application: OK
Uploading (1K): OK
Push Status: OK
Binding Service [mysql1]: OK
Staging Application: OK
Starting Application: OK
Getting tunnel connection info: OK
Service connection info:
username : uMe6Apgw00AhS
password : pKcD76PcZR7GZ
name : d7cb8afb52f084f3d9bdc269e7d99ab50
Starting tunnel to mysql1 on port 10000.
1: none
2: mysql
Which client would you like to start?: 2
89. …Using Caldecott
Launching 'mysql --protocol=TCP --host=localhost --port=10000 --
user=uMe6Apgw00AhS --password=pKcD76PcZR7GZ
d7cb8afb52f084f3d9bdc269e7d99ab50'
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 10944342
Server version: 5.1.54-rel12.5 Percona Server with XtraDB (GPL),
Release 12.5, Revision 188
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights
reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or 'h' for help. Type 'c' to clear the current input
statement.
mysql>
90. Running JUnit test with Caldecott
Configure your test code to use port + connection info
90
91. Summary
§ Monolithic applications are simple to develop and deploy
BUT applying the scale cube
§ Decomposes your application into services
§ Enables scaling for transactions and data volumes
§ Tackles application complexity
§ Enables scaling for development
§ Enables frequent, independent deployments
§ Make it easy to leverage other technologies
AND
§ Cloud Foundry simplifies the development and deployment of
“service-oriented” applications
91