Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more

Scaling Drupal in the Cloud with AWS
Nick Veenhof

Who?
+8
Years
in
Drupal
Fairly
heavily
invested
in
Search
+3
Years
at
Acquia
Tech
Lead
@
MollomBelgium
Boston
Barcelona
Belgium
@Nick_vh

Components
→Let’s get physical! Well.. Sort of.

→I want you all to regroup in all the components that a
website needs. One row = One Component

→I will be a site visitor, so I want you to start from the front
to the end. What is the first layer?

3 Layers
Webserver

EC2
Instance
MySQL

RDS
Domain

Route
53

Is Theory boring?
Describes
the
optimal
environment
and

how
this
relates
to
reality.
A
very

digestible
book

for
designing

distributed
systems.
This
book
exposes

software
patterns
that
every
self-‐
respecting
cloud
infrastructure

engineer

should
know.

http://the-‐cloud-‐book.com/

Cap Principle
Limoncelli,
Thomas
A.;
Chalup,
Strata
R.;

Hogan,
Christina
J.
(2014-‐09-‐01).
The
Practice

of
Cloud
System
Administration:
Designing
and

Operating
Large
Distributed
Systems,
Volume
2

(p.
21).
Pearson
Education.
Kindle
Edition.

The
CAP
Principle
CAP
stands
for

consistency,
availability,
and
partition

resistance.
The
CAP
Principle
states
that

it
is
not
possible
to
build
a
distributed

system
that
guarantees
consistency,

availability,
and
resistance
to

partitioning.
Any
one
or
two
can
be

achieved
but
not
all
three

simultaneously.

Ephemeralism
Webserver

EC2
Instance
MySQL

RDS
Domain

Route
53

Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Horizontal Scaling of Web Servers

Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Database Scaling
Replica
Primary

Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Caching
Replica
Primary
Memcache

ElastiCache

Spof Caching
Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Replica
Primary
Memcache

ElastiCache

Permanent Storage
Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Replica
Primary
Memcache

ElastiCache
Backup
Storage

S3

Version Control
Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Replica
Primary
Memcache

ElastiCache
Backup
Storage

S3
GitHub

Persistent Storage
Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Replica
Primary
Memcache

ElastiCache
Backup
Storage

S3
GitHub

Reverse Proxy
Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Replica
Primary
Memcache

ElastiCache
Backup
Storage

S3GitHub
Varnish

EC2
Instances
http://www.cloudreach.com/gb-‐en/2013/01/varnish-‐autoscaling-‐love-‐story/

Reverse Proxy (v2)
Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Replica
Primary
Memcache

ElastiCache
Backup
Storage

S3GitHub
Varnish

EC2
Instances
Nginx

EC2
Instances

Reverse Proxy (v3)
Webservers

EC2
Instances
MySQL

RDS
Domain

Route
53
Replica
Primary
Memcache

ElastiCache
Backup
Storage

S3
GitHub
Varnish
&
Nginx

Elastic
Beanstalk

Docker
(ECS)
+

Unexpected Spikes
→You could be hosting the next World Cup website

→Or under some page load DDOS from a script kiddie

→A website that is marketed heavily in the next weeks. But
is fairly idle in the rest of the year

→Former slashdot effect (now Reddit)

→…

Stacking up
WebServer
Stack
MySQL

RDS Replica
Primary
Object
Caching

Stack
Backup
Storage

S3
GitHub
+
Database
Stack
Watch
out
for
Aurora.

RDS
is
limited
by
Instance
Size
Load
Balancing
+

Page
Caching
Stack

Scaling Up
WebServer
Stack
MySQL

RDS Replica
Primary
Caching
Stack
Backup
Storage

S3
GitHub
+
Database
Stack
Watch
out
for
Aurora.

RDS
is
limited
by
Instance
Size
Load
Balancing
+

Page
Caching
Stack

Scaling Up
AWS
AutoScaling
Group
Minimum
2

Maximum
10

AutoScaling Policies
→CPUUtilization

→IF CPU(combined) > ’80% for 5+ min’ THEN ‘add
instance’

→Custom CloudWatch Metrics

→Infinite possibilities

Cloudformation Components
{

"ElasticLoadBalancer":
{

"Type":
"AWS::ElasticLoadBalancing::LoadBalancer",

"Properties":
{

"CrossZone":
"true",

"AvailabilityZones":
{

"Fn::GetAZs":
""

},

"LBCookieStickinessPolicy":
[

{

"PolicyName":
"CookieBasedPolicy",

"CookieExpirationPeriod":
"30"

}

],

Cloudformation Sub Stacks

"CacheStack":
{

"Type":
"AWS::CloudFormation::Stack",

"Properties":
{

"TemplateURL":
{

"Fn::Join":
[
"",

[
"https://s3.amazonaws.com/drupaljam.",
{
"Ref":
"AWS::Region"
},".",
{
"Ref":

"EnvironmentName"
},
"/cloudformation/",
"cache.template"
]

]

},

Example
→Make a Scalable Static Hosted Website that can handle
gradual increase of visitors beyond 2 web servers.

→Finance wise we are limited to 5 web servers.

→PHP, MySQL, Load Balancer.

→No Object Caching or Page Caching required

Parameters
→database password

→ssh key

→ssh IP restrictions

→etc…
"Parameters"
:
{

!

"KeyName":
{

"Description"
:
"EC2
KeyPair
to
enable
SSH
access
to
the
instances",

"Default"
:
"drupaljam",

"Type":
"String",

"MinLength":
"1",

"MaxLength":
"255",

"AllowedPattern"
:
"[x20-‐x7E]*",

"ConstraintDescription"
:
"can
contain
only
ASCII
characters."

},

!

"InstanceType"
:
{

"Description"
:
"WebServer
EC2
instance
type",

"Type"
:
"String",

"Default"
:
"m3.medium",

"ConstraintDescription"
:
"must
be
a
valid
EC2
instance
type."

},

!

"SiteName":
{

"Default":
"Drupal",

"Description"
:
"Drupal
Web
Site",

"Type":
"String"

},

AWS::ElasticLoadBalancing::LoadBalancer
→Define Listening Ports

→Define Instance Ports

→Define Cookie
Stickiness Policies

→Defines HealthCheck
so it can take
instances out rotation
"ElasticLoadBalancer"
:
{

"Type"
:
"AWS::ElasticLoadBalancing::LoadBalancer",

"Metadata"
:
{

"Comment"
:
"Configure
the
Load
Balancer
with
a
simple
health
check

},

"Properties"
:
{

"AvailabilityZones"
:
[
"us-‐east-‐1b","us-‐east-‐1d"
],

"LBCookieStickinessPolicy"
:
[
{

"PolicyName"
:
"CookieBasedPolicy",

"CookieExpirationPeriod"
:
"30"

}
],

"Listeners"
:
[
{

"LoadBalancerPort"
:
"80",

"InstancePort"
:
"80",

"Protocol"
:
"HTTP",

"PolicyNames"
:
[
"CookieBasedPolicy"
]

}
],

"HealthCheck"
:
{

"Target"
:
"HTTP:80/",

"HealthyThreshold"
:
"2",

"UnhealthyThreshold"
:
"5",

"Interval"
:
"10",

"Timeout"
:
"5"

}

AWS::AutoScaling::AutoScalingGroup
→Define Minimum &
Maximum

→Define Availability
Zones we can use

→Define Configuration to
execute
"WebServerGroup1"
:
{

"Type"
:
"AWS::AutoScaling::AutoScalingGroup",

"Properties"
:
{

"AvailabilityZones"
:
[
"us-‐east-‐1b","us-‐east-‐1d"
],

"LaunchConfigurationName"
:
{
"Ref"
:
"LaunchConfig1"
},

"MinSize"
:
"1",

"MaxSize"
:
"5",

"DesiredCapacity"
:
{
"Ref"
:
"WebServerCapacity"
},

"LoadBalancerNames"
:
[
{
"Ref"
:
"ElasticLoadBalancer"
}
],

"Tags"
:
[

{

"Key"

:
"Name",

"Value"
:
"Drupaljam
Drupal
Instance",

"PropagateAtLaunch"
:
“true"

}

]

}

AWS::AutoScaling::LaunchConfiguration
→Define Packages (apt/yum)

→Define Sources to extract
to folders (Drupal)

→Define files

→Define commands

→Define services to run

→Execute Script to initialize
"LaunchConfig1":
{

"Type"
:
"AWS::AutoScaling::LaunchConfiguration",

"Metadata"
:
{

"AWS::CloudFormation::Init"
:
{

"config"
:
{

"packages"
:
{

"yum"
:
{

"httpd"
:
[],

"php"
:
[],

"php-‐mysql"
:
[],

"php-‐gd"
:
[],

"php-‐xml"
:
[],

"php-‐mbstring"
:
[],

"mysql"
:
[],

"gcc"
:
[],

"make"
:
[],

"libstdc++-‐devel"
:
[],

"gcc-‐c++"
:
[],

"fuse"
:
[],

"fuse-‐devel"
:
[],

"libcurl-‐devel"
:
[],

"libxml2-‐devel"
:
[],

"openssl-‐devel"
:
[],

"mailcap"
:
[]

!

}

},

!

"sources"
:
{

"/var/www/html"
:
"http://ftp.drupal.org/files/projects/drupal-‐7.36.tar.gz",

"/home/ec2-‐user"
:
"http://ftp.drupal.org/files/projects/drush-‐7.x-‐4.5.tar.gz",

"/home/ec2-‐user/s3fs"
:
"http://s3fs.googlecode.com/files/s3fs-‐1.61.tar.gz"

},

!

"files"
:
{

"/etc/passwd-‐s3fs"
:
{

"content"
:
{
"Fn::Join"
:
["",
[
{
"Ref"
:
"S3Keys"
},
":",
{"Fn::GetAtt":
["S3Keys",
"SecretAccessKey"]},
"n"
]]},

"mode"
:
"000400",

"owner"
:
"root",

"group"
:
"root"

},

!

"/home/ec2-‐user/settings.php"
:
{

"content"
:
{
"Fn::Join"
:
["",
[

"<?phpn",

"n",

"$databases
=
array
(n",

"
'default'
=>n",

"
array
(n",

"
'default'
=>n",

"
array
(n",

"
'database'
=>
'",
{
"Ref"
:
"DBName"
},
"',n",

"
'username'
=>
'",
{
"Ref"
:
"DBUsername"
},
"',n",

"
'password'
=>
'",
{
"Ref"
:
"DBPassword"
},
"',n",

"
'host'
=>
'",
{"Fn::GetAtt"
:
["MasterDB",
"Endpoint.Address"]},
"',n",

"
'port'
=>
'",
{"Fn::GetAtt"
:
["MasterDB",
"Endpoint.Port"]},
"',n",

"
'driver'
=>
'mysql',n",

"
'prefix'
=>
'drupal_',n",

"
),n",

"
),n",

");n",

"n",

"$update_free_access
=
FALSE;n",

"n",

"$drupal_hash_salt
=
'0c3R8noNALe3shsioQr5hK1dMHdwRfikLoSfqn0_xpA';n",

"n",

"ini_set('session.gc_probability',
1);n",

"ini_set('session.gc_divisor',
100);n",

"ini_set('session.gc_maxlifetime',
200000);n",

"ini_set('session.cookie_lifetime',
2000000);n"

]]},

"mode"
:
"000444",

"owner"
:
"root",

"group"
:
"root"

}

},

!

"services"
:
{

"sysvinit"
:
{

"httpd"
:
{
"enabled"
:
"true",
"ensureRunning"
:
"true"
},

"sendmail"
:
{
"enabled"
:
"false",
"ensureRunning"
:
"false"
}

}

}

}

}

},

"Properties":
{

"ImageId"
:
{
"Fn::FindInMap"
:
[
"AWSRegionArch2AMI",
{
"Ref"
:
"AWS::Region"
},

{
"Fn::FindInMap"
:
[
"AWSInstanceType2Arch",
{
"Ref"
:
"InstanceType"
},
"Arch"
]
}
]
},

"InstanceType"
:
{
"Ref"
:
"InstanceType"
},

"SecurityGroups"
:
[
{"Ref"
:
"WebServerSecurityGroup"}
],

"KeyName"
:
{
"Ref"
:
"KeyName"
},

"UserData"
:
{
"Fn::Base64"
:
{
"Fn::Join"
:
["",
[

"#!/bin/bash
-‐vn",

"yum
update
-‐y
aws-‐cfn-‐bootstrapn",

!

"#
Helper
functionn",

"function
error_exitn",

"{n",

"
/opt/aws/bin/cfn-‐signal
-‐e
1
-‐r
"$1"
'",
{
"Ref"
:
"WaitHandle"
},
"'n",

"
exit
1n",

"}n",

!

"#
Install
Apache
Web
Server,
MySQL
and
Drupaln",

"/opt/aws/bin/cfn-‐init
-‐s
",
{
"Ref"
:
"AWS::StackId"
},
"
-‐r
LaunchConfig1
",

"
-‐-‐region
",
{
"Ref"
:
"AWS::Region"
},
"
||
error_exit
'Failed
to
run
cfn-‐init'n",

!

"#
Install
s3fsn",

Let’s
make
this
bigger
shall
we?

"LaunchConfig1":
{

"Type"
:
"AWS::AutoScaling::LaunchConfiguration",

"Metadata"
:
{

"AWS::CloudFormation::Init"
:
{

"config"
:
{

"packages"
:
{

"yum"
:
{

"httpd"
:
[],

"php"
:
[],

"php-‐mysql"
:
[],

"php-‐gd"
:
[],

"php-‐xml"
:
[],

"php-‐mbstring"
:
[],

"mysql"
:
[],

"gcc"
:
[],

"make"
:
[],

"libstdc++-‐devel"
:
[],

"sources"
:
{

"/var/www/html"
:
"http://ftp.drupal.org/files/projects/drupal-‐7.36.tar.gz",

"/home/ec2-‐user"
:
"http://ftp.drupal.org/files/projects/drush-‐7.x-‐4.5.tar.gz",

"/home/ec2-‐user/s3fs"
:
"http://s3fs.googlecode.com/files/s3fs-‐1.61.tar.gz"

},

"files"
:
{

"/etc/passwd-‐s3fs"
:
{

"content"
:
{
"Fn::Join"
:
["",
[
{
"Ref"
:
"S3Keys"
},
":",
{"Fn::GetAtt":
["S3Keys",
"SecretAccessKey"]},
"n"
]]},

"mode"
:
"000400",

"owner"
:
"root",

"group"
:
"root"

},

…

},

"Properties":
{

"ImageId"
:
{
"Fn::FindInMap"
:
[
"AWSRegionArch2AMI",
{
"Ref"
:
"AWS::Region"
},

{
"Fn::FindInMap"
:
[
"AWSInstanceType2Arch",
{
"Ref"
:
"InstanceType"
},

"Arch"
]
}
]
},

"InstanceType"
:
{
"Ref"
:
"InstanceType"
},

"SecurityGroups"
:
[
{"Ref"
:
"WebServerSecurityGroup"}
],

"KeyName"
:
{
"Ref"
:
"KeyName"
},

"UserData"
:
{
"Fn::Base64"
:
{
"Fn::Join"
:
["",
[

"#!/bin/bash
-‐vn",

"yum
update
-‐y
aws-‐cfn-‐bootstrapn”,

….

AWS::RDS::DBInstance
→Define size of MySQL
Instance

→Define MySQL Version

→Define MultiAZ or not
"MasterDB"
:
{

"Type"
:
"AWS::RDS::DBInstance",

"Properties"
:
{

"DBName"
:
{
"Ref"
:
"DBName"
},

"AllocatedStorage"
:
{
"Ref"
:
"DBAllocatedStorage"
},

"DBInstanceClass"
:
{
"Ref"
:
"DBClass"
},

"Engine"
:
"MySQL",

"EngineVersion"
:
"5.6",

"DBInstanceIdentifier"
:
"DrupalJamMasterDB",

"DBSecurityGroups":
[
{
"Ref":
"DBSecurityGroup"
}
],

"MasterUsername"
:
{
"Ref"
:
"DBUsername"
},

"MasterUserPassword"
:
{
"Ref"
:
"DBPassword"
},

"MultiAZ"
:
{
"Ref"
:
"MultiAZDatabase"
},

"Tags"
:
[{
"Key"

:
"Name",
"Value"
:
"Drupaljam
Drupal
Master
Database"
}]

},

"DeletionPolicy"
:
"Snapshot"

},

AWS::EC2::SecurityGroup
→Define Security Levels
between AWS Services

→Eg. Only allow traffic
between Load Balancer
and Instances on port 80

→Eg. Allow port 22 for the
IP range in the inputs
"WebServerSecurityGroup"
:
{

"Type"
:
"AWS::EC2::SecurityGroup",

"Properties"
:
{

"GroupDescription"
:
"Enable
HTTP
access
via
port
80,
locked
down
to
requests

from
the
load
balancer
only
and
SSH
access",

"SecurityGroupIngress"
:
[

{"IpProtocol"
:
"tcp",
"FromPort"
:
"80",
"ToPort"
:
"80",

"SourceSecurityGroupOwnerId"
:
{"Fn::GetAtt"
:
["ElasticLoadBalancer",

"SourceSecurityGroup.OwnerAlias"]},"SourceSecurityGroupName"
:
{"Fn::GetAtt"
:

["ElasticLoadBalancer",
"SourceSecurityGroup.GroupName"]}},

{"IpProtocol"
:
"tcp",
"FromPort"
:
"22",
"ToPort"
:
"22",
"CidrIp"
:
{
"Ref"
:

"SSHLocation"}}

]

}

}

And more…
→Make sure to start with VPC

→Be Region Agnostic as some are VPC Only

→Internal ELB, Internal IP’s

→Private Puppet/Chef Servers

→Define Security Model first

→Do not create tribal knowledge

Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more

Similar to Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more (20)

More from Dropsolid

More from Dropsolid (7)

Recently uploaded

Recently uploaded (20)

Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more