1.
PHP at 5000 Requests / Sec
Hootsuite’s Scaling Story
Bill Monkman
Lead Technical Engineer - Platform
@bmonkman

2.
Overview - Selected Current Architecture
Users lb1 lb2 lb3 ... Nginx Load balancers
web1 web2 web3 ... Nginx web servers
PHP-FPM PHP-FPM PHP-FPM PHP-FPM
Memcached cluster
mem1 ...
Mysql cluster
master slave
MongoDB cluster
master slave
master slave
shard1
shard2
Gearman cluster
geard1 geard2
worker1 ... ...
...
Services

3.
Technologies - at first
• Apache
• PHP
• MySQL

4.
Then...

5.
Problem
It’s hard to scale MySQL horizontally

6.
Solution - Caching
Memcached.
● Distributed cache, cluster of boxes with lots of RAM, trivial to scale
● Cache as much as possible, invalidate only when necessary
● Use cache instead of DB
● No joins - decouple entities (collection caching)
● Twemproxy!

7.
“There are only two hard things in
Computer Science: cache invalidation and
naming things.”
• Phil Karlton

8.
Solution - Caching
MvcModelBaseCaching
MvcModelBase
MvcModelMysql
SocialNetwork

9.
Solution - Caching
SELECT * FROM member WHERE org_id=888
set individual cache records
member_1 {data}
member_5 {data}
member_9 {data}
set collection cache
member_org_888 [1,5,9]
Automatic invalidation of collection cache

10.
Solution - Caching
It’s hard to scale MySQL horizontally
Now:
● No need to scale MySQL
● Able to serve the whole site on 1 MySQL server
● 500 MySQL SELECTs per second. 50,000 Memcached GETs.
● 99+% hit rate

11.
Then...

12.
Problem
Need a way to perform asynchronous, distributed tasks using a
single-threaded language.

13.
Solution - Gearman
Gearman.
● Distribute work to other servers to handle (workers also using
PHP, same codebase)
● Precursor to SOA where everything is truly distributed
● Many other solutions, queueing systems.

14.
Solution - Gearman
geard1 geard2
gearworker1 gearworker2 gearworker6

15.
Solution - Gearman
Need a way to perform asynchronous, distributed tasks using a
single-threaded language.
Now:
● Moved key tasks to Gearman
● Another cluster, scalable separately from web
● Discrete tasks, callable sync or async

16.
Then...

17.
Problem
Need to store data with the potential to grow too big to handle
effectively with MySQL.

18.
Solution - MongoDB
MongoDB.
● Certain data did not need to be highly relational
● NoSQL DB, many other solutions these days
● Mongo can be a pain, lots of moving parts
● Had to make our own sequencer where auto-incremented ids were
necessary

19.
Solution - MongoDB
Need to store data with the potential to grow too big to handle
effectively with MySQL.
Now:
● Multiple clusters containing amounts of data that likely would
have crushed MySQL
● Billions of rows per collection, many TB of data on disk

20.
Technologies
• Apache
• PHP
• MySQL
• Memcached
• Gearman
• MongoDB

21.
Then...

22.
Problem
With a codebase and an engineering team increasing in size, how do
we keep up the pace of development and maintain control of the
system?
(SVN, big branches, merge hell)

23.
Solution - Dark Launching
Dark Launching.
● Wrap code in block with a specific name
● That name will appear in a management page
● Can control whether or not that block is executed by modifying it’s value
● Boolean , random percentage, session-based, member list, organization
list, etc.

24.
Solution - Dark Launching
if (In_Feature::isEnabled(‘TWITTER_ADS’)) {
// execute new code
} else {
// execute old code
}

25.
Dark Launching - Reasons
• Control your code
• Limit risk -> raise confidence -> speed up pace of releases
• “Branching in Production”
• Learning happens in Production

26.
Solution - Dark Launching
With a codebase and an engineering team increasing in size, how do
we keep up the pace of development and maintain control of the
system?
Now:
● Work fast with more confidence
● Huge amount of control over production systems
● Typically 10+ code releases to production per day
● Push-based distribution with Consul

27.
Then...

28.
Problem
With a rapidly increasing codebase and amount of users / traffic
how do we keep visibility into the performance of the code?

29.
Solution - Monitoring
Statsd / Graphite.
Logstash / Elasticsearch / Kibana.
Sensu
● Statsd for metrics
● Logstash for log events
● Sensu for monitoring / alerting

30.
Solution - Monitoring
Statsd::timing('apiCall.facebookGraph', microtime(true) - $startTime);

31.
Solution - Monitoring
Logger::event('user liked from in-stream', In_Log::CATEGORY_UX, $logData);

32.
Solution - Monitoring
• Visibility into the performance and behaviour of your application
• Iterate upon your code, measure results
• Pairs well with dark launching
• Also systems like New Relic

33.
Solution - Monitoring
With a rapidly increasing codebase and amount of users / traffic
how do we keep visibility into the performance of the code?
Now:
● Able to watch performance / behaviour in real time.
● Able to view important events both in the aggregate or very
granular
● Able to control the system and watch the effect of changes

34.
Optimizations

35.
Optimizations
• Things expand beyond their initial scope
• Case in point: Translations

36.
Optimizations - Push work to users
• Within reason, push work up to users
• Make your users into a distributed processing grid
• e.g. Stream rendering

37.
Optimizations - Performance / Risks
• Performance is more important than clean code, business reqts
(in the instances where they may be mutually exclusive)
• Fine line between future proofing and premature optimization
• Don’t add burdensome processes, but make it easy for your team
to do things the right way
• Know your weak spots, protect against abuse

38.
Technologies
Linux
Nginx
ElasticSearch Varnish
PHP-FPM
MySQL
Jenkins
Scala
MongoDB
Consul
Gearman
Redis
Akka
Python
Memcached
HAProxy
jQuery
ZeroMQ
Backbone RabbitMQ
EC2
Zend
Docker
Cloudfront CDN
Logstash
Zookeeper
Kibana
Statsd/Graphite
Packer
Vagrant
Nagios
VirtualBox
Spark/Shark
Sensu
Symfony
Riak
Composer
Websockets
Comet
Hadoop
Ansible
Git
Webpack Redshift

39.
Problem
With a huge and growing monolithic codebase and over 80
engineers, how to keep scaling in a manageable way?

40.
Solution - SOA
SOA.
● Split up the system into independent services which communicate only via APIs
● Teams can work on their own services with encapsulated business logic and have their own
deployment schedules.
● We chose to use Scala/Akka for services, communicating via ZeroMQ
● SOA transition made easier by the “no joins” philosophy
● Tons of work

41.
Solution - SOA
SOM.
● “Service Oriented Monolith”
● When splitting up a monolithic codebase, dependencies are what kill you
● Fulfill dependencies by writing interim services using existing PHP code
● Maintain the contract and future scala services will be drop-in
replacements

42.
Solution - SOA
With a huge and growing monolithic codebase and over 130
engineers, how to keep scaling in a manageable way?
Today:
● Transitioning to Scala SOA
● PHP will still be used as the Façade, a thin layer built on top of
the business logic of the services it interacts with.

43.
Conclusion

44.
Thank You!
Bill Monkman
@bmonkman
More Info:
code.hootsuite.com