Distribute the workload, PHP Barcelona 2011
•
8 likes•2,678 views
Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow. By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our service by using systems such as: - gearman - memcache - daemons - message queues - load balancers and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily. During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (9)
Similar to Distribute the workload, PHP Barcelona 2011
Similar to Distribute the workload, PHP Barcelona 2011 (12)
More from Helgi Þormar Þorbjörnsson
Recently uploaded
Recently uploaded (20)
Distribute the workload, PHP Barcelona 2011
- 1. Distribute the workload Helgi Þormar Þorbjörnsson PHP Barcelona, 29th of October 2011 Saturday, 29 October 11
- 2. Who am I? Saturday, 29 October 11
- 3. Helgi Co-founded Orchestra.io Developer at PEAR From Iceland @h on Twitter Saturday, 29 October 11
- 4. Why Distribute? Efficiency Budget Perception Saturday, 29 October 11
- 5. Efficiency 10 small servers > 1 big Saturday, 29 October 11
- 6. Budget Spend wisely Commodity servers Cloud Computing (EC2) Saturday, 29 October 11
- 7. Perception Defer intensive processes Give instant feedback Users keep on browsing Saturday, 29 October 11
- 8. Saturday, 29 October 11
- 9. Ant Colonies Saturday, 29 October 11
- 10. Teamwork When faced with a problem they will solve the problem as one. Saturday, 29 October 11
- 11. Saturday, 29 October 11
- 12. Saturday, 29 October 11
- 13. Architect for Distribution Saturday, 29 October 11
- 14. Characteristics Decoupling Elasticity High Availability Concurrency Saturday, 29 October 11
- 16. Application DB API Cache FE Saturday, 29 October 11
- 17. API Application API DB API Cache FE Cache Saturday, 29 October 11
- 19. Cloud Computing Saturday, 29 October 11
- 20. Load Balancing Saturday, 29 October 11
- 21. My Favourite HA Proxy Nginx Saturday, 29 October 11
- 23. When do I need more servers? Saturday, 29 October 11
- 24. Needs to be around from the start! Saturday, 29 October 11
- 25. Keep records Saturday, 29 October 11
- 26. Spot trends Saturday, 29 October 11
- 27. Different types Hardware Performance Software Performance Availability Resourcing Saturday, 29 October 11
- 28. Applications New Relic CloudKick ScoutApp Nagios Cacti Circonus Saturday, 29 October 11
- 30. Plug into your monitoring Saturday, 29 October 11
- 31. Bringing together Monitoring and Elastic behaviour into one beautiful whole! Saturday, 29 October 11
- 32. Add some intelligence to add / remove servers as needed based on current information. Saturday, 29 October 11
- 33. Just make sure it doesn’t turn into... Saturday, 29 October 11
- 35. High Availability Saturday, 29 October 11
- 36. Get a highly available and resilient setup by following a few of those recommendations Saturday, 29 October 11
- 37. Remember, even Google has outages Saturday, 29 October 11
- 38. What to avoid Saturday, 29 October 11
- 39. Local Sessions Saturday, 29 October 11
- 40. Solution Store sessions in DB / Memcache Saturday, 29 October 11
- 41. Local Memory Saturday, 29 October 11
- 42. Solution Networked Memcache Saturday, 29 October 11
- 43. Local Files Saturday, 29 October 11
- 44. Local Uploads Saturday, 29 October 11
- 45. Writing to /tmp Saturday, 29 October 11
- 46. Solution Store on S3 or a networked FS Saturday, 29 October 11
- 47. Solution Serve up static files from CDNs Saturday, 29 October 11
- 48. Servers can vanish at any given time Saturday, 29 October 11
- 49. Internal APIs Saturday, 29 October 11
- 50. Application Internal Storage API GFS S3 FS Saturday, 29 October 11
- 51. Application Internal DB API Mongo MySQL Cache Saturday, 29 October 11
- 52. SOA Saturday, 29 October 11
- 53. Service Oriented Architecture Saturday, 29 October 11
- 54. Sort of :-) Saturday, 29 October 11
- 55. Eventually Consistent Saturday, 29 October 11
- 56. CAP Therom Saturday, 29 October 11
- 57. Consistency Availability Partition Tolerance Saturday, 29 October 11
- 58. Consistency All nodes see the same data at the same time Saturday, 29 October 11
- 59. Availability Node failures do not prevent survivors from continuing to operate Saturday, 29 October 11
- 60. Partition Tolerance The system continues to operate despite arbitrary message loss Saturday, 29 October 11
- 61. Consistency Availability Partition Tolerance Saturday, 29 October 11
- 62. Queue Systems Saturday, 29 October 11
- 63. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation Search Saturday, 29 October 11
- 64. Common Tools Gearman Hadoop ZeroMQ RabbitMQ And many others! Saturday, 29 October 11
- 65. New York Times 4TB of TIFF files Needed to get 11 million PDF versions Used Hadoop and EC2 100 machines took 24 hours Saturday, 29 October 11
- 67. Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same steps Answer passed back to Master Saturday, 29 October 11
- 68. Reduce Takes in answers from the map workers Combines together to get an answer There can be multiple reducers Saturday, 29 October 11
- 69. process petabytes of data in few hours on commodity server farm Saturday, 29 October 11
- 70. CouchDB Saturday, 29 October 11
- 71. CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce Views Easy Replication Saturday, 29 October 11
- 72. Gearman Saturday, 29 October 11
- 73. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Your App Gearman Job Server Gearman gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker Code Saturday, 29 October 11
- 74. pear.php.net/ net_gearman Saturday, 29 October 11
- 75. A Story! Saturday, 29 October 11
- 76. Financial Software Saturday, 29 October 11
- 77. 3000+ Clients Saturday, 29 October 11
- 78. Each one has 5 external data sources Saturday, 29 October 11
- 79. Each data source is a web service Saturday, 29 October 11
- 80. Ran every 6 hours every day Saturday, 29 October 11
- 81. Cron Gearman Web Services Processing 1 Job 1 1 4 5 2 2 5 4 3 3 Saturday, 29 October 11
- 82. But! That wasn’t enough Saturday, 29 October 11
- 83. Job kicked off on login Saturday, 29 October 11
- 86. Another Story! Saturday, 29 October 11
- 88. Near Real Time Cloud Analytics Saturday, 29 October 11
- 89. Clients install logging agent locally Saturday, 29 October 11
- 90. syslogd Saturday, 29 October 11
- 91. Public API Saturday, 29 October 11
- 92. Multiple Persistent Gearman Servers Saturday, 29 October 11
- 93. Internal DB API Saturday, 29 October 11
- 94. Load Agent syslogd Balanced API Load Balanced Worker Internal API Worker Gearman Worker Gearman CouchDB Persistent Saturday, 29 October 11
- 95. CouchDB Setup Saturday, 29 October 11
- 96. Write vs Read Saturday, 29 October 11
- 97. Writes Multi Master setup Replicated Deals with writes only Saturday, 29 October 11
- 98. Reads Multi Master setup Replicated from write cluster Slaves handle website requests Saturday, 29 October 11
- 99. Heavy Map/Reduce usage for data Saturday, 29 October 11
- 100. Questions? @h helgi@orchestra.io Joind.in: http://joind.in/4326 Saturday, 29 October 11