Scaling an API
•
2 likes•586 views
Scaling an API. A masterclass given at Seedcamp London product day 2012
Recommended
More Related Content
Viewers also liked
Similar to Scaling an API
Similar to Scaling an API (20)
Recently uploaded
Recently uploaded (20)
Scaling an API
- 1. Scaling an API From Reboots to Redis, how (not) to do it
- 2. { "fullName": "Ciarán Rooney", "workLife": { "company": "Skimlinks", "position": "CTO", "website": "http://skimlinks.com" }, "homeLife": { "email": "ciaran.rooney@gmail.com", "twitter": "@CiaranR", "website": "http://ciaran.ie" "hobby": "http://phplondon.org" } }
- 3. { "dataYear": "2012", "dataMonth": "august", "dataPeriod": "31 days", "dataRequests": { "apiRequests": "14,256,654,541", "impressions": "4,073,142,857", "uniqueUsers": "145,137,679" }, "customers": { "websitesPowered": "987,348", "customers": "26,379", } }
- 16. BIND / DNS eMail imageStorage
- 18. imageStorage
- 25. APIs
- 27. Client App
- 30. WordPress
- 31. WordPress Pinterest
- 33. Black Friday '09
- 34. Black Friday '09 Black Friday '10
- 35. Black Friday '09 Black Friday '11 Black Friday '10
- 39. WordPress
- 42. SkimWords ™
- 44. APIxx
- 45. CLUSTERxx APIxx APIxx APIxx APIxx APIxx APIxx APIxx APIxx ELB ELB
- 46. US-WEST US-EAST EU-WEST CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxx
- 49. Black Friday '10
- 50. Pinterest
- 56. CLUSTERxx Apache Flume Collector C Prefix Tree C NLP Filtering JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx ELB ELB
- 57. US-WEST US-EAST EU-WEST Apache Flume Master CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxx
- 59. Cacti
- 61. { "dataYear": "2012", "dataMonth": "august", "dataPeriod": "31 days", "dataRequests": { "apiRequests": "14,256,654,541", "impressions": "4,073,142,857", "uniqueUsers": "145,137,679" }, "customers": { "websitesPowered": "987,348", "customers": "26,379", } }
- 62. Scaling an API From Reboots to Redis, how (not) to do it Questions?
- 63. { "fullName": "Ciarán Rooney", "workLife": { "company": "Skimlinks", "position": "CTO", "website": "http://skimlinks.com" }, "homeLife": { "email": "ciaran.rooney@gmail.com", "twitter": "@CiaranR", "website": "http://ciaran.ie" "hobby": "http://phplondon.org" } }
Editor's Notes
- Good Afternoon big lunch, not put you sleep share my experiences So from reboots, To Redis, still works my journey Not how make Not why should But What happens when BIG
- So who I am? Ciaran, CTO Skimlinks. former seedcamp Content monetization company service ordinated architecture Internal and external APIs why me? Numbers!
- 14 Billion API requests 4 Billion Page views 145 Million Unique users monetize 1million Websites 26 thousand customers didn’t start like that
- Started Skimbit.com Rebranded skimit.com social decision making and bookmarking tool Launched 2007 topic create a page
- bookmark items brides maids shoes friends and family comments / rating bit like another site
- But we’ll come back to that site in a bit
- going ok steady base of users ticking along nicely something BIG happened
- We got Techcrunched!! Website didn’t go down So …
- Partied! cut short then something else happened
- hosted at FlexiScale anyone? No? Small, local, cloud CEO’s number on speed dial Everything we need
- App very straight forward It looked like this Few still like this? node.js on solar powered raspberry pi’s Ours & we were happy with it one day this happened ->
- three day outage to the hour wasn’t our fault Well really it was
- the cloud floated away human error deleted main storage restore via remount Not enough hardware ship from Germany Didn’t have a backup plan
- We panicked! CEO’s number no answer
- Our app is simple Just deploy Not that easy We thought But…
- DNS servers were OUR server Email records on OUR Server Even customer uploaded images on it This is how NOT to do it
- DNS was the hardest 24 Hour TTL on NS records
- Email right after Already using Google Apps 30 min TTL on MX Once DNS back ..
- Finally Back up images to S3
- Shortly after this We started October 2008
- Well more like this Fist customer live Nov 10 2008
- We business means new website Nov 2008
- New hosting company The Planet Real dedicated servers No cloud float Backward? Sensible
- New App New architecture Half dozen servers
- 2 API boxes Mostly read Writes local
- Master slave DB MySQL replication Batch writes to master
- Finally client app Reports Website Etc Last mention
- Simple Unbreakable
- Growth over the next few years Interesting events
- Signed Wordpress
- Pinterest took off!
- Zoom in Monetizable events More insight
- BF 2009
- BF 2010
- BF 2011
- Remember our site
- Hosting = planet Real severs Never float
- Archictecure = this Never could break
- Stopped recording Important lesson Only record what you need Later we did record
- International traffic growing 40% = USA 20% = UK 10% = France 30% =others like germany canada
- Started to improve New Website easly 2010
- New product Skimwords Analyses content Creates links
- Great thing New products go on new servers Eventually all on AWS This where start talking about the right way
- Created API boxen Apache PHP Memcached
- Scaled out API Added ELB Backed by MYSQL Skimwords = MongoDB
- Amazon regions Availabily zones Mysql Master slave Mongo Master Slave Mongo 1.6 Ultra DNS = GLB Failover Short TTL
- It worked Latency dropped huge
- World Graphs Blue = europe Green = us east Red = us west
- We were where BF ‘10
- And this was coming Pinterest over 1 year Growing fast
- First step New website Early 2011
- We bought Atma
- And did some research Bake off Apache Worker = Not thread safe PHP Drivers Mongo at the time
- Picked Python Why not scala? Many reasons Team skills
- New Jay Box Jay = Json Ngix PHP/Apache Tornado / Python Flume for logs+events
- Make a Cluster Jays ELB Mysql Memcached Flume C Tree Filter + Redis
- Again International Clusters Balance with Ultra DNS Mysql Master Slave Flume -> S3 Hadoop Hbase
- Other tools Puppet Jenkins-ci / RMP’s / Hubot Fastly Api Axle
- Monitoring Local Cacti Icinga aka nagios Pindom & Padgerduty
- NOC Always watching
- Remember numbers
- Good Afternoon big lunch, not put you sleep share my experiences So from reboots, To Redis, still works my journey Not how make Not why should But What happens when BIG
- My Details Hope you got something from this We learned a lot Hope you can apply to your business here for the rest of day Hope you all get a lot out of the rest of the week