Recommended
More Related Content
What's hot
What's hot (20)
Similar to Big data in action
Similar to Big data in action (20)
More from Tu Pham
More from Tu Pham (20)
Recently uploaded
Recently uploaded (20)
Big data in action
- 3. For the past 17 years, Google has been building out the world’s fastest, most powerful, highest quality cloud infrastructure on the planet. Images by ConnieZhou
- 4. Google Cloud Platform is built on the s a m e infrastructure that powers Google. Images by Connie Zhou
- 5. Google’s Platform “[Google's] ability to build, organize, and operate a huge network of servers and fiber- optic cables with an efficiency and speed that rocks physics on its heels. This is whatmakes Google Google: its physical network, its thousands of fiber miles, and those many thousands of servers that, in aggregate, add up to the mother of all clouds.” - Wired
- 7. Yes, We Can Power that Web Mobile Storage & Database Big Data Highly Scalable System Data Mining Cloud Platform
- 9. Google CloudPlatform 5 Source: Boston Consulting Group: The Mobile Revolution: How Mobile Technologies Drive a Trillion-DollarImpact IDC,2015 By 2020, there will be 8 Billion connected smart phones — 2X more than today. And 32 Billion connected “IOT”devices —6X more thantoday.
- 11. Google Compute Engine Cloud Platform • Flexible Infrastructure • Customer VM Size • Online Disk Resizing • Network • Internal Network • Firewall • Load Balancing • External Ip Address • Billing • Sustained Usage Discounts • Preemptible VM
- 12. App Engine • Fully Managed Platform • Popular Programming Language Support • Flexible and Scalable Application Storage • Auto-scaling • Versioning and Traffic Splitting • Local Developer Tools • Third-party Frameworks and Extensions Cloud Platform
- 13. • Global Presence • Flexible Delivery Options • Pull • Push • Data Reliability • Flow Control • Data Security And Protection Cloud Platform Pub Sub
- 14. • Reliable & Consistency Processing • Unified Programing Model • Intelligence Work Scheduling • Auto Scaling • Monitoring • Open Source Cloud Platform Cloud Data Flow
- 15. • Versioning • Static Sites • Resumable Transfers • Object Change Notifications • TB scale Cloud Platform Cloud Storage
- 16. Cloud SQL • Fully managed • Ease of Use • Highly Reliable • Flexible Charging • Security, Availability, Durability • Easy Migration & Data Portability • Optimized Mysql versions Cloud Platform
- 17. BigQuery • Fully Managed Big Data Analytics Service • Support SQL • Fast • Scalable • Flexible and Familiar • Security and Reliability Cloud Platform
- 18. Data Proc • Includes • Apache Hadoop • Apache Pig • Apache Hive • Apache Spark • Fast And Scalable Data Processing • Flexible Virtual Machines • Resizable Cluster Cloud Platform
- 19. Datalab • Powerful Data Exploration • Scalable • Data Management • Visualization • Open Source (Jupyter) Cloud Platform
- 20. 10+ Years of Tackling Big Data Problems Google CloudPlatform 13 Google Papers 20082002 2004 2006 2010 2012 2014 2015 GFS Map Reduce Flume Java Millwheel Open Source 2005 Google Cloud Products BigQuery Pub/Sub Dataflow Bigtable BigTable Dremel PubSub Apache Beam Tensorflow
- 21. Google’s Data Services for everyone
- 22. Confidential + Proprietary Storage andDatabases CloudStorage The Google Cloud data toolbox CloudSQL CloudBigtable CloudDatastore Big Data andAnalytics BigQuery CloudPub Sub CloudDataflow CloudDataproc CloudDatalab Machine Learning Cloud Machine Learning Cloud TranslateAPI Cloud VisionAPI Cloud SpeechAPI
- 23. A common configuration: draw conclusions CloudDatalab Events, metrics, etc. Stream Visualization and BI Raw logs, files, assets, Google Analytics data etc. Co-workers Batch Batch B C Applications and A Reports Confidential +Proprietary A serverless big data stack that scales automatically
- 24. Confidential &ProprietaryGoogle CloudPlatform 18 Complexities of Big Data Processing Programming Resource provisioning Performance tuning Monitoring Reliability Deployment& configuration Handling growingscale Utilization improvement s Time to Understanding Typical Big Data Processing
- 25. Confidential &ProprietaryGoogle CloudPlatform 19 Spend Time on ‘What’ not ‘How’ Time toUnderstanding Big Data Processing with Google Cloud Platform Programming More time to dig into your data
- 26. Big Data Lifecycle Cloud Logs Google App Engine Google Analytics Premium Cloud Pub/Sub BigQuery Storage (Tables) Cloud Bigtable (NoSQL) Cloud Storage (Files) Cloud Dataflow BigQuery Analytics (SQL) Analyze Batch Capture Process Store Stream Cloud Monitoring Real-time analytics Cloud Dataflow CloudML Real-time dashboard Real-time alerts Use Data Scientists Analysts Smart apps Catalog & Data Lifecycle Automation Cloud Datalab Cloud Dataproc Data Studio
- 27. Confidential &ProprietaryGoogle CloudPlatform 21 Emerging Big Data Challenges Real-time data ingestionBatch or streaming? Analytics at the speed of thought
- 28. Batch orStreaming? Why do you have to choose? Breakthrough #1
- 29. Google CloudPlatform Confidential &Proprietary We don’t really use MapReduceanymore UrsHölzle SVP Technical Infrastructure Google “ ”23
- 30. Confidential + Proprietary A common configuration: capturing input Cloud Pub/Sub Reliable, many-to-many, asynchronous messaging CloudStorage Powerful, simple and cost-effective objectstorage Raw logs, files, assets, Google Analytics data etc. Events, metrics, etc.
- 31. Confidential + Proprietary A common configuration: process and transform Events, metrics, etc. Cloud Dataflow Data processing engine for batch and streamprocessing Stream Batch Raw logs, files, assets, Google Analytics data etc.
- 32. Confidential + Proprietary A common configuration: process and transform Events, metrics, etc. Cloud Dataflow Data processing engine for batch and streamprocessing Stream Batch Batch CloudDataproc Managed Spark andHadoop Raw logs, files, assets, Google Analytics data etc.
- 34. Confidential + Proprietary A common configuration: draw conclusions Events, metrics, etc. Stream Batch Applications and Reports CloudDatalab Visualization and BI Co-workers Batch B C A Raw logs, files, assets, Google Analytics data etc.
- 35. Real-time data ingestion (and at scale) Breakthrough #2
- 36. Google confidential │ Do not distribute Overview: Data to process: Data in the Consolidated Audit Trail (CAT). A data repository of all equities and options orders, quotes, and events Challenges: How to process the CAT and organize 100 billion market events into an “order lifecycle” in a 4 hour window Store 6 years (~30PB) of data Cloud Bigtable to process and run queries and tolerate volume increases 6BILLION MARKET EVENTS WRITTEN PER HOUR 1.7 GIGs 6TBs 10 BN WRITTEN PER HOURBURSTS 1.7 GIGABYTES PER SECOND 10TERABYTES PER HOUR
- 37. Analytics at the speedof thought (and at scale) Breakthrough #3
- 38. Building what’s next 33 Scales automatically No setup or administration Stream up to 100,000 rowsp/sec Easily integrates with third-partysoftware Google BigQuery makes complex data analysis simple
- 39. Confidential + Proprietary Google BigQuery Performance Example ? Running an inefficient regular expression over 100 billion rowsin less than 60 seconds Source: https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-bigquery- query
- 40. Google BigQuery The Power of Google Dremel for everyone Storage Compute Fast Ingest Query Terabit Network
- 41. 1000-core Hadoop Cluster = 2.5 hours Before Making ad hocQueries with BigQuery <5min After ● 500+ Games ● Hundreds of Analysts ● Terabytes of Data Daily
- 43. “Right at the start of the partnership we were able to reduce time to insight from 96 hours to 30 minutes by using BigQuery, allowing us to react in real time to customer needs and provide better service..” GarySanders Head of the bank's digital analyticsfunction https://www.finextra.com/newsarticle/28566/lloyds-partners-google-on-data-analytics
- 44. Connect and Stay Informed Google Cloud Platform WebSite https://cloud.google.com Google Cloud Platform Blog https://googlecloudplatform.blogspot.com Google+ https://plus.google.com/+GoogleCloudPlatform Twitter https://twitter.com/GoogleCloud
- 45. Demo: Tracking Pixel • Compute Engine • Instance Template • Access Scope • Instance Group • Network • Firewall • Load Balancing • Stackdriver • BigQuery • Data Lab • App Engine • Source Repositories
- 46. Bring it all together
- 48. Thank you! Questions? ● tu@dyno.me ● @phamptu