Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pivotal Digital Transformation Forum: Journey to Become a Data-Driven Enterprise

Feras Alsamawi, Senior Field Engineer, Pivotal

Pivotal Digital Transformation Forum: Journey to Become a Data-Driven Enterprise

  1. 1. Journey To Becoming a Data-Driven Enterprise: Pivotal Big Data Suite Technical Overview Feras Alsamawi Senior Field Engineer Pivotal
  2. 2. 2© Copyright 2015 Pivotal. All rights reserved. Agenda •  Big Data Challenges •  The Value of Data •  Pivotal Big Data Suite •  The Open Data Platform •  Big Data Architectures
  3. 3. © Copyright 2015 Pivotal. All rights reserved. THE POWER OF 1 R X Increasing Freight Utilization Rail Predictive Maintenance Healthcare Predictive Diagnostics Power Driving Outcomes That Matter One Percent Improvement Equals $27B Industry Value by Reducing System Inefficiency $63B Industry Value by Reducing Process Inefficiency $66B Industry Value with Efficiency Improvements In Gas-fired Power Plant Fleets Source: General Electric
  4. 4. 4© Copyright 2015 Pivotal. All rights reserved. BIG DATA CHASM 70% of data generated by customers 80% of data stored 3% prepared for analysis 0.5% being analyzed <0.5% being operationalized 4 THE DATA DIVIDE
  5. 5. 5© Copyright 2015 Pivotal. All rights reserved. Software Is Eating The World Data Is Fueling Software SOFTWARE IS EATING THE WORLD
  6. 6. 6© Copyright 2015 Pivotal. All rights reserved. JOURNEY TO A DATA-DRIVEN ENTERPRISE Deploy analytic apps and automate at scale Perform advanced analytics Discover insights Modernize data infrastructure
  7. 7. 7© Copyright 2015 Pivotal. All rights reserved. The value of Data Time Value of Information µs ms s hour day month year yr+
  8. 8. 8© Copyright 2015 Pivotal. All rights reserved. Traditional Systems The value of Data Time Value of Information µs ms s hour day month year yr+
  9. 9. 9© Copyright 2015 Pivotal. All rights reserved. Speed Layer Traditional Systems Pivotal’s λ Architecture Serving Layer “Big Data” Time Value of Information µs ms s hour day month year yr+ Spring XD Pivotal HD Pivotal GemFire Batch Layer
  10. 10. 10© Copyright 2015 Pivotal. All rights reserved. WHY PIVOTAL FOR BIG DATA ? Complete platform SQL on Hadoop leadership Deployment options Open source Flexible licensing Advanced data services Pivotal Data Engineering Pivotal LabsPivotal Data Science
  11. 11. 11© Copyright 2015 Pivotal. All rights reserved. 11© Copyright 2013 Pivotal. All rights reserved. Big Data Suite Components
  12. 12. 12© Copyright 2015 Pivotal. All rights reserved. HDFSData Lake Expert System / Machine Learning In-Memory Real-Time Data Continuous Learning Continuous Improvement Continuous Adapting Data Stream Pipeline Multiple Data Sources Real-Time Processing Store Everything Pro-Active, Self-Improving, Machine Learning Systems
  13. 13. 13© Copyright 2015 Pivotal. All rights reserved. Ingest Transform Sink SpringXD GemFire Data Stream Needs an Agile, Scalable and Fast Solution HAWQ GPDB Data Lake
  14. 14. 14© Copyright 2015 Pivotal. All rights reserved. Ingest Transform Sink SpringXD Distributed Computing In-Memory Real-Time Data Spring XD Orchestrates and Automates all the Steps on Data Stream Pipelining Expert System / Machine Learning Extensible Open-Source Fault-Tolerant Horizontally Scalable HAWQ GPDB Data Lake
  15. 15. 15© Copyright 2015 Pivotal. All rights reserved. Ingest / SINK Process Analyze •  No coding required •  Dozens of built-in connectors •  Seamless integration with Kafka, Sqoop •  Create new connectors easily using Spring •  Call Spark, Reactor or RxJava •  Built-in configurable filtering, splitting and transformation •  Out-of-box configurable jobs for batch processing •  Import and invoke PMML jobs easily •  Call Python, R, Madlib and other tools •  Built-in configurable counters and gauges Spring XD State of the Art Data Pipeline Automation
  16. 16. 16© Copyright 2015 Pivotal. All rights reserved. Ingest Transform Sink SpringXD Distributed Computing GemFire Provides Scalable, Low-Latency Data Access, Storage and Event Processing Expert System / Machine Learning GemFire Extensible Open-Source Fault-Tolerant Horizontally Scalable HAWQ GPDB Data Lake
  17. 17. 17© Copyright 2015 Pivotal. All rights reserved. GemFire •  In-Memory Enterprise Data Grid •  Horizontally Scalable, Consistent, Highly Available •  Event handling •  Continuous Queries •  Enterprise Data Geo Distribution In-memory Real Time Data
  18. 18. 18© Copyright 2015 Pivotal. All rights reserved. Ingest Transform Sink SpringXD Distributed Computing Pivotal Provides SQL Based Advanced Analytics GemFire Extensible Open-Source Fault-Tolerant Horizontally Scalable Data Lake HAWQ GPDB
  19. 19. 19© Copyright 2015 Pivotal. All rights reserved. HAWQ •  Massively Parallel Processing RDBMS on HADOOP •  ANSI SQL on Hadoop •  Extremely high performance for analytics (not like Hive) •  Stores all data directly on HDFS •  Functions in MADlib, R/ Python/Java, Perl, pgsql or C languages Advanced SQL analytics in Hadoop Combining SQL with Hadoop is key for analytics SQL remains #1 choice for Data Science
  20. 20. 20© Copyright 2015 Pivotal. All rights reserved. Data Streaming Reference Architecture Data Feeds Transactional Apps Analytic Apps Data Stream Pipeline Distributed Computing Real-Time Data Expert Systems & Machine Learning Advanced Analytics HDFSData Lake
  21. 21. 21© Copyright 2015 Pivotal. All rights reserved. Data Streaming Reference Architecture Data Feeds Transactional Apps Analytic Apps Data Stream Pipeline HDFSData Lake GemFire HAWQ GPDB SpringXD
  22. 22. BUILT FOR THE SPEED OF BUSINESS
  23. 23. 23© Copyright 2015 Pivotal. All rights reserved. Financial Compliance BUSINESS PROBLEM •  Ensure compliance with Dodd-Frank and Basel Committee regulations •  Identify underlying risk and fraud while reducing the compliance department’s overburdened Emails Chats Trades Transactions Policy Securities Phone Calls Watch Lists … Financial compliance Data Lake Data integration Data clean up Modeling Classification and ranking Analyst user interfaces Feedback Analytics Analyst feedback Data integration: e.g., append trade information with email and chat communications Data cleanup: e.g., identify newsletters and spam emails Modeling: •  Predictive modeling to flag messages and trades •  Graph and cohort analysis Analyst feedback Reviewed fraud instances included in periodic model refreshes SOLUTION Ÿ  A data lake platform coupled with cutting edge data science techniques Ÿ  Flexible user interface to promote an adaptive, continuously learning compliance framework
  24. 24. 24© Copyright 2015 Pivotal. All rights reserved. Pivotal Topic & Sentiment Analysis Engine External Tables PXF HDFS Source: http Sink: hdfs Parallel Parsing of JSON (PL/Python) HAWQ Nightly Cron Jobs Topic Analysis through MADlib pLDA Unsupervised Sentiment Analysis (PL/Python) D3.js Spring XD Twitter Decahose (~55 million tweets/day)
  25. 25. 25© Copyright 2015 Pivotal. All rights reserved. 25© Copyright 2013 Pivotal. All rights reserved. IoT Architectures – The connected Car
  26. 26. 26© Copyright 2015 Pivotal. All rights reserved. The Connected Car Architecture INGESTION JSON / HTTP STREAM PROCESSING Spring XDTransformEnrich DATA LAKE Pivotal HDSink ADVANCED ANALYTICS HAWQ REAL-TIME DATA INSIGHTS GemFire MOBILE SERVICES MICROSERVICES Pivotal CF Dashboard Analytics App Simulator IoT APPS Rabbit MQ PUSH
  27. 27. 27© Copyright 2015 Pivotal. All rights reserved. Horizontally Scalable Fault Tolerant Extensible Open-Source STREAM PROCESSING Spring XD Rabbit MQ DATA LAKE Pivotal HD ADVANCED ANALYTICS HAWQ ENRICHER PREDICTIVE ANALYTICS + Timestamp & GUID + MPG, rangE & route MOBILE APP JSON REAL-TIME DATA INSIGHTS GemFire CAR SENSOR Sink Tap DASHBOARD
  28. 28. 28© Copyright 2015 Pivotal. All rights reserved. FOR FURTHER INFO… •  Pivotal Data Product Info, Docs and Downloads @ http://pivotal.io/big-data •  Pivotal Blog @ http://blog.pivotal.io •  Pivotal Data Science Blog @ http://blog.pivotal.io/data-science-pivotal •  Pivotal Academy @ https://pivotal.biglms.com •  Or reach out to your local Pivotal Account Executive…
  29. 29. Digital Transformation Forum Disrupt or Be Disrupted 19 OCTOBER · BMW WELT EVENT CENTRE · MUNICH

×