Data and its Role in Your Digital Transformation

1. The Role of Data in Your Digital Transformation Les Klein Field CTO Data Pivotal @LesKlein #PivotalForum #DigitalTransformation #Istanbul #BigData #Analytics

2. Great software companies leverage Big Data to fundamentally change the consumer experience and pioneer entirely new business models

3. 4© 2016 Pivotal Software, Inc. All rights reserved. $4BN Financial Services $26BN Hospitality $50BN Transportation $54BN Entertainment $30BN Automotive $3.2BN Industrial Products CLOUD NATIVE SOFTWARE IS CHANGING INDUSTRIES Data is Fueling Software

4. 5© Copyright 2015 Pivotal. All rights reserved. Hundreds of thousands of “trip” events each day 400+ billion of viewing-related events per day Five billion training data points for Price Tip feature Disruptors Use a LOT of Data

5. 6© Copyright 2015 Pivotal. All rights reserved. “We’ve found that when a host selects a price that’s within 5% of their tip, they’re nearly 4 times more likely to get booked” “The importance of accuracy and efficiency […], will continue to rise as we expand and improve products like uberPOOL and beyond.” “Over 75% of what people watch come from our recommendations” Data manifests as features in an app

6. 7© Copyright 2015 Pivotal. All rights reserved. (Data) Microservices Loosely coupled services architecture, bounded by context Cloud-Native Platforms Enabling continuous delivery & automated operations Open Source Database Innovation Extreme scale & performance advantages, built for the cloud Machine Learning Use of predictive analytics to build smart apps How are they accomplishing this?

7. 8© Copyright 2015 Pivotal. All rights reserved. These companies… Release new features in minutes, multiple times a day Support a micro-services architecture Consume a wide range of data sources and protocols Store and Analyze all their data Update algorithms and predictive models daily Continuously ask lots of questions of their data Modify data pipelines and add processing steps daily They are Data Driven, are you?

9. 10© Copyright 2015 Pivotal. All rights reserved. Converging Trends Innovation New Data New Processes New Insights The Journey to the Data-Driven Enterprise Data Science and Machine Learning Big Data Internet of Things & Social Media

10. 11© Copyright 2015 Pivotal. All rights reserved. HDFS Data Lake Ingest Store Analytics Hard to change Labor intensive Inefficient Coding based No real-time information Based on expensive ETL Migrating from a Reactive, Static and Constrained Model…

11. 12© Copyright 2015 Pivotal. All rights reserved. HDFSData Lake Expert System / Machine Learning In-Memory Real- Time Data Continuous Learning Continuous Improvement Continuous Adapting Data Stream Pipeline Multiple Data Sources Real-Time Processing Store Everything To Pro-Active, Self-Improving, Machine Learning Systems

12. 13© Copyright 2015 Pivotal. All rights reserved. “Companies need to learn how to catch people or things in the act of doing something and affect the outcome“ PAUL MARITZ Executive Chairman, Pivotal Real-time and Personalised Information in Context is what Wins!

14. 15© 2016 Pivotal Software, Inc. All rights reserved. The new normal: “an audience of one” DATA DEVICES Media Banks Delivery Services Marketers Government Individual s Employers Analytic Services Advertising Catalog Co-ops List Brokers Websites Information Brokers Credit BureausMedia Archives GOVERNMENTGOVERNMENT PHONE/ TV PHONE/ TV INTERNETINTERNET AD AGENCY AD AGENCY RETAILRETAIL

19. 20© 2016 Pivotal Software, Inc. All rights reserved. “A data native is someone who expects their world to not just be digital, but to be smart and to adjust immediately to their taste and habits.” Beyond tech savvy …..

23. 24© 2016 Pivotal Software, Inc. All rights reserved. µs ms s hour day month yr+ “Fast Data” “Big Data” year Traditional Systems Pivotal Data Science Labs Value of Data ($) Time Value of data over time…

24. 25© 2016 Pivotal Software, Inc. All rights reserved. Analytics - the journey so far (BI) Complexity Value of Analytics ($) Descriptive Analytics Diagnostic Analytics What happened? Why did it happen? Hindsight Insight

25. 26© 2016 Pivotal Software, Inc. All rights reserved. Analytics – where you need to be today Value of Analytics ($) Descriptive Analytics Diagnostic Analytics Predictive Analytics Prescriptive Analytics What happened? Why did it happen? What will happen? How can we make it happen? Hindsight Insight Foresight Complexity

26. 27© 2016 Pivotal Software, Inc. All rights reserved. Interesting questions are found here Value of Analytics ($) Descriptive Analytics Diagnostic Analytics Predictive Analytics Prescriptive Analytics What happened? Why did it happen? What will happen? How can we make it happen? Hindsight Insight Foresight Complexity Data science

27. 28© 2016 Pivotal Software, Inc. All rights reserved. Data Science at Pivotal Theory-driven approach ‘start with the problem and work towards the data’ ‘start with the data and work towards the problem’ Data-driven approach

29. 30© 2016 Pivotal Software, Inc. All rights reserved. User Feedback Scoping Data Review Model Building Feature Review Feature Engineering Model Evaluation Operationalization Our tried and tested approach helps extract value from data

30. “ THE MAGIC HAPPENS WHEN YOU MARRY THE TRADITIONAL ENGINEERING APPROACH WITH THE DATA SCIENCE ENABLED BY THE DATA LAKE. IT OPENS UP A WHOLE NEW WORLD OF POSSIBLE ‘WHAT IF’ QUESTIONS. ”DAVE BARTLETT, GE AVIATION

33. 34© 2016 Pivotal Software, Inc. All rights reserved. OSS Smart/Data-driven App Technology Needs Fast Ingest/Pipelining • Pipelines to consume streaming and batch data from various endpoints Speed/Serving Layer • In-memory data grids for real-time and aggregated data sets • Low latency Big Data Analytics Platform • Scale-out Storage • Ability to process large structured and unstructured data • Advanced SQL query capabilities • Ability to run machine learning at scale Platform for Scalable Web/Mobile Apps • Infrastructure Automation • Develop, run and manage Web applications without the complexity of building and maintaining the infrastructure

34. 35© 2016 Pivotal Software, Inc. All rights reserved. Fast Ingest/Pipelining Speed/Serving Layer Big Data Analytics Platform Platform for Scalable Web/Mobile Apps Pivotal’s Answer for Technology Needs Spring Cloud Data Flow GemFire Greenplum Database Pivotal HDB Pivotal Cloud Foundry Spring OSS

35. 36© 2016 Pivotal Software, Inc. All rights reserved. ● Silo’d and aging database systems ● Spaghetti data pipelines ● Expensive, proprietary data management systems ● Lack of structured platforms for continuous software delivery ● Monolithic application architectures ● Batch-oriented data integration ● Limited operationalization of analytics ● Proprietary systems Today’s Enterprise Challenge

36. 37© 2016 Pivotal Software, Inc. All rights reserved. Data Programming Model Cloud-Native Platform Microservices FrameworkPlatform Runtime Stream Data Platform Hadoop Spark DW Apps & Microservices DBMS IMDG K/V Store Relational DB Big Data & Machine Learning Modern Cloud-Native Data Architecture

37. 38© 2016 Pivotal Software, Inc. All rights reserved. 1. Modernize your data platforms: • Improve performance and scalability for mission-critical data workloads • Lower overall TCO by adopting industry-leading open source database technologies on modern cloud platforms 2. Prove out a specific business use-case: • Build a smart app end-to-end • Focus on a specific analytical model / data-microservice • Automate ingestion and data pipelines Pivotal can help you to:

38. 39© 2016 Pivotal Software, Inc. All rights reserved. Pivotal Offerings Big Data Suite: • Best-in-class open source data management software • GemFire: In-Memory Data Grid • Greenplum: Data Warehouse • Pivotal HDB (Apache HAWQ): Hadoop-Native SQL Pivotal Cloud Foundry • Industry’s Leading Cloud-Native Platform Spring Boot, Cloud and Data Flow • Modern-Java micro-services framework Pivotal Labs & Data Science • Agile software development and machine learning

39. Let’s build something MEANINGFUL Let’s build something MEANINGFUL

42. 43© 2016 Pivotal Software, Inc. All rights reserved. App Development Data analytics Cloud-native App platform Data Science & Model building Data Micro- service APP Must support scale-out query processing Must support scale-out query processing Must deliver as an APIMust deliver as an API Must embrace agile development, focus on outcomes Must embrace agile development, focus on outcomes Must support microservices, agile dev, and connect to big data analytics Must support microservices, agile dev, and connect to big data analytics A Real-World Example

43. 44© 2016 Pivotal Software, Inc. All rights reserved. Major Credit Card Issuer Builds Infrastructure Determining Fraud Markers Across 30 Disparate Sources Challenge: • Desire to identify potential fraudulent activity within Fortune 100 credit card issuer with 100K employees • Identifying markers across 30 internal data sources caused lags in reporting • Establish Infrastructure to support Palantir, Platfora and RSA reporting frameworks Solution: • Provided ability to store 3 months of data at 60-70TB, interest in moving to 3-year retention period • Build large-scale security analytics platform to scale with data needs, analytics and reporting requirements • Support organizational desire to evolve from pure BI to advanced, predictive analytics Pivotal Solution includes: GPDB, Hadoop, HAWQ

44. 45© 2016 Pivotal Software, Inc. All rights reserved. Leading Swiss Bank Re-shapes Business to Better Monetize and Regulate Fraudulent Activity with Big Data Challenge: • Needed to monetize data around privilege user access and assure security around fraudulent activity • Reduce the cost of monitoring privileged user access for regulatory requirements. (currently taking 45 people to do this) • Extend across the enterprise to solve business pains around faster data as opposed to just longer looks in “Hadoop” Solution: • Built on top of Big Data Suite to operationalize, automate and extend across the enterprise • What took 6 hours should now take less than 1 hour with real time decision-making • Capability of monitoring relationship manager’s activity with analytics to ensure regulatory compliance Pivotal Solution includes: Big Data Suite and Data Labs

45. 46© 2016 Pivotal Software, Inc. All rights reserved. Pivotal Data Innovation Across Industries RETAIL / TRANSPORT MEDIA / SERVICES TELCO / ISP FINANCIAL SERVICES HIGH TECH / ELECTRONICS EDUCATION GOVERNMENT OTHER INSURANCE AUTO / INDUSTRIAL HEALTHCARE MANUFACTURING