Feras Alsamawi, Senior Field Engineer, Pivotal

1. Journey To Becoming a Data-Driven Enterprise:
Pivotal Big Data Suite Technical Overview
Feras Alsamawi
Senior Field Engineer
Pivotal

2. 2© Copyright 2015 Pivotal. All rights reserved.
Agenda
• Big Data Challenges
• The Value of Data
• Pivotal Big Data Suite
• The Open Data Platform
• Big Data Architectures

3. © Copyright 2015 Pivotal. All rights reserved.
THE POWER OF 1
R
X
Increasing
Freight Utilization Rail
Predictive
Maintenance Healthcare
Predictive
Diagnostics Power
Driving Outcomes That Matter
One Percent Improvement Equals
$27B
Industry Value by
Reducing System
Inefficiency
$63B
Industry Value by
Reducing Process
Inefficiency
$66B
Industry Value with
Efficiency Improvements
In Gas-fired Power
Plant Fleets
Source: General Electric

4. 4© Copyright 2015 Pivotal. All rights reserved.
BIG DATA
CHASM
70%
of data
generated by
customers
80%
of data stored
3%
prepared for
analysis
0.5%
being
analyzed
<0.5%
being
operationalized
4
THE DATA DIVIDE

5. 5© Copyright 2015 Pivotal. All rights reserved.
Software Is Eating The World
Data Is Fueling Software
SOFTWARE IS EATING THE WORLD

6. 6© Copyright 2015 Pivotal. All rights reserved.
JOURNEY TO A DATA-DRIVEN ENTERPRISE
Deploy analytic apps and
automate at scale
Perform advanced analytics
Discover insights
Modernize data
infrastructure

7. 7© Copyright 2015 Pivotal. All rights reserved.
The value of Data
Time
Value of
Information
µs ms s hour day month year yr+

8. 8© Copyright 2015 Pivotal. All rights reserved.
Traditional
Systems
The value of Data
Time
Value of
Information
µs ms s hour day month year yr+

9. 9© Copyright 2015 Pivotal. All rights reserved.
Speed Layer
Traditional
Systems
Pivotal’s λ Architecture
Serving Layer
“Big Data”
Time
Value of
Information
µs ms s hour day month year yr+
Spring XD
Pivotal HD
Pivotal GemFire
Batch
Layer

10. 10© Copyright 2015 Pivotal. All rights reserved.
WHY PIVOTAL FOR BIG DATA ?
Complete
platform
SQL on Hadoop
leadership
Deployment
options
Open source
Flexible licensing
Advanced data
services
Pivotal Data Engineering Pivotal LabsPivotal Data Science

11. 11© Copyright 2015 Pivotal. All rights reserved. 11© Copyright 2013 Pivotal. All rights reserved.
Big Data Suite Components

12. 12© Copyright 2015 Pivotal. All rights reserved.
HDFSData Lake
Expert System /
Machine Learning
In-Memory
Real-Time Data
Continuous Learning
Continuous Improvement
Continuous Adapting
Data Stream Pipeline
Multiple Data Sources
Real-Time Processing
Store Everything
Pro-Active, Self-Improving, Machine Learning
Systems

13. 13© Copyright 2015 Pivotal. All rights reserved.
Ingest Transform Sink
SpringXD
GemFire
Data Stream Needs an Agile, Scalable and
Fast Solution
HAWQ GPDB
Data
Lake

14. 14© Copyright 2015 Pivotal. All rights reserved.
Ingest Transform Sink
SpringXD
Distributed
Computing
In-Memory
Real-Time Data
Spring XD Orchestrates and Automates all the
Steps on Data Stream Pipelining
Expert System /
Machine Learning
Extensible
Open-Source
Fault-Tolerant
Horizontally Scalable HAWQ GPDB
Data
Lake

15. 15© Copyright 2015 Pivotal. All rights reserved.
Ingest / SINK Process Analyze
• No coding required
• Dozens of built-in
connectors
• Seamless integration with
Kafka, Sqoop
• Create new connectors
easily using Spring
• Call Spark, Reactor or
RxJava
• Built-in configurable filtering,
splitting and transformation
• Out-of-box configurable jobs
for batch processing
• Import and invoke PMML
jobs easily
• Call Python, R, Madlib and
other tools
• Built-in configurable
counters and gauges
Spring XD
State of the Art Data Pipeline Automation

16. 16© Copyright 2015 Pivotal. All rights reserved.
Ingest Transform Sink
SpringXD
Distributed
Computing
GemFire Provides Scalable, Low-Latency
Data Access, Storage and Event Processing
Expert System /
Machine Learning
GemFire
Extensible
Open-Source
Fault-Tolerant
Horizontally Scalable HAWQ GPDB
Data
Lake

17. 17© Copyright 2015 Pivotal. All rights reserved.
GemFire
• In-Memory Enterprise Data Grid
• Horizontally Scalable, Consistent, Highly
Available
• Event handling
• Continuous Queries
• Enterprise Data Geo Distribution
In-memory Real Time Data

18. 18© Copyright 2015 Pivotal. All rights reserved.
Ingest Transform Sink
SpringXD
Distributed
Computing
Pivotal Provides SQL Based
Advanced Analytics
GemFire
Extensible
Open-Source
Fault-Tolerant
Horizontally Scalable
Data
Lake
HAWQ GPDB

19. 19© Copyright 2015 Pivotal. All rights reserved.
HAWQ
• Massively Parallel Processing
RDBMS on HADOOP
• ANSI SQL on Hadoop
• Extremely high performance for
analytics (not like Hive)
• Stores all data directly on
HDFS
• Functions in MADlib, R/
Python/Java, Perl, pgsql or C
languages
Advanced SQL analytics in Hadoop
Combining SQL with Hadoop is key for analytics
SQL remains #1 choice for Data Science

20. 20© Copyright 2015 Pivotal. All rights reserved.
Data Streaming Reference Architecture
Data Feeds Transactional Apps Analytic Apps
Data Stream Pipeline
Distributed
Computing
Real-Time Data
Expert Systems &
Machine Learning
Advanced
Analytics
HDFSData Lake

21. 21© Copyright 2015 Pivotal. All rights reserved.
Data Streaming Reference Architecture
Data Feeds Transactional Apps Analytic Apps
Data Stream Pipeline
HDFSData Lake
GemFire HAWQ GPDB
SpringXD

22. BUILT FOR THE SPEED OF BUSINESS

23. 23© Copyright 2015 Pivotal. All rights reserved.
Financial Compliance
BUSINESS PROBLEM
• Ensure compliance with Dodd-Frank and Basel
Committee regulations
• Identify underlying risk and fraud while reducing the
compliance department’s overburdened
Emails Chats Trades
Transactions Policy Securities
Phone Calls Watch Lists …
Financial compliance
Data Lake
Data
integration
Data clean
up
Modeling
Classification
and ranking
Analyst user interfaces
Feedback
Analytics
Analyst feedback
Data integration: e.g., append
trade information with email and
chat communications
Data cleanup: e.g.,
identify newsletters and
spam emails
Modeling:
• Predictive modeling to flag
messages and trades
• Graph and cohort analysis
Analyst feedback
Reviewed fraud instances
included in periodic model
refreshes
SOLUTION
Ÿ A data lake platform coupled with cutting edge data
science techniques
Ÿ Flexible user interface to promote an adaptive,
continuously learning compliance framework

24. 24© Copyright 2015 Pivotal. All rights reserved.
Pivotal Topic & Sentiment Analysis Engine
External
Tables
PXF
HDFS
Source: http Sink: hdfs
Parallel Parsing
of JSON
(PL/Python)
HAWQ
Nightly Cron Jobs
Topic Analysis
through MADlib pLDA
Unsupervised
Sentiment Analysis
(PL/Python)
D3.js
Spring XD
Twitter Decahose (~55 million tweets/day)

25. 25© Copyright 2015 Pivotal. All rights reserved. 25© Copyright 2013 Pivotal. All rights reserved.
IoT Architectures
– The connected Car

26. 26© Copyright 2015 Pivotal. All rights reserved.
The Connected Car Architecture
INGESTION
JSON / HTTP
STREAM PROCESSING
Spring XDTransformEnrich
DATA LAKE
Pivotal HDSink
ADVANCED ANALYTICS
HAWQ
REAL-TIME DATA INSIGHTS
GemFire
MOBILE SERVICES
MICROSERVICES
Pivotal CF Dashboard Analytics App Simulator
IoT APPS
Rabbit MQ
PUSH

27. 27© Copyright 2015 Pivotal. All rights reserved.
Horizontally Scalable
Fault Tolerant
Extensible
Open-Source
STREAM PROCESSING
Spring XD
Rabbit MQ
DATA LAKE
Pivotal HD
ADVANCED ANALYTICS
HAWQ
ENRICHER PREDICTIVE ANALYTICS
+ Timestamp
& GUID
+ MPG, rangE
& route
MOBILE
APP
JSON
REAL-TIME DATA INSIGHTS
GemFire
CAR
SENSOR
Sink
Tap
DASHBOARD

28. 28© Copyright 2015 Pivotal. All rights reserved.
FOR FURTHER INFO…
• Pivotal Data Product Info, Docs and Downloads @ http://pivotal.io/big-data
• Pivotal Blog @ http://blog.pivotal.io
• Pivotal Data Science Blog @ http://blog.pivotal.io/data-science-pivotal
• Pivotal Academy @ https://pivotal.biglms.com
• Or reach out to your local Pivotal Account Executive…

29. Digital Transformation Forum
Disrupt or Be Disrupted
19 OCTOBER · BMW WELT EVENT CENTRE · MUNICH