Practical Approach to BIG DATA and USE CASES in big data

1. June 15, 2013
AXEOR and IBM Present…
Capitalize on the Power of Big Data to Transform
Marketing
Presented by: Nick Kabra, Advisor to Axeor and IBM Big Data Practice

2. The Big Question about BIGGGGGG DATA:
 BIG data is a big buzzword to make big
money by solving seemingly BIG …..
PROBLEMS….
 The 6Vs…. Can there be more…
 Volume
 Velocity
 Variety
© 2013 IBM Corporation 2
 Veracity
 Visualization
 Value

3. 3
By BRUCE BARTLETT, The Fiscal Times
June 14, 2013
Ever since former CIA employee Edward Snowden leaked information about the top secret PRISM
program – a government system for monitoring vast amounts of electronic data – people have been
asking what, exactly, the government does with all that data.
Read more at http://www.thefiscaltimes.com/Columns/2013/06/14/Is-PRISMs-Big-Data-about-Big-
Money.aspx#SuEj0mSZMJuXXFhw.99
NSA PRISM

4. 4
APPLICATIONS……

5. 5
•Big Data in Dodd Frank reporting-
•5 years data to be saved, USI, LEI, UPI, UCI – no format.
•Reproduce the data in specific time for authorities
•How do you report to DCM, SEF, DCO
•Different reporting formats for SEC, CFTC, FED and newly formed
•Data to be sent to SDRs, SEF in different formats and fields
•FATCA
Dodd-Frank Act… Volcker… FATCA

6. 6
Insights for Valuation
Building an investment recommendation platform
Make investment recommendations and investment decisions
Twitter and FB feeds, Hoovers online
Equity /Bond research houses
Kiplinger, the street, finviz, smartmoney, seeking alpha
Bloomberg, Reuters, Telekurs, Telerate, Markit,
Used Tableau, Pentaho, Platfora, Acunu, Elastic Search for
filtering, heat maps, Pareto chart, cascading filters and co-
relations. Find the outliers. Recommend to hedge funds in
real-time. Used Drill.

7. 7
Trading, Risk, Regulatory Reporting
Discovery-to-Decision Making using operational insights with
minimal latency via visualization
This program is a convergence of BIG data, data discovery, business
intelligence and analytics.
Implement a common trade and asset representation across all asset
classes and functions.
It includes end-to-end trade capture through risk management to the
subledger as a “Single Source of Truth”.
Architecting, Designing and Implementation using data collection, Hadoop,
messaging, algorithms, analytics and visualization technologies.
The END GOAL…

8. 8
Too Many/MUCH of EVERYTHING… BOMBARDMENT???
Too many Databases, too many technologies, too many tools,
too many analytics methods…
??
???

9. 9
SO WHERE DO YOU START
Technical Requirements
Consider the Cloud
What hardware you need
*Master Node and Secondary Master Node
*Slave Nodes
*Network, RAM, CPU, Application server, Power and utility costs etc
What software will you need
Unix system (Redhat, Ubuntu, CentOS etc.)
JVM or JRE
Apache Hadoop (packaged version from Cloudera, MapR,
Hortonworks, Big Insights or plain vanilla
NOSQL and Columnar database (from among the 150 odd) –
Cassandra, MongoDB, Accumulo, Riak, neo4J, hypergraphDB,
orientDB,
Analytics DB– Teradata, netezza, greenplum etc
MySQL database – for queries
Analytics – Descriptive, predictive, prescriptive

10. 10
SO YOU Decided… NOW WHAT….
Scoping your Big DATA Engagement
Identify the use case – Proof of Value
Identify the team
*You need senior management support – someone powerful
*Decision makers and Business owner, budget
*Line of Business- PM, Data owner, SME
*Tech team-infrastructure head, hadoop admin, security, DB team, BA, PM, architect, developer,
QA
Identify the data sources
Size the H/W, Cluster, Cloud, Replication etc.
Stakeholders buy-in, project plan, evangelize, risk assessment
Deploy –H/W, S/W, network, monitoring –Ganglia or Nagios, security, DB, BCP
Collect Data – from data sources, connectors, push or pull, data aggregation, integration,
Visualization -Use various tools or build your own(Make/buy), annotations, extensible, ease of use
Analytics – Descriptive, predictive, prescriptive, A/B
Deepen Insights- Go-live, Find outliers, drill deeper, iterations, interpret root causes, validate results
Measure ROI – trends, performance, ROO, RONS

11. Example Problem: Marketing Campaign
 Jane is an analyst at an e-
commerce company
 How does she figure out good
targeting segments for the next
marketing campaign?
 She has some ideas… …and
lots of data
User
profiles
Transaction
information
Access
logs

12. Traditional System Solution 1: RDBMS
 ETL the data from
MongoDB and
Hadoop into the
RDBMS
– MongoDB data must
be flattened,
schematized, filtered
and aggregated
– Hadoop data must be
filtered and aggregated
 Query the data
using any SQL-
based tool
User
profiles
Access
logs
Transaction
information

13. Traditional System Solution 2: Hadoop
 ETL the data from
Oracle and MongoDB
into Hadoop
– MongoDB data must be
flattened and
schematized
 Work with the
MapReduce team to
write custom code to
generate the desired
analyses
User
profiles
Access
logs
Transaction
information

14. Traditional System Solution 3: Hive
 ETL the data from
Oracle and MongoDB
into Hadoop
– MongoDB data must be
flattened and
schematized
 But HiveQL queries
are slow and BI tool
support is limited
– Marshaling/Coding
User
profiles
Access
logs
Transaction
information

15. What Would Google Do?
Distributed
File System
Batch
processing
Interactive
analysis
NoSQL
GFS MapReduce Dremel BigTable
HDFS
Hadoop
MapReduce
??? HBase
Build Apache Drill to provide a true open
source solution to interactive analysis of Big
Data

16. Why Apache Drill Will Be Successful
Resources
• Contributors have
strong backgrounds
from companies like
Oracle, IBM Netezza,
Informatica, Clustrix
and Pentaho
Community
• Development done in
the open
• Active contributors
from multiple
companies
• Rapidly growing
Architecture
• Full SQL
• New data support
• Extensible APIs
• Full Columnar
Execution
• Beyond Hadoop
Bottom Line: Apache Drill enables NoSQL and SQL Work
Side-by-Side to Tackle Real-time Big Data Needs

17. 17
Vertical and Horizontal Domains

18. 18
Some Current Applications
Some Current Applications
Vertical Refine Explore Enrich
Retail & Web
• Log Analysis
• Ad Optimization
• Cross Channel Analytics
• Social Network Analysis
• Event Analytics
• Dynamic Pricing
• Session & Content
Optimization
• Recommendation Engines
Retail
• Loyalty Program
Optimization
• Brand and Sentiment
Analysis
• Dynamic Pricing/Targeted
Offer
Intelligence • Threat Identification • Person of Interest Discovery • Cross Jurisdiction Queries
Finance
• Risk Modeling & Fraud
Identification
• Trade Performance
Analytics
• Surveillance and Fraud
Detection
• Customer Risk Analysis
• Real-time upsell, cross
sales marketing offers
Energy
• Smart Grid: Production
Optimization
• Grid Failure Prevention
• Smart Meters
• Individual Power Grid
Manufacturing • Supply Chain Optimization • Customer Churn Analysis
• Dynamic Delivery
• Replacement parts
Healthcare&
Payer
• Electronic Medical Records
(EMPI)
• Clinical Trials Analysis
• Supply Chain Optimizations
• Insurance Premium
Determination
Page 2

19. 19
APPENDIX……

20. 20
Retail Banking
Consumer Lending/
Card Services
Commercial Banking
Investment Banking/
Asset Management
Insurance
Market Mix Modeling
Acquisition and
Behavioral
Scorecards
Credit Risk
Management
Asset Liability
Matching
Pure Premium
Modeling
Customer
Satisfaction &
Experience
Collections
Management
Stress Testing and
Scenario Analysis
Portfolio
Optimization
Price Optimization
Management
Credit Risk
Management
Market Research
Analytics
Market Research
Analytics
Interest Rate Risk
Management
Catastrophe
Management &
Reinsurance
Channel
Optimization
Call Center
Optimization
Daily/Weekly Risk
Measurement and
Reporting
VaR Modeling Loss Modeling
Portfolio Stress
Testing
Web Analytics
BASEL Compliance /
Monitoring
Regulatory
Compliance
Agency Incentive
Compensation
Customer Lifetime
Value
Fraud Management
Tools
Pricing Collateralized
Debt
Lapsation Modeling
Cross-sell Targeting
Banking

21. 21
Customer
Acquisition
CRM Marketing
Risk
Management
Operations
Fraud Control &
Collection
Card Services
Product Feature
Selection
Customer
Life Time Value
Market
Profiling &
Sizing
Behavioral
Scorecard
Staffing
Analysis
Collections
Scorecard
Merchant
Performance
Scorecard
Product Pricing
Offer
Optimization /
Customization
Market Mix
Optimization
Risk Based
Pricing
ATM/Branch
Positioning
Delinquency
and Roll
Rate
Analysis
Merchant Fraud
Modeling
Acquisition
Scorecard
Customer
Loyalty
Tracking
Market Spend
Optimization
Credit / Market
Risk Evaluation
Financial
Reporting
and
Analysis
Payoff /
Foreclosure
Analysis
Merchant
Acquisition
Cross Sell
Response
Models
Churn / Attrition
Analytics
Promotion
Effectiveness
Portfolio Health
Tracker
Channel
Process
Management
Self-cure
Propensity
Analysis
Merchant
Retention
Analysis
Segmentation
and Targeting
Customer
Experience /
Value
Customer /
Brand Equity
Portfolio Stress
Testing
Cross-channel
Synergies
Recovery
Maximization
Call Center
Optimization
Contact
Strategies
Optimization
Survey Score
Analysis
Product
Positioning
Capital
Allocation
Investment
Planning
Fraud
Pattern
Recognition
Customer
Behaviour
Analysis
KPI
Measurement &
Reporting
Reactivation /
Silent
Attrition
Key Driver /
Trigger
Analysis
Model
Risk
Management
Asset
Liability
Management
Fraud
Detection /
Investigation
Framework
Customer
Lifetime Value
Retail Banking

22. 22
Risk Management Research
Data Management &
Performance Reporting
Marketing & CRM
Portfolio Stress Testing &
Risk Assessment
End-to-end Market / Equity
Research & Opportunity
Identification
Data Aggregation and
Quality Assessment
Customer Segmentation
and Analysis
Fraud Analysis
Financial Analysis of Target
Companies for M&A
Performance Analytics and
Reporting
Segment Performance and
Reporting
Optimal Asset Allocation
Strategy
Financial Analysis of
Assets and Portfolio
Interactive Dashboards
Segment P&L Analysis and
Forecasting
Servicing Rights Valuation
Market and Industry
Specific Analysis and
Reporting
Reporting and Analysis
Support for End Customers
Cross-sell / Up-sell
Strategy
Cash Flow Modeling And
Forecasting
Financial and Compliance
Reporting
Driver Analysis for
Customer Satisfaction
Instrument Valuation /
Pricing
Issue Resolution
Workflows
Corporate Banking

23. 23
Investment Banking

24. 24
Trade
Services
Data
Management
Record
Keeping
Corp. Actions
Risk
Management
Reconciliation Compliance Reporting
Slippage
costs
Pricing data
analytics
Tax reclaims
Event
monitoring
VaR modeling Failed trades
Trade
surveillance
Executive
reports
Security
selection
Reference
data
Managed
accounts
Unusual trade
activity
Asset class
concentration
Claims review
Exeption
management
Trade reports
Execution
quality
Market data
analytics
Cash
Management
Competitive
intelligence
Inventory
management
Cash
movement
Do Not trade
lists
Ad Hocs
reporting
Trade sizing
decisions
Trade data
analysis
Cross selling
opportunities
Risk
attribution
models
Event driven
monitoring
Performance
measurement
Value add across the entire range of services
Predictive
work
Reporting
solutions
Tax mgmt. Trade support
Modal
validation
Handling big
data sets
Support
services
Cost effective
reporting
dashboards
Handling big
data sets
Dashboards
Architecture
setup
Cleaning
functions
Inquisitive
analytics
Reporting
Text mining
solutions
Performance
measurement
Data
monetization
Scalability
solutions
Monitoring
systems
Internal fraud
monitoring
Claims
handling
Investment Banking Services

25. 25
Pricing / Risk
Management
Marketing
Distribution
Channels
Underwriting
Claims
Management
Investment
Management
Corporate
Loss Modeling
RFM Direct
Response
Marketing
Product
Selection
Fraud
Detection
Fraud
Detection
Early
Reinsurance
Recoverable
Tagging
Yield
Management
Pure Premium
Models
Campaign
Reporting
Productivity
Automated
Underwriting
Severity
Forecasting
Reinsurance
Optimization
Hedging
Strategies
Competitive
Market Analysis
Market
Planning
Production
Forecasting
Straight
through
processing
Claims Staffing
Optimization
Asset Liability
Matching
Price
Optimization
Management
Market Mix
Modeling
Goal-Setting
Expense
Management
Productivity
Analysis
Portfolio
Optimization
Class Plan
Development
Advertising Lift
Measurement
Agency
Segmentation
Expense
Management
Customer
Lifetime Value
Performance
Measurement
Loss Cost
Driver Analysis
Customer
Satisfaction &
Experience
analytics
Claims
Customer
Satisfaction
Analysis
Insurance