SlideShare a Scribd company logo
1 of 31
Download to read offline
Big Data Application Architectures - Fraud Detection
Agenda
Define the problem
Establish the expected outcome
Dive into each pillar
Determine a Solution
Understand the applicability
Financial
Institutions risk
Loss of
Charterand a host of other penalties through
noncompliance with federal money
laundering legislation.
Big Data Evolution
Legacy Systems Current Systems
Big Data
Advanced Analytics
Timely Info Accurate Thoughtful
Marketing Operations Bankers CEOs
• Next Best Action
• Recommended Interventions
• Lifestyle Yield Management
• Seasonal Personal Impact
• Theft Profiling
• Fraudulent Transaction
Identification
• Remote Shutdown
• Site Monitoring
• Recommended Interventions
• Risky Customer Profiling
• Call Center Monitoring
• Churn Scoring
• Payment System Errors
• Money Laundering
prevention
• Compliance
• Data Entry Intervention
?
Personalization of offers &
banking experience
Risk Reduction &
ComplianceCustomer Churn PreventionFraud Detection
Areas of Opportunity for Financial Analytics
Expected Outcome
$
Big Data
Challenges
Architectural Considerations
Fraud Detection Reference Architecture
Apps data
from devices
News and
other alerts
Solution UX
Provisioning API (Pull)
User Profile Information
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
User Recent Activity Store
Gateway
Data Lake
Gateway
App Backend
Data Path
Optional solution component
Main solution component
Thin Client
Presentation & Business ConnectivityData Processing, Analytics and ManagementDevice Connectivity
Personal
mobile
devices
Trades
and/or
transactions
Business
systems
Reference Architecture with Azure Services
Solution UX
Provisioning API
User Profile Information
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
User Recent Activity Store Store
Data Lake
Gateway
App Backend
Personal
mobile
devices
Business
systems
Presentation & Business ConnectivityData Processing, Analytics and ManagementDevice Connectivity
Apps data
from devices
News and
other alerts
Gateway
Data Path
Optional solution component
Main solution component
Thin Client
Trades
and/or
transactions
Demo
Woodgrove Financial
User Profile and Metadata Stores
App Backend Solution UX
Provisioning API
User Profile Information
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
User Recent Activity Information
Data Lake
Gateway
(Kafka,
IoT Hub,
Event Hubs)
Data Path
Optional solution component
Main solution component
Metadata
Store
Gateway
Trades
and/or
transactions
Thin Client
News and
other alerts
Apps data
from
devices
Device Identity, Registry and State Stores
Metadata Store
Authority for all registered sources
Stores identity information and authentication secrets
User Profile Information
Indexed list of all Users and their demographics – Secure, Governed, Audit Controlled
Contains discovery and reference data related to Users
Can define a schema model or use a vertical industry standard schema for metadata
Can contain structured metadata and links to externally stored operational data
User Recent Activity
Contains operational data related to the Users’ most recent activities:
- “Last known values” for each User
- Aggregated or computed values
- Stream of device data events containing Geo location and Time based tagging
Stream Processors
App Backend Solution UX
Provisioning API
Identity and Registry Stores
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
Device State Store
Data Lake
Data Path
Optional solution component
Main solution component
Gateway
Trades
and/or
transactions
Thin Client
News and
other alerts
Apps data
from
devices
Stream Processing: Data Flow
After ingress through the Gateway (Ingestion), the flow of data
through the system is facilitated by data pumps and analytics tasks
Data flow can be driven by:
• Apache Storm on Azure HDInsight
• Apache Spark on Azure HDInsight
• Azure Stream Analytics
• Custom Event Processors
Each can perform tasks
in flight:
• Data aggregation
• Data enrichment
• Complex event processing
… and can output data
to:
• Azure Data Lake
• Azure Blobs/Tables
• HDInsight / HBase
• Azure SQL DB
• Time Series Databases
• Event Hub
• Service Bus Queues
Stream Processor Examples
Queue
Device Registry Store
Device Metadata
Processor
Data Lake
Device State Store
Device State
Processor
Notification
Processor
Raw Telemetry Processor
App Backend
Rules Processor
Event Hub
Stream Transformation
Processor
Secondary Stream
Processor
Data Path
Optional solution component
Main solution component
Gateway
Trades
and/or
transactions
Thin Client
News and
other alerts
Apps data
from
devices
App Backend
App Backend Solution UX
Provisioning API
Identity and Registry Stores
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
Device State Store
Storage
Cloud
Gateway
Data Path
Optional solution component
Main solution component
Gateway
Trades
and/or
transactions
Thin Client
News and
other alerts
Apps data
from
devices
High-Scale Compute Models
Scale-appropriate compute models
Actor Frameworks / Service Fabric Reliable Actors: distributed
compute fabric hosting device actors.
Service Fabric Reliable Collections: highly available with
replicated and local state management.
Azure Batch: job scheduling and compute management for
highly parallelizable compute workloads.
Simple programming logic in vastly scalable
compute nodes
Data Analytics
App Backend Solution UX
Provisioning API
Identity and Registry Stores
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
Device State Store
Data Lake
Cloud
Gateway
Data Path
Optional solution component
Main solution component
Gateway
Trades
and/or
transactions
Thin Client
News and
other alerts
Apps data
from
devices
Data Analytics
Event Hub
NRT Events
Stream Processing
(ASA, Storm or
Spark)
Alerts
Batch Events
Fetching &
Updating
Reference Data
Interceptor (Rules)
Spark
Hive/Pig
U-SQL
Azure Data Lake Store Azure Data Lake Analytics
SQL DB
ML
Reports and
Dashboards
Real Time Scoring
Training ML Models
Relational Data
Data Analytics
Real-Time Analysis
Aggregation/Reduction, Temporal Queries, State
Correlation, Threshold Detection, Alerting
Data-At-Rest Analysis
Time-Series, Map/Reduce, Correlation
Machine Learning
Pattern Detection, Behavior Prediction
Plausibility Analysis, Anomaly and Fraud Detection
Power BI
HDInsight
Stream Analytics
Data Factory
Machine Learning
Presentation and Business Connectivity
App Backend Solution UX
Provisioning API
Identity and Registry Stores
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
Device State Store
Data Lake
Cloud
Gateway
Data Path
Optional solution component
Main solution component
Gateway
Trades
and/or
transactions
Thin Client
News and
other alerts
Apps data
from
devices
WebHDFS
YARN
U-SQL
Analytics Service HDInsight
(managed Hadoop Clusters)
Analytics
Store
Azure Data Lake
Cortana Intelligence Suite
Action
People
Automated
Systems
Apps
Web
Mobile
Bots
Intelligence
Dashboards &
Visualizations
Cortana
Bot
Framework
Cognitive
Services
Power BI
Information
Management
Event Hubs
Data Catalog
Data Factory
Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream Analytics
Intelligence
Data Lake
Analytics
Machine
Learning
Big Data Stores
SQL Data
Warehouse
Data Lake Store
Data
Sources
Apps
Sensors
and
devices
Data
Reference Architecture with Azure Services
Solution UX
Provisioning API
User Profile Information
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
User Recent Activity Store Store
Data Lake
Gateway
App Backend
Personal
mobile
devices
Business
systems
Presentation & Business ConnectivityData Processing, Analytics and ManagementDevice Connectivity
Apps data
from devices
News and
other alerts
Gateway
Data Path
Optional solution component
Main solution component
Thin Client
Trades
and/or
transactions
Money Laundering Prevention
Fraud Detection
$ $ $
¥
Placement Layering Integration
Process
Know your
Customer
Transaction
Monitoring
Pattern
Detection
Machine Learning
Decision Tree Classification
Cluster
Analysis
Cloud
Anti-Money Laundering
Power BI
Fund monitoring
dashboard
Big Data Storage for
Multiple Sources
HDInsight Azure Data
Lake
Azure Data
Warehouse
SQL Azure Azure Machine
Learning
SQL
Financial Data
Real-time fraud detection feedback
Information Services
HDInsight Streaming
Analytics
Data Science Modeling
• Similar to linear regression
• Weights independent variables
• Useful with categorical
independent variable
• Offers coefficients to inform
management decision-making
• Very useful with internal
analytical teams to interpret
data
• Useful for diagnosing gaps in
data and customer outreach
• Helps drive understanding of
demand drivers
• Uses decision trees & votes
• Forest
• Compares results between
various outcomes
• Votes upon outcomes
• Evaluates based upon a
series of logical questions or
“forest”
• Jungle
• Useful when a forest
produces too many logical
branches
• Produces a series of weighted
edges and nodes
• Trained in input data
• Useful for complex tasks, like
speech recognition when
allowed to train in depth
• Very good with complex
interactions
• Enables retailers to better
identify behaviour patterns &
certain shopping activities
Reference Architecture & Azure Services
Solution UX
Provisioning API
User Profile Information
Stream Processors
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
User Recent Activity Store Store
Data Lake
Gateway
App Backend
Personal
mobile
devices
Business
systems
Presentation & Business ConnectivityData Processing, Analytics and ManagementDevice Connectivity
Apps data
from devices
News and
other alerts
Gateway
Data Path
Optional solution component
Main solution component
Thin Client
Trades
and/or
transactions
nishant.thacker@microsoft.com
© 2016 Microsoft Corporation. All rights reserved.

More Related Content

What's hot

Big Data Architecture and Design Patterns
Big Data Architecture and Design PatternsBig Data Architecture and Design Patterns
Big Data Architecture and Design PatternsJohn Yeung
 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouseJames Serra
 
AI powered decision making in banks
AI powered decision making in banksAI powered decision making in banks
AI powered decision making in banksPankaj Baid
 
Dataiku & Snowflake Meetup Berlin 2020
Dataiku & Snowflake Meetup Berlin 2020Dataiku & Snowflake Meetup Berlin 2020
Dataiku & Snowflake Meetup Berlin 2020Harald Erb
 
Business Intelligence, Portals, Dashboards and Operational Matrix with ShareP...
Business Intelligence, Portals, Dashboards and Operational Matrix with ShareP...Business Intelligence, Portals, Dashboards and Operational Matrix with ShareP...
Business Intelligence, Portals, Dashboards and Operational Matrix with ShareP...Optimus BT
 
Hadoop and Manufacturing
Hadoop and ManufacturingHadoop and Manufacturing
Hadoop and ManufacturingCloudera, Inc.
 
Self Service Analytics and a Modern Data Architecture with Data Virtualizatio...
Self Service Analytics and a Modern Data Architecture with Data Virtualizatio...Self Service Analytics and a Modern Data Architecture with Data Virtualizatio...
Self Service Analytics and a Modern Data Architecture with Data Virtualizatio...Denodo
 
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Cloudera, Inc.
 
SQL Azure the database in the cloud
SQL Azure the database in the cloud SQL Azure the database in the cloud
SQL Azure the database in the cloud Eduardo Castro
 
Demystifying Data Warehousing as a Service - DFW
Demystifying Data Warehousing as a Service - DFWDemystifying Data Warehousing as a Service - DFW
Demystifying Data Warehousing as a Service - DFWKent Graziano
 
Modern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-Baltagi
Modern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-BaltagiModern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-Baltagi
Modern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-BaltagiSlim Baltagi
 
Enterprise Data Management
Enterprise Data ManagementEnterprise Data Management
Enterprise Data ManagementBhavendra Chavan
 
Introduction to Customer Data Platforms
Introduction to Customer Data PlatformsIntroduction to Customer Data Platforms
Introduction to Customer Data PlatformsTreasure Data, Inc.
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureJames Serra
 
The Customer Data Platform, the Future of the Marketing Database
The Customer Data Platform, the Future of the Marketing DatabaseThe Customer Data Platform, the Future of the Marketing Database
The Customer Data Platform, the Future of the Marketing DatabaseRedEye
 

What's hot (20)

Data analytics
Data analyticsData analytics
Data analytics
 
Big Data Architecture and Design Patterns
Big Data Architecture and Design PatternsBig Data Architecture and Design Patterns
Big Data Architecture and Design Patterns
 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouse
 
Presentation on Big Data
Presentation on Big DataPresentation on Big Data
Presentation on Big Data
 
AI powered decision making in banks
AI powered decision making in banksAI powered decision making in banks
AI powered decision making in banks
 
Data Warehousing
Data WarehousingData Warehousing
Data Warehousing
 
Dataiku & Snowflake Meetup Berlin 2020
Dataiku & Snowflake Meetup Berlin 2020Dataiku & Snowflake Meetup Berlin 2020
Dataiku & Snowflake Meetup Berlin 2020
 
Business Intelligence, Portals, Dashboards and Operational Matrix with ShareP...
Business Intelligence, Portals, Dashboards and Operational Matrix with ShareP...Business Intelligence, Portals, Dashboards and Operational Matrix with ShareP...
Business Intelligence, Portals, Dashboards and Operational Matrix with ShareP...
 
Hadoop and Manufacturing
Hadoop and ManufacturingHadoop and Manufacturing
Hadoop and Manufacturing
 
Self Service Analytics and a Modern Data Architecture with Data Virtualizatio...
Self Service Analytics and a Modern Data Architecture with Data Virtualizatio...Self Service Analytics and a Modern Data Architecture with Data Virtualizatio...
Self Service Analytics and a Modern Data Architecture with Data Virtualizatio...
 
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19
 
SQL Azure the database in the cloud
SQL Azure the database in the cloud SQL Azure the database in the cloud
SQL Azure the database in the cloud
 
Demystifying Data Warehousing as a Service - DFW
Demystifying Data Warehousing as a Service - DFWDemystifying Data Warehousing as a Service - DFW
Demystifying Data Warehousing as a Service - DFW
 
Modern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-Baltagi
Modern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-BaltagiModern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-Baltagi
Modern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-Baltagi
 
Overview of Big data(ppt)
Overview of Big data(ppt)Overview of Big data(ppt)
Overview of Big data(ppt)
 
Enterprise Data Management
Enterprise Data ManagementEnterprise Data Management
Enterprise Data Management
 
Introduction to Customer Data Platforms
Introduction to Customer Data PlatformsIntroduction to Customer Data Platforms
Introduction to Customer Data Platforms
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse Architecture
 
The Customer Data Platform, the Future of the Marketing Database
The Customer Data Platform, the Future of the Marketing DatabaseThe Customer Data Platform, the Future of the Marketing Database
The Customer Data Platform, the Future of the Marketing Database
 
What is ETL?
What is ETL?What is ETL?
What is ETL?
 

Similar to Big Data Application Architectures - Fraud Detection

WebAction In-Memory Computing Summit 2015
WebAction In-Memory Computing Summit 2015WebAction In-Memory Computing Summit 2015
WebAction In-Memory Computing Summit 2015WebAction
 
50 Shades of Data - Dutch Oracle Architects Platform (February 2018)
50 Shades of Data - Dutch Oracle Architects Platform (February 2018)50 Shades of Data - Dutch Oracle Architects Platform (February 2018)
50 Shades of Data - Dutch Oracle Architects Platform (February 2018)Lucas Jellema
 
A Winning Strategy for the Digital Economy
A Winning Strategy for the Digital EconomyA Winning Strategy for the Digital Economy
A Winning Strategy for the Digital EconomyEric Kavanagh
 
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...AgileNetwork
 
Microsoft SQL Server 2008 R2 and BizTalk Server Presentation
Microsoft SQL Server 2008 R2 and BizTalk Server PresentationMicrosoft SQL Server 2008 R2 and BizTalk Server Presentation
Microsoft SQL Server 2008 R2 and BizTalk Server PresentationMicrosoft Private Cloud
 
The 4th Generation Kingland platform
The 4th Generation Kingland platformThe 4th Generation Kingland platform
The 4th Generation Kingland platformKingland
 
Big Data Analytics Webinar
Big Data Analytics WebinarBig Data Analytics Webinar
Big Data Analytics WebinarEckerson Group
 
Event Driven Architecture (EDA), November 2, 2006
Event Driven Architecture (EDA), November 2, 2006Event Driven Architecture (EDA), November 2, 2006
Event Driven Architecture (EDA), November 2, 2006Tim Bass
 
Overview of Composable SaaS Models
Overview of Composable SaaS ModelsOverview of Composable SaaS Models
Overview of Composable SaaS ModelsGabe Pei
 
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSetsEnabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSetsStreamsets Inc.
 
Analytics in Your Enterprise
Analytics in Your EnterpriseAnalytics in Your Enterprise
Analytics in Your EnterpriseWSO2
 
Relevant Pension Portalv4
Relevant Pension Portalv4Relevant Pension Portalv4
Relevant Pension Portalv4ebstlr
 
Business Analytics Paradigm Change
Business Analytics Paradigm ChangeBusiness Analytics Paradigm Change
Business Analytics Paradigm ChangeDmitry Anoshin
 
Incentius - Portfolio of Capabilities
Incentius - Portfolio of CapabilitiesIncentius - Portfolio of Capabilities
Incentius - Portfolio of CapabilitiesSujeet Pillai
 
Big Data on AWS - Toronto FSI Symposium - October 2016
Big Data on AWS - Toronto FSI Symposium - October 2016Big Data on AWS - Toronto FSI Symposium - October 2016
Big Data on AWS - Toronto FSI Symposium - October 2016Amazon Web Services
 

Similar to Big Data Application Architectures - Fraud Detection (20)

WebAction-Sami Abkay
WebAction-Sami AbkayWebAction-Sami Abkay
WebAction-Sami Abkay
 
WebAction In-Memory Computing Summit 2015
WebAction In-Memory Computing Summit 2015WebAction In-Memory Computing Summit 2015
WebAction In-Memory Computing Summit 2015
 
50 Shades of Data - Dutch Oracle Architects Platform (February 2018)
50 Shades of Data - Dutch Oracle Architects Platform (February 2018)50 Shades of Data - Dutch Oracle Architects Platform (February 2018)
50 Shades of Data - Dutch Oracle Architects Platform (February 2018)
 
A Winning Strategy for the Digital Economy
A Winning Strategy for the Digital EconomyA Winning Strategy for the Digital Economy
A Winning Strategy for the Digital Economy
 
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
 
Microsoft SQL Server 2008 R2 and BizTalk Server Presentation
Microsoft SQL Server 2008 R2 and BizTalk Server PresentationMicrosoft SQL Server 2008 R2 and BizTalk Server Presentation
Microsoft SQL Server 2008 R2 and BizTalk Server Presentation
 
Big Data Application Architectures - IoT
Big Data Application Architectures - IoTBig Data Application Architectures - IoT
Big Data Application Architectures - IoT
 
The 4th Generation Kingland platform
The 4th Generation Kingland platformThe 4th Generation Kingland platform
The 4th Generation Kingland platform
 
Big Data Analytics Webinar
Big Data Analytics WebinarBig Data Analytics Webinar
Big Data Analytics Webinar
 
Event Driven Architecture (EDA), November 2, 2006
Event Driven Architecture (EDA), November 2, 2006Event Driven Architecture (EDA), November 2, 2006
Event Driven Architecture (EDA), November 2, 2006
 
Azure IoT Suite
Azure IoT Suite Azure IoT Suite
Azure IoT Suite
 
Machine Data Analytics
Machine Data AnalyticsMachine Data Analytics
Machine Data Analytics
 
Overview of Composable SaaS Models
Overview of Composable SaaS ModelsOverview of Composable SaaS Models
Overview of Composable SaaS Models
 
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSetsEnabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
 
Analytics in Your Enterprise
Analytics in Your EnterpriseAnalytics in Your Enterprise
Analytics in Your Enterprise
 
Relevant Pension Portalv4
Relevant Pension Portalv4Relevant Pension Portalv4
Relevant Pension Portalv4
 
Business Analytics Paradigm Change
Business Analytics Paradigm ChangeBusiness Analytics Paradigm Change
Business Analytics Paradigm Change
 
Incentius - Portfolio of Capabilities
Incentius - Portfolio of CapabilitiesIncentius - Portfolio of Capabilities
Incentius - Portfolio of Capabilities
 
Big Data on AWS - Toronto FSI Symposium - October 2016
Big Data on AWS - Toronto FSI Symposium - October 2016Big Data on AWS - Toronto FSI Symposium - October 2016
Big Data on AWS - Toronto FSI Symposium - October 2016
 
Technologies
TechnologiesTechnologies
Technologies
 

More from DataWorks Summit/Hadoop Summit

Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache RangerUnleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache RangerDataWorks Summit/Hadoop Summit
 
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science PlatformEnabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science PlatformDataWorks Summit/Hadoop Summit
 
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSenseDouble Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSenseDataWorks Summit/Hadoop Summit
 
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit
 
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit
 
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and MLMool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and MLDataWorks Summit/Hadoop Summit
 
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit
 
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit
 

More from DataWorks Summit/Hadoop Summit (20)

Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in ProductionRunning Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in Production
 
State of Security: Apache Spark & Apache Zeppelin
State of Security: Apache Spark & Apache ZeppelinState of Security: Apache Spark & Apache Zeppelin
State of Security: Apache Spark & Apache Zeppelin
 
Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache RangerUnleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache Ranger
 
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science PlatformEnabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science Platform
 
Revolutionize Text Mining with Spark and Zeppelin
Revolutionize Text Mining with Spark and ZeppelinRevolutionize Text Mining with Spark and Zeppelin
Revolutionize Text Mining with Spark and Zeppelin
 
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSenseDouble Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSense
 
Hadoop Crash Course
Hadoop Crash CourseHadoop Crash Course
Hadoop Crash Course
 
Data Science Crash Course
Data Science Crash CourseData Science Crash Course
Data Science Crash Course
 
Apache Spark Crash Course
Apache Spark Crash CourseApache Spark Crash Course
Apache Spark Crash Course
 
Dataflow with Apache NiFi
Dataflow with Apache NiFiDataflow with Apache NiFi
Dataflow with Apache NiFi
 
Schema Registry - Set you Data Free
Schema Registry - Set you Data FreeSchema Registry - Set you Data Free
Schema Registry - Set you Data Free
 
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
 
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
 
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and MLMool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and ML
 
How Hadoop Makes the Natixis Pack More Efficient
How Hadoop Makes the Natixis Pack More Efficient How Hadoop Makes the Natixis Pack More Efficient
How Hadoop Makes the Natixis Pack More Efficient
 
HBase in Practice
HBase in Practice HBase in Practice
HBase in Practice
 
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)
 
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS HadoopBreaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
 
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
 
Backup and Disaster Recovery in Hadoop
Backup and Disaster Recovery in Hadoop Backup and Disaster Recovery in Hadoop
Backup and Disaster Recovery in Hadoop
 

Recently uploaded

Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 

Recently uploaded (20)

Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 

Big Data Application Architectures - Fraud Detection

  • 2. Agenda Define the problem Establish the expected outcome Dive into each pillar Determine a Solution Understand the applicability
  • 3. Financial Institutions risk Loss of Charterand a host of other penalties through noncompliance with federal money laundering legislation.
  • 4. Big Data Evolution Legacy Systems Current Systems Big Data Advanced Analytics Timely Info Accurate Thoughtful
  • 5. Marketing Operations Bankers CEOs • Next Best Action • Recommended Interventions • Lifestyle Yield Management • Seasonal Personal Impact • Theft Profiling • Fraudulent Transaction Identification • Remote Shutdown • Site Monitoring • Recommended Interventions • Risky Customer Profiling • Call Center Monitoring • Churn Scoring • Payment System Errors • Money Laundering prevention • Compliance • Data Entry Intervention ? Personalization of offers & banking experience Risk Reduction & ComplianceCustomer Churn PreventionFraud Detection Areas of Opportunity for Financial Analytics
  • 9. Fraud Detection Reference Architecture Apps data from devices News and other alerts Solution UX Provisioning API (Pull) User Profile Information Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) User Recent Activity Store Gateway Data Lake Gateway App Backend Data Path Optional solution component Main solution component Thin Client Presentation & Business ConnectivityData Processing, Analytics and ManagementDevice Connectivity Personal mobile devices Trades and/or transactions Business systems
  • 10. Reference Architecture with Azure Services Solution UX Provisioning API User Profile Information Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) User Recent Activity Store Store Data Lake Gateway App Backend Personal mobile devices Business systems Presentation & Business ConnectivityData Processing, Analytics and ManagementDevice Connectivity Apps data from devices News and other alerts Gateway Data Path Optional solution component Main solution component Thin Client Trades and/or transactions
  • 12. User Profile and Metadata Stores App Backend Solution UX Provisioning API User Profile Information Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) User Recent Activity Information Data Lake Gateway (Kafka, IoT Hub, Event Hubs) Data Path Optional solution component Main solution component Metadata Store Gateway Trades and/or transactions Thin Client News and other alerts Apps data from devices
  • 13. Device Identity, Registry and State Stores Metadata Store Authority for all registered sources Stores identity information and authentication secrets User Profile Information Indexed list of all Users and their demographics – Secure, Governed, Audit Controlled Contains discovery and reference data related to Users Can define a schema model or use a vertical industry standard schema for metadata Can contain structured metadata and links to externally stored operational data User Recent Activity Contains operational data related to the Users’ most recent activities: - “Last known values” for each User - Aggregated or computed values - Stream of device data events containing Geo location and Time based tagging
  • 14. Stream Processors App Backend Solution UX Provisioning API Identity and Registry Stores Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) Device State Store Data Lake Data Path Optional solution component Main solution component Gateway Trades and/or transactions Thin Client News and other alerts Apps data from devices
  • 15. Stream Processing: Data Flow After ingress through the Gateway (Ingestion), the flow of data through the system is facilitated by data pumps and analytics tasks Data flow can be driven by: • Apache Storm on Azure HDInsight • Apache Spark on Azure HDInsight • Azure Stream Analytics • Custom Event Processors Each can perform tasks in flight: • Data aggregation • Data enrichment • Complex event processing … and can output data to: • Azure Data Lake • Azure Blobs/Tables • HDInsight / HBase • Azure SQL DB • Time Series Databases • Event Hub • Service Bus Queues
  • 16. Stream Processor Examples Queue Device Registry Store Device Metadata Processor Data Lake Device State Store Device State Processor Notification Processor Raw Telemetry Processor App Backend Rules Processor Event Hub Stream Transformation Processor Secondary Stream Processor Data Path Optional solution component Main solution component Gateway Trades and/or transactions Thin Client News and other alerts Apps data from devices
  • 17. App Backend App Backend Solution UX Provisioning API Identity and Registry Stores Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) Device State Store Storage Cloud Gateway Data Path Optional solution component Main solution component Gateway Trades and/or transactions Thin Client News and other alerts Apps data from devices
  • 18. High-Scale Compute Models Scale-appropriate compute models Actor Frameworks / Service Fabric Reliable Actors: distributed compute fabric hosting device actors. Service Fabric Reliable Collections: highly available with replicated and local state management. Azure Batch: job scheduling and compute management for highly parallelizable compute workloads. Simple programming logic in vastly scalable compute nodes
  • 19. Data Analytics App Backend Solution UX Provisioning API Identity and Registry Stores Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) Device State Store Data Lake Cloud Gateway Data Path Optional solution component Main solution component Gateway Trades and/or transactions Thin Client News and other alerts Apps data from devices
  • 20. Data Analytics Event Hub NRT Events Stream Processing (ASA, Storm or Spark) Alerts Batch Events Fetching & Updating Reference Data Interceptor (Rules) Spark Hive/Pig U-SQL Azure Data Lake Store Azure Data Lake Analytics SQL DB ML Reports and Dashboards Real Time Scoring Training ML Models Relational Data
  • 21. Data Analytics Real-Time Analysis Aggregation/Reduction, Temporal Queries, State Correlation, Threshold Detection, Alerting Data-At-Rest Analysis Time-Series, Map/Reduce, Correlation Machine Learning Pattern Detection, Behavior Prediction Plausibility Analysis, Anomaly and Fraud Detection Power BI HDInsight Stream Analytics Data Factory Machine Learning
  • 22. Presentation and Business Connectivity App Backend Solution UX Provisioning API Identity and Registry Stores Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) Device State Store Data Lake Cloud Gateway Data Path Optional solution component Main solution component Gateway Trades and/or transactions Thin Client News and other alerts Apps data from devices
  • 23. WebHDFS YARN U-SQL Analytics Service HDInsight (managed Hadoop Clusters) Analytics Store Azure Data Lake
  • 24. Cortana Intelligence Suite Action People Automated Systems Apps Web Mobile Bots Intelligence Dashboards & Visualizations Cortana Bot Framework Cognitive Services Power BI Information Management Event Hubs Data Catalog Data Factory Machine Learning and Analytics HDInsight (Hadoop and Spark) Stream Analytics Intelligence Data Lake Analytics Machine Learning Big Data Stores SQL Data Warehouse Data Lake Store Data Sources Apps Sensors and devices Data
  • 25. Reference Architecture with Azure Services Solution UX Provisioning API User Profile Information Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) User Recent Activity Store Store Data Lake Gateway App Backend Personal mobile devices Business systems Presentation & Business ConnectivityData Processing, Analytics and ManagementDevice Connectivity Apps data from devices News and other alerts Gateway Data Path Optional solution component Main solution component Thin Client Trades and/or transactions
  • 26. Money Laundering Prevention Fraud Detection $ $ $ ¥ Placement Layering Integration Process Know your Customer Transaction Monitoring Pattern Detection Machine Learning Decision Tree Classification Cluster Analysis
  • 27. Cloud Anti-Money Laundering Power BI Fund monitoring dashboard Big Data Storage for Multiple Sources HDInsight Azure Data Lake Azure Data Warehouse SQL Azure Azure Machine Learning SQL Financial Data Real-time fraud detection feedback Information Services HDInsight Streaming Analytics
  • 28. Data Science Modeling • Similar to linear regression • Weights independent variables • Useful with categorical independent variable • Offers coefficients to inform management decision-making • Very useful with internal analytical teams to interpret data • Useful for diagnosing gaps in data and customer outreach • Helps drive understanding of demand drivers • Uses decision trees & votes • Forest • Compares results between various outcomes • Votes upon outcomes • Evaluates based upon a series of logical questions or “forest” • Jungle • Useful when a forest produces too many logical branches • Produces a series of weighted edges and nodes • Trained in input data • Useful for complex tasks, like speech recognition when allowed to train in depth • Very good with complex interactions • Enables retailers to better identify behaviour patterns & certain shopping activities
  • 29. Reference Architecture & Azure Services Solution UX Provisioning API User Profile Information Stream Processors Analytics & Machine Learning Business Integration Connectors and Gateway(s) User Recent Activity Store Store Data Lake Gateway App Backend Personal mobile devices Business systems Presentation & Business ConnectivityData Processing, Analytics and ManagementDevice Connectivity Apps data from devices News and other alerts Gateway Data Path Optional solution component Main solution component Thin Client Trades and/or transactions
  • 31. © 2016 Microsoft Corporation. All rights reserved.

Editor's Notes

  1. Today’s financial services market is highly competitive, complex, and difficult. Particularly with today’s legislation, it is becoming increasingly more important to reduce risk, increase compliance, detect fraud, retain customers, and know your customers better.
  2. Over the course of time, data is evolving. Legacy systems have evolved into the current systems of today. Systems like As systems change and evolve to become more timely, accurate, and thoughtful greater opportunities for return on system investments are realized. Big Data and Advanced Analytics systems offer superior return-on-investment.
  3. A host of opportunities exist to utilize this technology suite in the arena of financial analytics. Left to Right Personalization of offers and tailored banking experiences allow opportunities to engage with customers in a positive way based on their data. Next best action offers surface suspected needs and offer the opportunity for sales lift. Recommended interventions allow for programmatic intervention based upon customer churn. Lifestyle yield management allows for bankers to tailor plans & recommendations based on the life state of customers (retiree versus recent graduate) Many customers of financial institutions are impacted by seasonality in their employment or lifestyle. By recognizing and making offers to these customers based on their needs, banks can increase their profitability. Fraud Detection allows banks to reduce risk and their cost of operations. Theft profiling & fraudulent transaction detection allow for proactive intervention & prevention of fraud. Remote shutdown & site monitoring allow banks to reactively intervene in ATM and physical locations in the event of fraud. Customer Churn Prevention increases revenue by increasing customer lifetime. Churn scoring allows for identification of at-risk customers, and is the basis for all other churn applications. Personalized interventions allow for customized per-customer interventions to be created based upon churn scoring & personalization. Similarly risky customers can be profiled to identify characteristics and intervene. Call center monitoring allows for use of perceptual intelligence to be applied to identify churn behavior based on call center operations. Risk Reduction & Compliance are a key way institutions can reduce operational costs. Prevention of Payment System errors and Money Laundering prevention can substantially reduce risk to fines & lost funds. Data entry is similarly a source of risk; identifying and preventing data entry errors can save time & money.
  4. With the large amounts of data potentially available for analysis, managing data flows efficiently can be a challenge. Huge amounts of data to process (volume) A mixture of structured and unstructured data (variety) New data that’s generated extremely frequently (velocity) Data quality so that it can be trusted (veracity)
  5. Between 2.17 & 3.61 Trillion dollars are laundered annually. The process of detecting and preventing money laundering at a perceptual level is fairly straightforward, but implementation of systems to detect and prevent money laundering are incredibly complex Money laundering has 3 primary processes; placement, layering, & integration. Placement is where funds from illegal activities are introduced to the financial system. Layering is the suite of transactions designed to clean the money. Integration is when funds are redistributed back through business transactions. To prevent this malicious process from happening, process controls can be implemented to prevent money laundering. As you can see, these primarily fall into the 3 categories indicated in process, and can be supported by the various machine learning algorithms mentioned on the right. (Click Again) using Cortana analytics in the process and to drive the machine learning behind money laundering prevention can prevent money laundering.
  6. This diagram shows both the hot path & cold path outlined. The hot path informs directly from the information services layer as data is entered from field services. The cold path involves data storage & use of machine learning to inform hot path development and directly predict into the visualization layer. It is important to stress both the hot path and cold path of the solution here, as both are required to yield superior results. Storm, Spark, & Azure Stream Analytics are the tools useful for the hot path implementations of the rules gleaned from ML. Azure data factory is the orchestration tool. These are referenced again in the Machine Learning layer, as further development here is used to increase hot path value. HDInsight & Azure Data Lake are big data stores. Azure DW and SQL Azure are relational data stores for extracting further value from the big data stores. Azure Machine Learning provides a platform for data evaluation, data science and prediction. This is where the real value for the solution is created.
  7. Data Factory  http://azure.microsoft.com/en-us/services/data-factory/ Data Catalog http://azure.microsoft.com/en-us/services/data-catalog/ Event Hubs http://azure.microsoft.com/en-us/services/event-hubs/ Stream Analytics http://azure.microsoft.com/en-us/services/stream-analytics/
  8. Data Lake http://azure.microsoft.com/en-us/campaigns/data-lake/ SQL Data Warehouse http://azure.microsoft.com/en-us/services/sql-data-warehouse/ HDInsight http://azure.microsoft.com/en-us/services/hdinsight/ Stream Analytics http://azure.microsoft.com/en-us/services/stream-analytics/
  9. Machine Learning https://studio.azureml.net/ HDInsight http://azure.microsoft.com/en-us/services/hdinsight/ Stream Analytics http://azure.microsoft.com/en-us/services/stream-analytics/
  10. Power BI https://powerbi.microsoft.com/ Azure Web Sites http://azure.microsoft.com/en-us/services/app-service/web/