We are in the midst of an exciting time. There is an explosion of very interesting data, and emergence of powerful new technologies for harnessing data, and devices that enable humans to receive tremendous benefits from it. What is required are innovative processes that enable the creation and delivery of value from all of that data. More often than not, it is the predictive (what will happen?) and prescriptive (how to make it happen!) analytics that produces this value, not the raw data itself. Agile software teams are continuously involved in projects that involve rich, complex, and messy data. Often this data represents innovative analytics opportunities. Being analytics-aware gives these teams the opportunity to collaborate with stakeholders to innovate by creating additional value from the data. This session is aimed at making Agile software teams more analytics-aware so that they will recognize these innovation opportunities. The trouble with conventional analytics (like conventional software development) is that it involves long, phased, sequential steps that take too long and fail to deliver actionable results. This deck will examine the convergence of the following elements of an exciting emerging field called Agile Analytics:
sophisticated analytics techniques, plus
lean learning principles, plus
agile delivery methods, plus
so-called "big data" technologies
Learn:
The analytical modeling process and techniques
How analytical models are deployed using modern technologies
The complexities of data discovery, harvesting, and preparation
How to apply agile techniques to shorten the analytics development cycle
How to apply lean learning principles to develop actionable and valuable analytics.
4. THE DIFFERENCE
Data
Engineering
Lean
Learning
Streaming data
pipelines &
adaptive
architectures
Continuously
challenge your
assumptions by
measuring
results.
Discovery of
patterns and
signals hidden in
data
Agile
Delivery
Data
Science
Deliver business
value early and
often. Build your
platform over time,
not all up front.
Your Business
Questions
=
Fast results &
Early Value
Data
Guided
Market
Advantage
8. DATA LAKE DONE RIGHT
8
Operational systems
communicate directly
with each other via
services
Operational systems
push data to the lake
via topical queues
Data scientists
explore the lake for
potential insights
Lakeshore marts and
services curate and
organize the data for
self-service analysis
Multi-tiered data lake
for processing,
distribution, serving
9. ADAPTIVE ARCHITECTURE PRINCIPLES
9
Enable low latency
data streaming
Store raw, low-level,
historized data
Enable NoSQL
presentation
Enable inexpensive
scaling
Simplify data ingestion
Drive logic closer to the
business
Enable emergent design
Enable easy recreation
of data
12. Discover &
Explore
Analyze & Act
Data Convergence Analytical Divergence
Discover
Harvest
Filter
Integrate Augment
Analyze
Act
Analytical Opportunities
HOW DATA SCIENCE WORKS Can we
anticipate what
the customer
will want to do
next?
13. THE “DATA SCIENTIST”
Machine Learning Statistical Modeling
Artificial Neural Networks
Decision Tree Learning
Support Vector Machines
Unsupervised Learning
…and many more…
Bayesian Classification
Monte Carlo Simulation
Logistic Regression
K-Nearest Neighbor
…and many more…
Feature Engineering
Feature Extraction
Dimension Reduction
Domain expertise
Programming Skills
Functional Programming
Data “Wrangling”
Map/Reduce, SQL, & NoSQL
22. Typical Timeline
3-6 months 1-2 months 2-4 months
22
Data Convergence Analytical Divergence
Discover
Harvest
Filter
Integrate Augment
Analyze
Act
Analytical Opportunities
CONVENTIONAL DATA SCIENCE
If we knew X,
we could do Y
23. Analytical Divergence
Analytical Opportunities
If we knew X,
we could do Y
Data Convergence
Discover
Harvest
Filter
Integrate Augment
Analyze
Act
Repeat this cycle solving small problems every few days
LEARN
MEASURE
BUILD
LEAN DATA SCIENCE
25. Retain high value
customers
Problem
solved or
continue?
High value business
goal
What’s the
smallest, simplest
thing we can do?
Is it useful &
actionable?
Repeat!
What leads to customers
leaving?
LIKE THIS EXAMPLE…
Common features of
defectors?
Shopping behaviors of
defectors?
What do defectors say
about us?
Customers’ sentiment
before defecting?
What encourages
customers to stay?
Do incentives reduce
defection rates?
30. Retain high value
customers
Like this example…
Common features of
defectors?
What leads to customers
leaving?
Shopping behaviors of
defectors?
What do defectors say
about us?
Customers’ sentiment
before defecting?
What encourages
customers to stay?
Do incentives reduce
defection rates?
31. Problem
solved or
continue?
What leads to customers
leaving?
Like this example…
Common features of
defectors?
Shopping behaviors of
defectors?
What do defectors say
about us?
Customers’ sentiment
before defecting?
What encourages
customers to stay?
Do incentives reduce
defection rates?