WSO2Con USA 2015: Keynote - The Future of Real-Time Analytics and IoT

1. The Future Of Analytics And IoT
It Can’t Happen Without You
Mike Gualtieri, Principal Analyst
November 3, 2015 San Francisco
Twitter: @mgualtieri

2. #Priority

3. © 2015 Forrester Research, Inc. Reproduction Prohibited 3
52%
53%
53%
54%
58%
64%
64%
65%
66%
73%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Better leverage big data and analytics in business decision-making
Create a comprehensive strategy for addressing digital technologies like mobile,
social & smart products
Create a comprehensive digital marketing strategy
Better comply with regulations and requirements
Improve differentiation in the market
Increase influence and brand reach in the market
Address rising customer expectations
Improve our ability to innovate
Reduce costs
Improve our products /services
Improve the experience of our customers
Customer experience is a top priority for
business leaders
› Base: 3,005 global data and analytics decision-makers
› Source: Global Business Technographics Data And Analytics Online Survey, 2015

4. For you For all For segments For you
Demographic
Relationships
Hyper-Personal,
Real-Time
Relationships
Personal
Relationships
Mass
Relationships
CustomerExperience
1800 1900 1950 2000 2015

5. #Celebrity

6. Customers want and increasingly expect
to be treated like celebrities.

7. • Learn individual customer
characteristics and
behaviors
• Detect customer needs and
desires in real-time
• Adapt applications to serve
an individual customer
Celebrity experiences must:

8. #Analytics

9. © 2015 Forrester Research, Inc. Reproduction Prohibited 9
Learn Model Detect Adapt
True BI means having four kinds of analytics
Predictive
Analytics
Streaming
Analytics
Descriptive
Analytics
(Advanced Analytics)
Prescriptive
Analytics
Batch Real-time
Most firms invest here They must invest here too

10. © 2015 Forrester Research, Inc. Reproduction Prohibited 10
Source: Forrester Research
Advanced analytics is rightly surging
“What is your firm's/business unit's current use of the following technologies?”
Source: Forrester's Global Business Technographics Data And Analytics Survey, 2015 and 2014
Base: 1805 (2015), 1063 (2014)
19%
19%
24%
31%
34%
22%
22%
35%
31%
43%
53%
54%
50%
50%
69%
39%
42%
42%
42%
42%
43%
43%
46%
48%
52%
54%
55%
56%
57%
69%
Non modeled data exploration and discovery
Search/interactive discovery
Streaming analytics
Metadata generated analytics
OLAP
Advanced visualization
Text analytics
Location analytics
Predictive analytics
Process analytics
Embedded analytics
Web analytics
Dashboards
Performance analytics
Reporting
2015
2014
Sweet!Most of your
competitors
still haven’t
started!

11. #Data

12. Some say that data is the new oil.

13. It’s more like the Sun – virtually limitless.

14. © 2015 Forrester Research, Inc. Reproduction Prohibited 14
Every industry is graced with more data
› Richer transactional data from portfolio of hundreds of
business applications
› Usage and behavior data from web and mobile apps
› Social media data
› Log data
› IoT device sensor and event data
› Data economy – firms buying and selling data

15. Using your best estimate, what is the size of
all data stored within your company?
Source: Forrester Research, September 2015
Base: 100 US Managers and above currently using Hadoop for processing and analyzing data.
Enterprises have plenty of data from both internal and
external sources
10-49
Terabytes
5%
50-99
Terabytes
12%
100-500
Terabytes
54%
Greater than
500
Terabytes
29%
Internal
business
data
49%
External
source data
51%
What % of the data available is from internal
business applications (ERP and business
applications) versus external sources
(social, IoT)?

16. All data is born fast!

17. 110010011011001
010010011
010011001101
0100
CustomerData
Transactions
DataWarehosue
IoT
But, analytics is usually done much later.

18. #WhyWait

19. Perishable insights can have exponentially more
value than after-the-fact traditional historical
analytics.

20. #Perishable

21. How can you prevent this dude from fleecing
you right now?

22. What are movers and shakers saying about
equities that we cover right now?

23. How can you know if your baby is sleeping
soundly or if something is wrong right now?

24. How can you warn other drivers that the
road is slippery to avoid a crash right now?

25. What offers should you make to your customer if
they are within proximity of your store right now?

26. What music should you play if your customer is
jogging right now?

27. #

28. © 2015 Forrester Research, Inc. Reproduction Prohibited 28
Most firms struggle make insights actionable
Insights for lines of
business:
Source: October 16, 2014, “The Customer-Activated Enterprise” Forrester report

29. Applications are blind – IoT can make them
see.

30. © 2015 Forrester Research, Inc. Reproduction Prohibited 30
Learn Model Detect Adapt
Build analytics into your applications
Predictive
Analytics
Streaming
Analytics
Descriptive
Analytics
(Advanced Analytics)
Prescriptive
Analytics
Batch Real-time
Most firms invest here They must invest here too

31. #Predictive

32. © 2015 Forrester Research, Inc. Reproduction Prohibited 32
Top 3 most read Forrester
research reports in Q3 2015.
#1 Does Customer Experience Really Drive
Business Success?
#2 Forrester Wave: Big Data Predictive
Analytics, Q2 2015
#3 The US Customer Experience Index,
Q12015

33. ANALYTICS
PREDICTIVETechniques, tools, and technologies that use
data to find models – models that can
anticipate outcomes with a significant
probability of accuracy.

34. © 2015 Forrester Research, Inc. Reproduction Prohibited 34
Ways to create predictive models
› Handcraft business rules or program code to
create a predictive model based on human
experts.
› Use big data predictive analytics tools to build
models that analyze data with machine learning
algorithms to build a predictive model.
› Sometimes you need both.

35. © 2015 Forrester Research, Inc. Reproduction Prohibited 35
Predictive models can be very powerful and
profitable, but understand that:
› Predictive models are about probabilities, not absolutes
• E.g. 78% chance you will like Better Call Saul
› Accurate predictive models may not exist for every
question
• E.g. Economists have a very poor record
› Prediction models are not necessarily and probably not
causative

36. Correlation does not imply causation.

37. Data scientists use a
combination of statistical
and machine learning
algorithms to find
patterns and predictive
models.

38. © 2015 Forrester Research, Inc. Reproduction Prohibited 38
Are you a data scientist?
38
K-means clustering
Association rules
Boosting trees
CHAID
Cluster analysis
Feature selection
Independent components analysis
Kohonen Networks (SOFM)
Neural networks
Social network analysis (SNA)
Random forests
Mars regression splines
Linear and logistic regression
Naïve Bayesian classifiers
Optimal binning
Partial least squares
Response Optimization
Root cause analysis
Support vector machines
Natural language processing

39. © 2013 Forrester Research, Inc. Reproduction Prohibited 39
Big data is the fuel and machine learning is the
engine
› Classifiers
• Predict a specific event, characteristic, or behavior
› Recommenders
• Make a recommendation
› Clusters
• Find groups that share common characteristics

40. What customers are likely to plan a vacation in
the next 6 weeks?

41. #Prescriptive

42. Adapt and act in real-time.

43. ANALYTICS
PRESCRIPTIVETools, techniques, and technologies that are
used to determine the next best decision or
action using a combination of methods including
business rules, descriptive/predictive/streaming
analytics, and optimization.

44. How can you influence swing voters to vote for
you?
Image source: iStockphoto

45. 45© 2015 Forrester Research, Inc. Reproduction Prohibited
You must support both analytical and
intuitive decision makers.
Analytical
Intuitive
Automated
Decision can be modeled
Information is available
Algorithm can be defined
Expert
Decision is not modeled
Unknown or fuzzy information
Experience and judgment
Image sources: Cryteria, Eurogamer.net, Business Insider, CBS Entertainment, Cinema Blend

46. 46© 2015 Forrester Research, Inc. Reproduction Prohibited
Prescriptive analytics relies on a
combination of approaches
• Business rules and/or program code based on
human expertise and experience.
• Use previously built predictive models.
• Use previously solved mathematical
optimizations.
You will likely need a combination of all of
these to make the best possible decision.

47. #Streaming

48. Streaming analytics can detect and act on
perishable insights.

49. DEFINITION
FORRESTERStreaming analytics filter, aggregate, enrich,
and analyze a high throughput of data from
disparate live data sources to identify patterns,
detect urgent situations, and automate
immediate actions in real-time.

50. © 2015 Forrester Research, Inc. Reproduction Prohibited 50
Real-time means business-time
› A customer walks into a shopping mall
› A shopper clicks on an online add
› A temperature sensor spikes
› A stock price rises
› A customer uses a credit card
› A customer wakes up

51. © 2015 Forrester Research, Inc. Reproduction Prohibited 51
Thinking in streams is different…
› Ingest
› Filter
› Transform
› Normalize
› Link
› Enrich
› Correlate
› Location/motion (geofencing)
› Time windows
› Temporal pattern detection
› Business logic/rules execution
› Action interfaces
Continuous ETL Continuous Analytics

52. How can an online retailer
sell more motorcycle
helmets and optimize
profits?
› Temporal pattern detection
› Time windows
› Business logic/rules
execution
› Action interfaces

53. © 2015 Forrester Research, Inc. Reproduction Prohibited 53
Streaming analytics enables an entirely new real-
time selling model
› Analytic: When has this user
viewed at least three
motorcycle safety products
including at least one helmet?
› Action: Display most profitable
motorcycle helmets.
› Analytic: What is the real-time
daily total sales of motorcycle
helmets?
› Action: If sales trending lower
than usual, then dynamically
lower price.
Temporal Pattern Detection Time Window

54. © 2015 Forrester Research, Inc. Reproduction Prohibited 54
Try doing that with plain-old, conventional SQL
SELECT SUM(CustomerViews.view)
WHERE
CustomerViews.productCategory =
‘MCSafety’ AND …

55. #

56. Hadoop is designed for volume.

57. Spark is designed for speed.

58. Spark on Hadoop minimizes the latency of loading
HDFS data into memory, and DevOps can use
YARN to tune performance for simultaneous jobs.

59. © 2015 Forrester Research, Inc. Reproduction Prohibited 59
Spark and Hadoop can coexist in the same
cluster.

60. Hadooponomics makes batch analytics at
scale feasible

61. SELECT * FROM Hadoop;
Pure SQL for
Hadoop
Boosted SQL for
Hadoop
• Apache Hive
• Apache Drill
• Apache Phoenix
(for HBase)
• Cloudera Impala
• Presto
• Spark SQL
(through Spark)
• Actian Vortex
• HP Vertica SQL
on Hadoop
• IBM Big SQL
• JethroData
• Pivotal HAWQ
• Microsoft
Polybase
• Oracle Big Data
SQL
• Teradata
QueryGrid
Database+ SQL for
Hadoop

62. Spark’s directed acyclic graph (DAG) engine
maximizes parallelization for batch jobs.

63. © 2015 Forrester Research, Inc. Reproduction Prohibited 63
Spark also includes a growing number of
specialized tools

64. #Lambda

65. © 2015 Forrester Research, Inc. Reproduction Prohibited 65
Learn Model Detect Adapt
Only the analytical enterprise can compete and
win in the age of celebrity
Predictive
Analytics
Streaming
Analytics
Descriptive
Analytics
(Real-time)
Prescriptive
Analytics
(Continuous Batch)
þ þ þ þ
Invest here Invest here Invest here

66. © 2015 Forrester Research, Inc. Reproduction Prohibited 66
Combine application middleware and analytics to
create industrial strength “lambda” applications

67. #

68. 110010011011001
010010011
010011001101
0100
Historical
Transactions
Customerdata
Security
Silos must disappear or appear to disappear.

69. Performance of analytics and services must be
blazing fast.

70. Scale to handle any amount of data.

71. Fault-tolerance is non-negotiable.

72. Confidential information must be secure.

73. Solutions and/or platforms must fit seamlessly
into existing architectures.

74. Embed and act on real-time analytics in
existing and new applications.

75. Diverse, community-based rapid innovation.

76. #You

77. Your customers
are ready to be
treated like
celebrities!

78. What kind of celebrity
customer experiences
can you create with a
great open source
platform?

79. © 2015 Forrester Research, Inc. Reproduction Prohibited 79
Think like a venture capitalist to find
opportunities in your business.
1. Walk through critical or challenging business processes
- At each step of the business process ask how analytics could
improve the process
2. Walk through customer experience to improve customer
experience
- At each step of the customer journey, ask how analytics could
help create celebrity customer experiences

80. Build real-time, predictive apps to
make it happen.

81. forrester.com
Thank you
Mike Gualtieri
mgualtieri@forrester.com
Twitter: @mgualtieri