The document discusses AWS IoT Analytics and provides an overview of its components and capabilities. It describes how AWS IoT Analytics can be used to collect, preprocess, enrich, store, analyze and visualize IoT device data at scale. Examples are given of how various customers like Valmet and iDevices are using AWS IoT Analytics for applications like predictive maintenance, product optimization, and gaining business insights.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
AWS IoT Analytics
S a r a h C o o p e r , G M o f I o T A n a l y t i c s a n d A p p s , A W S
K i p L a r s o n , P r i n c i p a l P r o d u c t M a n a g e r , A W S
K i m m o D j u p s j ö b a c k a , I T A r c h i t e c t , V a l m e t
E r i c F e r g u s o n , C h i e f S o f t w a r e A r c h i t e c t , i D e v i c e s
N o v e m b e r 2 9 , 2 0 1 7
G e t t i n g S t a r t e d w i t h I o T A n a l y t i c s

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect from the session
• Analytics in IoT
• Introducing AWS IoT Analytics
• AWS IoT Analytics Components
• Customer Story by Valmet
• Customer Story by iDevices

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Devices
Sense & Act
Cloud
Storage & Compute
Intelligence
Insights & Logic → Action
Three pillars of IoT

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why is AWS IoT Analytics important?
Industrial
automation
Improved product
design
Optimized business
processes
Improved user
experience

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analytics within AWS IoT
Devices Cloud
AWS
Greengrass
Rules Engine
AWS
Amazon
Redshift
Amazon Kinesis
Collect
Preprocess
& Enrich
Store
Analyze
Visualize
Corp Apps
Corp Data Center
Enterprise Applications
&
Non-IoT Data
Contextual
Data
Intelligence
Device shadow
Message Broker
Device Gateway
Device Registry
AWS IoT AnalyticsAWS IoT Core

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introducing
AWS IoT Analytics

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics
IoT Analytics is a fully managed IoT analytics service
that collects, preprocesses, enriches, stores,
analyzes, and visualizes IoT device data at scale.
From raw
sensor data to
sophisticated
IoT analytics

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics requirements
High volume of
data
Clean data Contextual
information

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges with AWS IoT data and analytics
High volume Multiple sources Noisy and no
standard format
Incomplete and no
contextual
information

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What customers are asking for
Business relevant
reporting
Preprocessing
unstructured, noisy
data
Data collection from
multiple sources
Time series data
storage
Advanced analytics
and machine
learning

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
From AWS IoT data to sophisticated analytics
Thing Registry
Log InkLevel
Queue SQS
Ink Level
CloudWatch Log
Job History Tables
DynamoDB
deviceID/usage/
inklevels
Forward Ink
Levels Rule
Low Ink Test
Queue SQS
Log Ink Remaining
Lambda Worker
Alarm State
Worker Lambda
Send Mobile
App Alert
deviceID/events/
consumables Ink Replace Rule
Device Enricher
Worker Lambda
deviceID/events/jobs
Send Ink Alarm
Queue SQS Notifier Lambda
Low Ink Alarm
Queue SQS
Log Job
Queue SQS
Log Job Lambda
Worker
Job Ink Process
Queue SQS
Job Calculation
Lambda Worker
Job Alarm Threshold
Lambda Worker
Print Job
Raw Data
Storage
Lambda
Logging
2
2
21
3 3
4 4
Ink Level Remaining
CloudWatch Alarm
5 5 8
8
4
Sales Cloud
SFDC Customer,
Ink Order Table &
Promotions Table
6
7
7
5
6
Low Ink Test
Lambda Worker
Check Ink
Threshold
9
9
deviceID/inbound/
inkrefill
SFDC Ingest
Firehouse
10
11 11
9
12
19
14
14
18
Print Job Error
13 13
Device Enricher
Worker Lambda
Thing Registry
16
14
15 15
14
16
16
17
17 17
20
20
21
21
deviceID/*/*
Backup Rule
Job History &
Ink Level APIs
Enrich Job
Queue SQS
Enrich Job
Lambda Worker
Job Alarm
Queue SQS
Job Entry Completion
Lambda
19

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics overview
Create IoT data Collect Preprocess
& enrich
Store Analyze & visualize

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics components
DatasetsPipelines
Collect Preprocess
& enrich
Store Analyze & visualize,
machine learning
Channels Datastores Notebooks & Amazon
QuickSight
QueryCreate IoT data
Operational
and business
applications

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IoT Analytics: Collect via channels
• Entry point to AWS IoT Analytics
• Native integration with AWS IoT Core
• Authoritative store of raw data from devices
• Built-in micro-batching partitioning by date
• Supports both binary and JSON data
MQTT
Amazon
Kinesis

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IoT Analytics: Preprocess & enrich
via pipelines
• AWS IoT specific device data preparation
pipeline
• Consumes raw data from a Channel and
sends processed data to a datastore
• Set of activities for filtering/transforming/
contextualizing messages
• Works best with JSON data

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics: Pipeline activities
• Consume from a channel
• Send to a datastore
• Invoke AWS Lambda
• Add/remove/select attributes
• Apply regex
• Filter messages
• Validate messages
• Evaluate math expression
• Enrich from MQTT topic
• Enrich from AWS IoT device registry
• Enrich from AWS IoT device shadow

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics: Store via datastores
• Store optimized for analytical & time series
queries against processed data
• Is not a database, but an abstraction on top
of several database technologies
• Works on semistructured JSON data
• Partitioned by time

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics: Analyze via datasets
• Result of analysis against data store
• Run on schedule or ad hoc
• Conceptually similar to materialized view
• Accessible from the console/API/Jupyter
Notebooks/Amazon QuickSight

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics: Powerful analytical
notebooks
• Jupyter-based machine learning notebooks
(using Amazon SageMaker)
• Integration with existing AWS IoT Analytics
datasets
• Built-in templates for predictive maintenance
and other IoT-specific use cases

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Analytics: Visualize
via Amazon QuickSight
• Native integration with Amazon
QuickSight
• Makes it easy to build visualizations,
perform ad hoc analysis, and get
business insights.

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hear from our customers

22. Dialogue with data
Valmet Industrial Internet

23. Ÿ Valmet is the leading global developer and supplier of process
technologies, automation, and services for the pulp, paper, and energy
industries
#1 in paper, board, and tissue
#1–2 in pulp and services
#1–3 in energy and automation
Ÿ One of the world’s 300 sustainability leaders
– Dow Jones sustainability index for the fourth consecutive year,
GDP Climate A- list and Ethibel
Ÿ 220 years of industrial history in 2017
Ÿ Our vision to become the global champion in serving our customers
This is Valmet
14 December, 2017 © Valmet | Industrial Internet23
Total net sales, MEUR
2,926
Personnel
12,012
2016

24. We want to become the most wanted Industrial Internet partner
for our customers
14 December, 2017 © Valmet | Industrial Internet24
Valmet develops its Industrial
Internet solutions building on
the unique combination of
process technology, services,
and automation.
Industrial Internet
solutions
Customer’s
process
Valmet
experts
Valmet’s
competence
network

25. Valmet has a long history in the digitalization of process industries
14 December, 2017 © Valmet | Industrial Internet25
1970
Elmatic-100 system,
electronic
instrumentation
1990’s
PaperIQ, QCS
metsoDNA, Dynamic Network
of Applications
1980’s
Damatic, the first Distributed Control System
(DCS)
Sensodec Condition Monitoring System
Damatic XD, modular second generation DCS
2000’s
Multivariable Model
Predictive Controls (MPC)
24/7 ProCenter for DCS/QCS
PaperIQ Select
2010’s
Metso PQV
web inspection system
2015
Valmet DNA
2015
Valmet IQ
1960
The Airmatic
a pneumatic
measurement and
control system
2017
Augmented and virtual
reality applications
Valmet
Performance
Centers
Distributing
controls &
gathering
performance data
Increasing
availability
Increasing
productivity
through
information
services
Embedded
intelligence &
advanced
information
Advantages to
customer
industries:
Benchmarking and
best practice
sharing capability

26. Today, customers are extensively utilizing our
Industrial Internet capabilities
14 December, 2017 © Valmet | Industrial Internet26
Online connections
with customers
Performance
agreements with
remote
connections
Co-creation of
advanced analytics
with customers
Valmet-supplied
lines with Valmet
DCS
540
420
81,000 90
Advanced process
control installations
350800 Ongoing
Valmet
experts
Valmet’s
competence
network
Customer’s
expert
380 Condition
Monitoring (CM)
references with
over 70,000 I/O
tags

27. Applications and services
Valmet Industrial Internet offering
Data visualization, reporting, and guidance
• Equipment and process performance reporting
• Fleet performance reporting
• Visualization and guidance with augmented and virtual reality
Asset reliability optimization
• Process component tracking and monitoring
• Equipment predictive and prescriptive diagnostics
• Overall equipment efficiency optimization
Operations performance optimization
• Real time analytics to improve production, quality, and cost
• Advanced Process Control (APC) and plant or mill-wide optimization
• Fleet performance optimization
Valmet Performance Center
• Remote analytics, monitoring, and optimization
• On-demand expert support
• Data discovery and analysis
14 December, 2017
27

28. Valmet Industrial Internet Platform: Logical architecture
14 December, 2017 © Valmet | Industrial Internet28
Customer
Valmet
&
Partners
INDUSTRIAL INTERNET PLATFORM

29. Valmet and AWS IoT Analytics PoC

30. Valmet IoT Analytics PoC case: Paper strength prediction *
14 December, 2017 © Valmet | Industrial Internet30
Ÿ A paper production line produces paper at a speed of about 100 km/h (60 miles/hour)
Ÿ Paper is wound to a reel at the end of production line, about one reel per hour
Ÿ A paper sample is taken from the top of each reel
Ÿ A laboratory measures several paper quality properties from the sample
Ÿ For this project, we used only one quality property “paper strength”
Ÿ We are trying to find a way to predict paper strength based on available process measurements
* We have implemented this already earlier with other tools

31. Paper mill
Gathering data from a paper mill to AWS
14 December, 2017 © Valmet | Industrial Internet31
AWS IOT
Topic process
Topic lab
Ÿ Reads daily files
Ÿ Generates machine messages
– Tag filtering
– Time windowing
– Tag grouping
– Target topic by tags groups
Valmet DNA
Historian
Dumper
Dumper
Lab
dataLab
dataLab
dataLab
data
Process
dataProcess
dataProcess
dataProcess
data
Machine message
generator

32. Selecting relevant data,
implement in IoT Analytics data pipe
14 December, 2017 © Valmet | Industrial Internet32
15 min
Selecting data from right time window
• Calculate the 15-min. average from the time
window in the end of the reel ( red point)
End of reel
Laboratory values:
• Select the closest value after the end of reel
inside an accepted time window, e.g., 20
minutes
Reel length

33. Data cleaning, implemented
in AWS IoT Analytics data pipe
14 December, 2017 © Valmet | Industrial Internet33
Process data average is not acceptable
if given constraints are violated.
Example:
• Headbox lip opening setpoint has to be
constant over the time period used to
calculate average values of process data
à STD is low
• Machine speed cannot change over a given
limit over the averaging period
à Max – min < given limit
15 min
Lip opening
End of reel
Speed
Discarded time windowApproved time windows

34. Data analyses in Jupyter Notebook
14 December, 2017 © Valmet | Industrial Internet34
Data for modeling (matrix M x N )
Time vector Process data Laboratory data
Data for the modelling is arranged in M x N matrix.
M accepted time points
N columns (one time vector, process data vectors, lab value vectors)
Each row contains data from such a time point where reel ends and laboratory
samples are taken. Process data passes also also the given conditions for the
cleaning.
Correlation matrix
Sorting
Variable
selection
Regression
model
R^2
maximation
Visualization
Measured and
Predicted tear strength,
based on 10 variables

36. iDevices is a proud member of the Hubbell family.
AWS IoT Analytics
D e c e m b e r | 2 0 1 7

37. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
AGENDA
37
I. iDevices Overview
II. iDevices Analytical Use Cases
III. Typical Analytics Workflow
IV. Use Case Examples
I. Anomaly Detection
II. Marketing Dashboard
V. Q&A

38. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
Deployed Products
• Mixture of “plug-in” and “in-wall”
products, primarily focusing on
power/lighting and temperature
control
Raw Data Sources
• Real-time usage/activity/connectivity
metrics
• Voice request metrics (i.e. request
counts/status/timestamps/etc.)
Hubbell Family
• Extending our technology into the
commercial and industrial markets
IDEVICES OVERVIEW
38

39. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
Operational Insights
• Anomaly detection
• Predictive maintenance
• Runtime performance monitoring
Business Insights
• Consumer engagement
• Feature usage / validation
• Product segmentation
Customer Features
• Predictive scheduling
• Historical display
• Aggregated regional statistics
IDEVICES ANALYTICAL USE CASES
39

40. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
THE PLAN
40
Phase 1 Phase 2 Phase 3
Collect
Data
Insights???

41. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
Raw Data Visualization / Exploration
• Leverage Amazon QuickSight and Jupyter notebooks to explore
and visualize raw data
• Train and evaluate ML models
Data Enrichment
• Use Jupyter as a playground for prototyping pipeline
modifications and use case-specific visualizations
Pipeline Formation
• Modify ingestion pipeline to preprocess (i.e. filter, transform,
normalize, enrich) incoming data in real-time, based on above
Storage
• Route processed data to application-specific datastores
Consumption
• Trigger delivery to appropriate dashboard or hosted ML model
TYPICAL ANALYTICS WORKFLOW
41

42. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
OUR INGESTION PIPELINE
42
Alexa Skill
Amazon Cloud
Watch Log
Subscription
Firehose
Adapter Lambda
Amazon
Redshift
Kinesis
Firehose
Alexa
Before
After
Alexa Alexa Skill IoT Analytics

43. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
IOT ANALYTICS COMPONENTS
43
Devices
Alexa Skill
Lambda
Alexa Usage
Amazon
SNS
Binned Metrics
Anomaly Detector

44. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
# create datastore
aws iotanalytics create-datastore --datastore-name binned_datastore
PIPELINE CREATION
44
# create channel
aws iotanalytics create-channel --channel-name alexausage_channel
# create pipeline
aws iotanalytics create-pipeline --pipeline-name alexausage_pipeline --pipeline-activities=‘[{
“channel”: {
“name”: “alexausage_channel_activity”,
“channelName”: “alexausage_channel”,
”next”: “binned_store_activity”},
”datastore”: {
“name”: “binned_store_activity”,
“datastoreName”: “binned_datastore”}
}]’

45. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
# create topic rule
aws iot create-topic-rule --rule-name alexausage_rule --topic-rule-payload=‘{
"sql": "SELECT * FROM 'iot/alexausage'",
"ruleDisabled": false,
"awsIotSqlVersion": "2016-03-23",
"actions": [{
"iotAnalytics": { "channelArn": "arn:aws:iot:us-west-2:[AWSAcctID]:channel/alexausage_channel”}
}]
}’
PIPELINE CREATION (cont)
45
# create dataset
aws iotanalytics create-dataset --dataset-name alexausage_dataset –-actions=‘[{
“actionName”: “alexausage_action”,
“queryAction”: { “sqlQuery”: “select * from binned_datastore” }
}]’

46. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
Description / Business Need
Automated reporting of degraded Alexa
request performance
Previous State:
• Custom pipeline and storage using CloudWatch
logs, Kinesis Firehose, and Redshift
• Basic monitoring / graphing reliant on visual
inspection / interpretation of output
Challenges:
• Anomalous behavior can span multiple domains
• Hardcoded “threshold” reporting can be error
prone and incomplete
USE CASE #1: ANOMALY DETECTION
46

47. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
DEMO
USE CASE #1: ANOMALY DETECTION (cont)
47

48. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
Future Extensions
Add additional input features to account for more complex correlations
• i.e. latency, failures per customer, failures per server, ISP outages, etc.
• Experiment with new features to help with correlation
Deploy detector in other domains
• Backend server monitoring, etc.
Experiment with other architectures
• Neural networks to see if they can determine better correlation between input features
• Unsupervised real-time anomaly detection to automatically learn “normal” patterns from
streaming data, without overhead of hand classifying anomalous examples
USE CASE #1: ANOMALY DETECTION (cont)
48

49. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
Description / Business Need
Visibility to quantitative metrics related to
customer trends and usage of our deployed
products
Previous State:
• Visibility limited to only operational metrics
• No formal outlet for marketing and sales teams to
consume
Challenges:
• Many decisions on product feature sets and
roadmap were based on sometimes invalid
assumptions not grounded in real usage
• No direct correlation between sales numbers and
actual customer engagement
USE CASE #2: MARKETING DASHBOARD
49

50. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
USE CASE #2: MARKETING DASHBOARD (cont)
50
Devices
Alexa Skill
Lambda
Alexa Usage
SNS
Binned Metrics
Time-series Alexa Data
Time-series Device Data
Device Usage
Anomaly Detector
QuickSight
Dashboard

51. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
DEMO
USE CASE #2: MARKETING DASHBOARD (cont)
51

52. © 2017 iDevices, LLC. All information contained in this document is confidential and proprietary to iDevices, LLC.
USE CASE #2: MARKETING DASHBOARD (cont)
52

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Get Started with
AWS IoT Analytics
aws.amazon.com/iot-analytics

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
A W S I o T A n a l y t i c s