A key advantage of the IoT is that it enables you to expand the potential of constrained devices by giving them access to the computational power of The Cloud. In this session I’ll show you how I transformed a lowly Raspberry Pi Zero webcam into a smart security camera (with motion detection, threat detection and alert notifications) by combining open source software with cloud based image analysis. Attendees can expect a short explanation of how to set up their own Raspberry Pi Zero webcam, an introduction to Image Analysis as a Service, and finally a demonstration of how I used Node.js and a range of cloud based API’s (including Amazon’s Rekognition, Lambda and Step Function services) to help my smart security camera distinguish between an unwanted guest and the neighbours cat. http://iottechday.nl/sessions/building-a-smart-security-camera-with-raspberry-pi-zero-node-js-and-the-cloud/

1. Building a Smart Security Camera
with Raspberry Pi Zero, Node.js
and The Cloud
Mark West
#iottechday

2. Who Am I?
• IT Consultant at Bouvet.
• Hacker and Maker.
• Java, JavaScript, AI, Cloud & IoT.
• Active member of javaBin - the
Norwegian JUG.
@markawest

3. Talk Outline
Motivation and
Requirements
Building a
Motion
Activated
Camera
Adding Smart
Image Analysis
with AWS
Evaluation and
Lessons Learned
@markawest

4. Motivation and Requirements
Input from Stakeholder
Motivation and
Requirements
Building a Motion
Activated Camera
Adding Smart
Image Analysis
with AWS
Evaluation and
Lessons Learned
@markawest

5. Initial Motivation
@markawest

6. Project Requirements
Functional
• Monitor activity in the
garden.
• Send warning when activity
detected.
• Live video stream.
Non-functional
• In place as soon as possible.
• Low cost.
• Portable.
@markawest

7. Functional Design
1. Activity
in garden
2. Camera
detects
movement
3. Camera sends
alert email with
snapshot
@markawest

8. Building a Motion Activated
Camera
Motivation and
Requirements
Building a Motion
Activated Camera
Adding Smart
Image Analysis
with AWS
Evaluation and
Lessons Learned
@markawest

9. Pi Zero W Essentials Kit: €30
Camera Module: €28
Camera Adapter: €6
ZeroView: €9
Total Cost: €73
Hardware Shopping List
@markawest

10. Assembled Camera
Rear View Front View
@markawest

11. Implementing the Functional Design
1. Activity
in garden
2. Camera
detects
movement
3. Camera sends
alert email with
snapshot
@markawest

12. Motion
(https://motion-project.github.io)
• Open source motion detection software.
• Excellent performance on the Raspberry Pi Zero.
• Built-in Web Server for streaming video.
• Detected activity or ‘motion’ triggers events.
• Works out of the box. No need for additional programming.
@markawest

13. Motion
Streaming
Demo
@markawest

14. Example
Alert Email
1 2
1
@markawest

15. How the Motion Software works
1 2 3 4
@markawest
1 2 3

16. How the Motion Software works
1 2 3 4
1 2 3
@markawest
1 2 3

17. How the Motion Software works
1 2 3 4
1 2 3
@markawest
1 2 3

18. Alarm Snapshots from Web Camera
@markawest

19. Project Requirements : Evaluation
Functional
• Monitor activity in the
garden.
• Send warning when activity
detected.
• Live video stream.
Non-functional
• In place as soon as possible.
• Low cost.
• Portable.
@markawest

20. False Alarms from Web Camera
cat
cloud
@markawest

21. The Motion Software
focuses on the amount of
changed pixels, and not the
cause of the changed
pixels!
@markawest

22. Adding Smart Image Analysis
with AWS
Motivation and
Requirements
Building a Motion
Activated Camera
Adding Smart
Image Analysis
with AWS
Evaluation and
Lessons Learned
@markawest

23. Improving Alarm Accuracy with
Smart Image Analysis
person
@markawest
cat

24. Project Requirements Reloaded
Functional
• Monitor activity in our
garden.
• Send warning when activity
detected.
• Live video stream.
Non-functional
• In place as soon as possible.
• Low cost.
• Portable.
@markawest

25. Project Requirements Reloaded
Functional
• Monitor activity in our
garden.
• Send warning when human
activity detected.
• Live video stream.
Non-functional
• In place as soon as possible.
• Low cost.
• Portable.
@markawest

26. Finding an Image Analysis Solution
OpenCV
• Use Face Detection to find out if
a human was in the snapshot.
• Problem: What if the subject
was facing away from the
camera, or wearing a mask?
TensorFlow
• Train and use a Neural Network
to find humans in the snapshot.
• Problem: Potentially a fun
project, but would take more
time than I had available.
@markawest

27. @markawest

28. AWS Rekognition
• Part of Amazon Web Services suite of Cloud based services.
• Image Analysis as a Service, offering a range of API’s.
• Built upon Deep Neural Networks.
• Officially launched in November 2016.
• Alternatives: Google Vision, MicroSoft Computer Vision, Clarafai.
@markawest

29. AWS Rekognition Demo
@markawest

30. Adding AWS to the Web Camera
Amazon Web Services (AWS)
Send warning
email (if snapshot
contains a person)
Snapshot analysed
via AWS
Rekognition
@markawest
Camera pushes
snapshot to AWS
Email alert sent
(with snapshot)

31. AWS IAM
AWS
Rekognition
AWS Simple
Email Service
AWS S3
(storage)
Amazon Web Service Flow
Overview
AWS Step Function
(workflow)
Upload
Trigger
1
2
4
5
6
calls
3
uses
uses
@markawest

32. AWS Software Developer Kit (SDK)
• SDK’s exist for a range of programming languages and platforms.
(including Python, Ruby, .NET, iOS, Go, C++ and Java)
• Provides an API for consuming AWS Services.
• For this project I used the Node.js AWS SDK.
(including the upload from Camera to S3)
@markawest

33. AWS IAM
AWS
Rekognition
AWS Simple
Email Service
AWS S3
(storage)
AWS Lambda Functions
Code Building Blocks (aka Microservices)
AWS Step Function
(workflow)
Upload
Trigger calls
uses
uses
@markawest

34. AWS Lambda Functions
• Code units, based on Java, C#, Python or Node.js
• Serverless.
• High availability out of the box.
• Pay as you go model, with a generous free tier.
• Native AWS SDK gives easy access to other AWS services.
@markawest

35. AWS Lambda Demo
@markawest

36. AWS IAM
AWS S3
(storage)
AWS Step Functions
Orchestration of Lambda Functions
AWS Step Function
(workflow)
Upload
Trigger calls
AWS
Rekognition
AWS Simple
Email Service
uses
uses
@markawest

37. AWS Step Functions
• Coordinate and orchestrate Lambda Functions into Workflows or
State Machines.
• Launched in December 2016.
• Defined via JSON files, displayed as visual workflows.
• Provide the same benefits as AWS Lambda (High Availability,
Serverless, Scalable, “Pay as you go” pricing model).
@markawest

38. Step Function for Image Processing
Error
Handler
(sends
Error
Email via
AWS
SES)
Sends snapshot to AWS Rekognition
Evaluate AWS Rekognition response
Send Alert Email?
Send Alert Email
via AWS SES
Archive image in AWS S3
1
2
3
4
5
6 1
2
3
4
5
6
@markawest

39. Smart Camera
(plus AWS Step Function)
Demo
@markawest

40. Evaluation and Lessons
Learned
Motivation and
Requirements
Building a Motion
Activated Camera
Adding Smart
Image Analysis
with AWS
Evaluation and
Lessons Learned
@markawest

41. Project Requirements : Evaluation
Functional
• Monitor activity in the
garden.
• Send warning when human
activity detected.
• Live video stream.
Non-functional
• In place as soon as possible.
• Low cost.
• Portable.
@markawest

42. AWS Rekognition : ThePositives
• Consistent results.
• Successfully handles ’partial’ images.
• Handles poor quality images (motion
blur, poor lighting).
• No more false alarms.
@markawest

43. AWS Rekognition : Some Negatives
• Black box.
• Unable to provide feedback.
• Struggles with ‘noisy’ pictures.
• Results are only as good as the pictures
you upload (resulting in a 80-95% hit
rate).
@markawest

44. Activity Timeline
Improving AWS Rekognition Hit Rate
@markawest
Activity
Event
Start
1 Activity
Event
Stop
2 3 4 5

45. Activity Timeline
Improving AWS Rekognition Hit Rate
@markawest
Activity
Event
Start
1 Activity
Event
Stop
2 3 4 5
Most
Pixels
Changed

46. Activity Timeline
Improving AWS Rekognition Hit Rate
@markawest
Activity
Event
Start
1 Activity
Event
Stop
2 3 4 5
Most
Central
Activity

47. Activity Timeline
Improving AWS Rekognition Hit Rate
@markawest
Activity
Event
Start
1 Activity
Event
Stop
2 3 4 5
Upload all snapshots

48. Improving AWS Rekognition Hit Rate
Upload One Snapshot
when activity detected
• Lower cost, due to less calls
to the AWS stack.
• Potentially lower hit rate.
Upload Many Snapshots
when activity detected
• Higher cost due to
increased calls to the AWS
stack.
• Potentially higher hit rate.
@markawest

49. Some Benefits of Using AWS
• Very easy to get started with, due to Free Tier, automated
infrastructure and good documentation.
• Large ecosystem with a broad range of services.
• Serverless - so hot right now!
@markawest

50. Some Drawbacks of Using AWS
• Developer Tooling for Lambda Functions not yet mature.
• Step Functions need a graphical editor.
• Potential for vendor lock-in.
@markawest

51. AWS Monthly Cost
Actual AWS Cost March
2017
$1.07
(under free tier)
Estimated AWS Cost After
Free Tier Expiration
$8.00
(same usage as March 2017)
@markawest

52. S3 for Web Hosting
• Created a HTML Report listing all
False Alarm snapshots.
• Report updated by a Lambda
Function triggered by a timer.
• Hosted on S3 as a static website.
@markawest

53. Would you like to know more?
markwest1972/
smart-security-
camera
utbrudd.bouvet.no
@markawest
slideshare.net/
markawest
Slides Blog Code

54. please rate my talk in the official
IoT Tech Day app
@markawest