[Longer ver] From Software to Hardware: How Do I Track My Cat with JavaScript

@girlie_mac
KittyCam.js
Smile, you’re on camera!Smile, you’re on camera!
Tomomi Imura

@girlie_mac
Tomomi (@girlie_mac) is a(n):
● San Francisco Dweller
● Front-End Engineer
● Open Web + Tech Advocate
● N00b Hardware Hacker
● Sr. Developer Advocate at Nexmo
● Cat lady of InterWeb

@girlie_mac
HTTP Status Cats at https://http.cat
Thanks Rogério Vicente for the .cat domain & the API!

@girlie_mac
CC-BY-SA 3.0
https://commons.wikimedia.org/wiki/File:Basic_robot.jpg

@girlie_mac
Arduino
● MCU-based kit
● Open-Source hardware &
software (IDE)
● The first developer-friendly
boards

@girlie_mac
Sketch
● Language for Arduino
● Loosely based on C
● Wut, C?
● HALP!!1!!!
#include <Servo.h>
#include <Wire.h>
#include <Firmata.h>
#define I2C_WRITE B00000000
#define I2C_READ B00001000
#define I2C_READ_CONTINUOUSLY B00010000
#define I2C_STOP_READING B00011000
#define I2C_READ_WRITE_MODE_MASK B00011000
#define I2C_10BIT_ADDRESS_MODE_MASK B00100000
#define I2C_MAX_QUERIES 8
#define I2C_REGISTER_NOT_SPECIFIED -1
#define MINIMUM_SAMPLING_INTERVAL 1
int analogInputsToReport = 0;
byte previousPINs[TOTAL_PORTS];
byte pinConfig[TOTAL_PINS];
byte portConfigInputs[TOTAL_PORTS];
int pinState[TOTAL_PINS];
unsigned long currentMillis;

@girlie_mac
Raspberry Pi
● $35 single-board computer
● Broadcom chip with
ARM-compatible CPU & GPU
● Runs Linux
● More language choices: C,
C++, Python...

@girlie_mac
But… I want to code in

@girlie_mac
Johnny-Five
● JavaScript robotics framework
● Works with
Arduino-compatible Boards
● IO plugins for more platform
supports
● http://johnny-five.io/

@girlie_mac
Awww, JavaScript all the way!

@girlie_mac
KittyCam
Raspberry Pi camera with cat facial
detection!
● Hardware: Raspberry Pi, camera, and PIR
sensor
● Software: Node.js + J5 + More
open-source goodies

@girlie_mac
Motion detected!
snap!
Mmm… donut

@girlie_mac
1. Raspberry Pi 3
2. 5MP Camera
Board Module
3. Pyroelectric
Infrared (PIR)
motion sensor
4. Female/Female
wires

@girlie_mac
Programming Raspberry Pi
Pre-installed on Raspbian OS:
C /
C++

@girlie_mac
Programming RPi with Node.js

@girlie_mac
Installing Node ARM
$ wget
https://nodejs.org/dist/v4.4.5/node-v4.4.5-l
inux-armv7l.tar.xz
$ tar -xvf node-v4.4.5-linux-armv7l.tar.xz
$ cd node-v4.4.5-linux-armv7l
$ sudo cp -R * /usr/local/

@girlie_mac
kittyCam.js
1. Detect motion
2. Take a photo
3. Cat facial detection
4. Store the photo in cloud
5. Real-time view on web
6. SMS the photo link
HELL,
YEAH!

@girlie_mac
kittyCam.js
1. Detect motion w/ Johnny-Five IR.Motion obj
2. Take a photo w/ Raspistill, command line tool
3. Cat facial detection w/ KittyDar
4. Store the photo in Cloudinary
5. Publish & subscribe the url to display on web
via PubNub
6. Send a text message via Nexmo

@girlie_mac
$ Raspistill
canvas
catDetect.js
app.js
kittyDar
Motion detected
take a photo
Store the photo if cats are detected
display photos on web browsers
real-time anywhere
Johnny-Five
w/ raspi-io
child
process
Returns url
Notify with SMS via

@girlie_mac
Johnny-Five w/ Raspi-io
const five = require('johnny-five');
const raspi = require('raspi-io');
let board = new five.Board({io: new raspi()});
board.on('ready', () => {
console.log('board is ready');
...
});

@girlie_mac
Motion
const five = require('johnny-five');
const raspi = require('raspi-io');
const board = new five.Board({io: new raspi()});
board.on('ready', function() {
// Create a new `motion` hardware instance
const motion = new five.Motion('P1-7');
...
});
a PIR is wired on pin 7
(GPIO 4)
VCC
Ground
Data

@girlie_mac
Raspistill Command Line Tool
$ raspistill -o myPhoto.jpg

@girlie_mac
PIR Sensor > Run Camera
const child_process = require('child_process');
board.on('ready', () => {
const motion = new five.Motion('P1-7');
motion.on('motionstart', () => { // Motion detected
let filename = 'photo/image_'+i+'.jpg';
let args = ['-w', '320', '-h', '240', '-o', filename, '-t', '1'];
let spawn = child_process.spawn('raspistill', args);
spawn.on('exit', function() {
console.log('A photo is saved as '+filename);
...
motion
detected!
Take a photo!
Spawns a new process w/ a given shell command

@girlie_mac
Processing Photo
spawn.on('exit', () => {
let imgPath = __dirname + '/' + filename;
// Child process: read the file and detect cats with KittyDar
let args = [imgPath];
let fork = child_process.fork(__dirname+'/detectCatsFromPhoto.js');
fork.send(args);
// the child process is completed
fork.on('message', (base64) => {
if(base64) {
uploadToCloud(base64); // Send to cloud storage
}
}); ...
Create another worker
by running a new
instance of V8 engine.

@girlie_mac
detectCatsFromPhoto.js
const kittydar = require('kittydar');
const Canvas = require('canvas');
process.on('message', (m) => {
fs.readFile(m[0], (err, data) => {
...
let canvas = new Canvas(w, h);
let ctx = canvas.getContext('2d');
ctx.drawImage(img, 0, 0, w, h, 0, 0, w, h);
let cats = kittydar.detectCats(canvas);
console.log('There are', cats.length, 'cats in this photo');
if(cats.length > 0) {base64Img = canvas.toDataURL();}
process.send(base64Img);
process.exit(0);
});
Running in an another process
a cat detected!
KittyDar

@girlie_mac
KittyDar
● Open-source JavaScript cat facial
detection written by Heather Arthur
● Takes a canvas obj & calculate the
locations of cats in the image
let cats = kittydar.detectCats(canvas);

@girlie_mac
KittyDar: Behind the Scene
1. Chops up the image up into many “windows”
2. Extracts data by measuring a set of gradients,
from light & dark in order to find edges
3. Compares the direction of these edges to the
edges found in known cat images
Neural network (JSON w/ vector data) is
pre-trained w/ thousands pics of cats & non-cats

@girlie_mac
Cat Facial Detection
http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf

@girlie_mac
Cat Facial Detection
The annotation data sequence:
(Number of points, default is 9), (Left Eye), (Right
Eye), (Mouth), (Left Ear-1), (Left Ear-2), (Left Ear-3),
(Right Ear-1), (Right Ear-2), and (Right Ear-3)

@girlie_mac
Cat Facial Data
9 247 94 294 92 273
127 213 81 207 29
247 53 286 50 320
20 322 74
00000561_012.jpg.cat00000561_012.jpg

@girlie_mac
Cat Facial Data Collection

@girlie_mac
Neural Network
Positive
Negative
JSON data
hog-descriptor
kittydar.jsCompare data

@girlie_mac
Send the Pic w/ Kitty to Cloud
const cloudinary = require('cloudinary');
// the child process is completed
fork.on('message', (base64) => {
if(base64) {
cloudinary.uploader.upload(base64, (result) => {
// Done! - Get the URL and do more stuff
});
} else deletePhoto(imgPath);
});

@girlie_mac
View Photos Real-time via Socket
const pubnub = require(pubnub);
Publish Subscribe

@girlie_mac
http://www.girliemac.com/RPi-KittyCam/

@girlie_mac
Send SMS via Nexmo
const Nexmo = require(nexmo);
let nexmo = new Nexmo({//config with API keys});
nexmo.message.sendSms(
FROM_NUMBER, TO_NUMBER, ' '+ url, options,
(err, responseData) => {
if (err) console.log(err);
else console.dir(responseData);
});

@girlie_mac
QA Team
Lead QA:
Jamie
Ginger Basil Alice YugiVenom
@kittenVenom

@girlie_mac
github.com/girliemac/RPi-KittyCam

@girlie_mac
Next Steps
● Upgrade Hardware
○ Raspberry Pi 3
○ NoIR Night Vision Camera
● Upgrade Node (was 0.12) & all dependencies
● More features (Maybe)
○ Cat Identification w/ RFID
○ Photo Booth w/ Filter effects & props

@girlie_mac
Next Project?
● Selfie bot
(à la Mannie the Selfie
Cat)
Mannie the Selfie Cat by
@yoremahm on Instagram
https://www.instagram.com/yoremahm/

@girlie_mac
Tack så mycket!
@girlie_mac
github.com/girliemac

@girlie_mac
Attribution:
Emoji by Emoji-One (CC-BY 4.0)

[Longer ver] From Software to Hardware: How Do I Track My Cat with JavaScript

Recommended

Recommended

More Related Content

Similar to [Longer ver] From Software to Hardware: How Do I Track My Cat with JavaScript

Similar to [Longer ver] From Software to Hardware: How Do I Track My Cat with JavaScript (20)

More from Tomomi Imura

More from Tomomi Imura (20)

Recently uploaded

Recently uploaded (20)

[Longer ver] From Software to Hardware: How Do I Track My Cat with JavaScript