Presentation at Devoxx Poland 2016 - 22/06/2016 at Krakow by Maximiliano Firtman

1. Maximiliano Firtman
@firt
the Physical World meets the Web

2. mobile+web developer & trainer

7. Slides, links and tests
firt.mobi/talks/physical

8. questions
yes, please
QA at the end

9. Let’s Start!

10. Agenda
1- Mobile and IoT
2- The Physical Web
3- Progressive Web Apps
4- Connecting through Web APIs
5- VR|MR Worlds and the Web

11. 1- Mobile and IoT

21. The web comes to save us

22. 1- Mobile and IoT
- Every gadget, sensor, device will
have mobile support
- We don’t want to install apps
- We don’t want to make everything public

23. 2- The Physical Web

24. The Physical Web is an open
approach to enable quick and
seamless interactions with physical
objects and locations.

25. 1- Open Spec
2- Beacons BLE
3- Also mDNS over WiFi
Goal: broadcast URLs

26. image from physical web website
Eddystone protocol

27. 1- Buy beacons
2- Install “Beacon Toy” for Android
3- Also mDNS over WiFi
Start broadcasting URLs!

30. Chrome on Android (49+)
Chrome on iOS
Current support

32. Live demo

33. 2- The Physical Web
- Broadcast URL in physical places
- Open standard
- Anything can broadcast a URL

34. 3- Progressive Web Apps

35. Progressive Web Apps
1- Open Definition
2- Embraced by Google
3- Followed by Firefox, Opera,
Samsung and Microsoft

36. A Progressive Web App is a model for
creating app-like experiences using
latest Web technologies progressively

37. Features
1- Instant Loading
2- Discoverable
3- Network independent
4- Responsive
5- Installable
6- Secure
7- Linkable
8- Re-engageable
9- Works everywhere
10- Fast

38. progressive
enhancement
progressive
enhancement

39. Progressive Web Apps
1- It’s a website!
2- Adds native installation
3- Adds Web Push Notifications
4- Adds hardware & platform access

40. Progressive Web Apps
FUTUREPARTIAL
But they work anyway, anywhere!

41. Live demo

42. Progressive Web Apps
1- HTML5
2- Service Workers
3- Web App Manifest
4- Web Push Notifications
5- (optionally) Hardware APIs

43. 3- Progressive Web Apps
- Best of native and web worlds
- We can serve app-like experiences
- The user can use it from the browser,
or install it
- Push messages for re-engagement

44. 4- Web APIs

45. Device Sensors
Web Audio
Web Bluetooth
web apis

46. DEVICE SENSORS
1- Mostly useful on mobile devices
2- Use device’s hardware & sensors
3- Different Levels
a- Mature APIs
b- Experimental APIs
c- Future APIs

47. a- Mature APIs
Geolocation (GPS & WiFi)
Accelerometer and Magnetometer
Compass
Camera (still picture)

48. Know more about these APIs…
Thursday 14.30
Breaking the Limits of the Web

49. b- Experimental APIs
Get User Media
Ambient Light
Speech Recognition and Synthesis
WebMIDI

50. Get User Media
Get the stream video from the camera
Recognize objects, faces, QR codes
Augmented reality

51. <code>
navigator.getUserMedia(
{video:true,audio:true},
function(stream){
var videoSource =
URL.createObjectURL(stream);
},
function(){
alert("Error getting video feed");
}
);
JAVASCRIPT

52. <code>
navigator.getUserMedia(
{video:true,audio:true},
function(stream){
var source = URL.createObjectURL(stream);
},
function(){
alert("Error getting video feed");
}
);
JAVASCRIPT

53. Get User Media

54. Live demo

55. Web Midi
Connect to music devices:
synthesizers, keyboards, guitars, drum
machines, lightning systems
Send and receive MIDI messages

56. <code>
navigator.requestMIDIAccess({sysex: false})
.then(onMIDISuccess, onMIDIFailure);
function onMIDISuccess(midi) {
var inputs = midi.inputs.values();
for (var input=…) {
input.value.addEventListener(“midimessage”,
function(event) {
// event.data has the bytes
});
}
}
JAVASCRIPT

57. <code>
navigator.requestMIDIAccess({sysex: false})
.then(onMIDISuccess, onMIDIFailure);
function onMIDISuccess(midi) {
var inputs = midi.inputs.values();
for (var input=…) {
input.value.addEventListener(“midimessage”,
function(event) {
// event.data has the bytes
});
}
}
JAVASCRIPT

58. <code>
navigator.requestMIDIAccess({sysex: false})
.then(onMIDISuccess, onMIDIFailure);
function onMIDISuccess(midi) {
var inputs = midi.inputs.values();
for (var input=…) {
input.value.addEventListener(“midimessage”,
function(event) {
// event.data has the bytes
});
}
}
JAVASCRIPT

59. <code>
navigator.requestMIDIAccess({sysex: false})
.then(onMIDISuccess, onMIDIFailure);
function onMIDISuccess(midi) {
var inputs = midi.inputs.values();
for (var input=…) {
input.value.addEventListener(“midimessage”,
function(event) {
// event.data has the bytes
});
}
}
JAVASCRIPT

60. Web Midi

61. Ambient Light
Get info about current light conditions
Measures in lux

62. <code>
window.addEventListener('devicelight',
function(event) {
var lux = event.value;
});
JAVASCRIPT

63. <code>
window.addEventListener('devicelight',
function(event) {
var lux = event.value;
});
JAVASCRIPT

64. Live demo

65. Ambient Light
FLAG FUTURE

66. Speech Synthesis and Recognition
Let us interact with the user with voice
Synthesis: let the Web speak
Recognition: let the Web listen to user

67. <code>
var u = new SpeechSynthesisUtterance();
u.rate = 1;
u.volume = 1;
u.pitch = 1;
u.text = “Hello World”;
u.lang = “en-GB”;
speechSynthesis.speak(u);
JAVASCRIPT

68. <code>
var u = new SpeechSynthesisUtterance();
u.rate = 1;
u.volume = 1;
u.pitch = 1;
u.text = “Hello World”;
u.lang = “en-GB”;
speechSynthesis.speak(u);
JAVASCRIPT

69. <code>
var u = new SpeechSynthesisUtterance();
u.rate = 1;
u.volume = 1;
u.pitch = 1;
u.text = “Hello World”;
u.lang = “en-GB”;
speechSynthesis.speak(u);
JAVASCRIPT

70. <code>
recognition = new SpeechRecognition();
recognition.continuous = true;
recognition.interimResults = true;
recognition.addEventListener(“result”, function(e) {
if (e.type==“result") {
for (…) {
var result = e.results[i][0];
var confidence = result.confidence*100);
var text = result.transcript;
}
}
};
JAVASCRIPT

71. <code>
recognition = new SpeechRecognition();
recognition.continuous = true;
recognition.interimResults = true;
recognition.addEventListener(“result”, function(e) {
if (e.type==“result") {
for (…) {
var result = e.results[i][0];
var confidence = result.confidence*100);
var text = result.transcript;
}
}
};
JAVASCRIPT

72. <code>
recognition = new SpeechRecognition()
recognition.continuous = true;
recognition.interimResults = true;
recognition.addEventListener(“result”, function(e) {
if (e.type==“result") {
for (…) {
var result = e.results[i][0];
var confidence = result.confidence*100);
var text = result.transcript;
}
}
};
JAVASCRIPT

73. <code>
recognition = new SpeechRecognition()
recognition.continuous = true;
recognition.interimResults = true;
recognition.addEventListener(“result”, function(e) {
if (e.type==“result") {
for (…) {
var result = e.results[i][0];
var confidence = result.confidence*100);
var text = result.transcript;
}
}
};
JAVASCRIPT

74. <code>
recognition = new SpeechRecognition()
recognition.continuous = true;
recognition.interimResults = true;
recognition.addEventListener(“result”, function(e) {
if (e.type==“result") {
for (…) {
var result = e.results[i][0];
var confidence = result.confidence*100);
var text = result.transcript;
}
}
};
JAVASCRIPT

75. Live demo

76. Speech Synthesis
FLAG FUTURE

77. Speech Recognition

78. c- Future APIs
Geofencing
WebUSB
WebNFC

79. Geofencing
Get a web or PWA activation when
user enters or leaves a geo area

80. WebUSB
Let us connect with any USB hardware
connected to the device

81. WebNFC
Near Field Communication
Access to Tap to Pay, Tap to Share, etc.

82. WEB AUDIO
1- Generate dynamic audio
2- Listens to low level audio
3- Useful for ultrasound
communication with devices!

83. <code>
// Sending
ssocket = new SonicSocket({alphabet: '0123456789'});
ssocket.send(‘31415');
// Receiving
sserver = new SonicServer({alphabet: '0123456789'});
sserver.on('message', function(message) {
// Expect message to be '31415'.
console.log(message);
});
sserver.start()
JAVASCRIPT
Using Sonicnet.js library

84. <code>
// Sending
ssocket = new SonicSocket({alphabet: '0123456789'});
ssocket.send(‘31415');
// Receiving
sserver = new SonicServer({alphabet: '0123456789'});
sserver.on('message', function(message) {
// Expect message to be '31415'.
console.log(message);
});
sserver.start()
JAVASCRIPT
Using Sonicnet.js library

85. <code>
// Sending
ssocket = new SonicSocket({alphabet: '0123456789'});
ssocket.send(‘31415');
// Receiving
sserver = new SonicServer({alphabet: '0123456789'});
sserver.on('message', function(message) {
// Expect message to be '31415'.
console.log(message);
});
sserver.start()
JAVASCRIPT
Using Sonicnet.js library

86. <code>
// Sending
ssocket = new SonicSocket({alphabet: '0123456789'});
ssocket.send(‘31415');
// Receiving
sserver = new SonicServer({alphabet: '0123456789'});
sserver.on('message', function(message) {
// Expect message to be '31415'.
console.log(message);
});
sserver.start()
JAVASCRIPT
Using Sonicnet.js library

87. <code>
// Sending
ssocket = new SonicSocket({alphabet: '0123456789'});
ssocket.send(‘31415');
// Receiving
sserver = new SonicServer({alphabet: '0123456789'});
sserver.on('message', function(message) {
// Expect message to be '31415'.
console.log(message);
});
sserver.start()
JAVASCRIPT
Using Sonicnet.js library

88. Live demo

89. WEB AUDIO

90. WEB BLUETOOTH
1- Scan BLE devices
2- Scan services available
3- Connect to these services
4- Send and receive data

91. WEB BLUETOOTH
Huge access for IoT
Mixed with the Physical Web
Better with Bluetooth 5.0

92. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

93. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

94. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

95. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

96. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

97. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

98. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

99. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

100. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
}).then(function (device) {
return device.gatt.connect();
}).then(function (server) {
return server.getPrimaryService(“heart_rate”);
}).then(function (service) {
return service.getCharacteristic(
“heart_rate_control_measurement”);
}).then(function (ch) {
return ch.startNotifications().then(function() {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});;
JAVASCRIPT

101. <code>
navigator.bluetooth.requestDevice({
filters: [‘services’: ‘heart_rate’]
})
.then(device => { device.gatt.connect() })
.then(server => server.getPrimaryService(“heart_rate”); })
.then(service => service.getCharacteristic(
“heart_rate_control_measurement”);})
.then(ch => return ch.startNotifications().then(() => {
ch.addEventListener('characteristicvaluechanged',
function() { event.target.value });
});
});
JAVASCRIPT

102. Heart Rate Sensor Demo

103. BB8 Controller

104. Drone controller

105. Live demo

106. WEB BLUETOOTH
FLAG

107. 4- Web APIs
- Device Sensor APIs:
mature, experimental, future
- Web Audio
- Web Bluetooth
- TV Streaming

108. 5- VR|MR World
and the Web

109. Virtual Reality
Mixed Reality
new worlds

110. VR and the Web
Samsung Internet Browser for Gear VR
Oculus Web Browser
Third party browsers

111. Live demo

113. WebVR
Access to VR devices in your browser
(Oculus, Cardboard, Vive)
Experimental support
Available as a polypill

114. WebVR
BUILD DEV FLAG

115. WebVR
DEVPOLYFILL POLYFILL

116. MR and the Web
Mixed Reality with Windows Holographic
Hololens

117. Live demo

118. MR and the Web
We are in very early stages
We need content to get out of the flat
hologram

119. - We are in early stages
- We will need new APIs and concepts
- VR and MR mixed with Physical Web
and Web Bluetooth
5- VR|MR World
and the Web

120. the physical world meets the web
1- Mobile and IoT
2- The Physical Web
3- Progressive Web Apps
4- Connecting through Web APIs
5- VR|MR World and the Web

121. Foto de freefoto.com
firtman@gmail.com
@firt
firt.mobi/hpmwfirt.mobi/mh5