Presented at Berlin.js May 31st 2018 for the #jsconfeu Special. Almost every video call begins with the same clumsy questions. Can you hear me now? Did I just turn off my camera instead of my mic now? But what if we could take the awkward troubleshooting out of the conversation, and solve it with code instead? In this talk, Ingvild Indrebø will give you a glimpse into aspects of WebRTC, WebAudio and Canvas, by showing you how she used these technologies to build a user-friendly and accessible tool to make sure you’re all set for your video call.

1. Can you hear me now?
Saying goodbye to clumsy video calls, thanks to code.
Ingvild Indrebø @IngvildIndrebo

3. People struggling
is no joke.

8. The technical
perspective:
• Get access to the camera
• Get access to the
microphone
• Play sound from the
speakers

9. The technical
perspective:
• Get access to the camera
• Get access to the
microphone
• Play sound from the
speakers

10. Is this really
a user problem?

11. Can you
see yourself?

12. WebRTC

13. WebRTC allows browsers to
communicate peer to peer.

14. WebRTC can be used for all
kinds of data transfer.

16. mediaStreamgetUserMedia Grant permission

17. getUserMedia
navigator.mediaDevices
.getUserMedia({ video: true, audio: true })

18. getUserMedia
navigator.mediaDevices
.getUserMedia({ video: true, audio: true })
.then(playStream)

19. getUserMedia
navigator.mediaDevices
.getUserMedia({ video: true, audio: true })
.then(playStream)
.catch(handleError);

20. <video id=“video” playsinline/>

21. <video id=“video” playsinline/>
function playStream(mediaStream) {
}

22. <video id=“video” playsinline/>
function playStream(mediaStream) {
const videoEl = document.getElementById('video');
}

23. <video id=“video” playsinline/>
function playStream(mediaStream) {
const videoEl = document.getElementById('video');
videoEl.srcObject = mediaStream;
}

24. <video id=“video” playsinline/>
function playStream(mediaStream) {
const videoEl = document.getElementById('video');
videoEl.srcObject = mediaStream;
videoEl.play();
}

26. Can you
hear me?

27. We already have access to
the microphone.
But can we hear anything?

28. “WebAudio is a system for controlling audio on
the Web, allowing developers to choose audio
sources, add effects to audio, create audio
visualizations, apply spatial effects (such as
panning) and much more…
https://developer.mozilla.org

29. Audio sources
• computed mathematically (such as OscillatorNode)
• recordings from audio/video files (<audio/>, <video/>)
• WebRTC MediaStream

30. Audio context
DestinationInput Analyze + Effects

31. WebAudio
const audioContext = new AudioContext();

32. WebAudio
const audioContext = new AudioContext();
const audioSrc = audioContext.createMediaStreamSource(
mediaStream
);

33. WebAudio
const audioContext = new AudioContext();
const audioSrc = audioContext.createMediaStreamSource(
mediaStream
);
const analyser = audioContext.createAnalyser();

34. WebAudio
const audioContext = new AudioContext();
const audioSrc = audioContext.createMediaStreamSource(
mediaStream
);
const analyser = audioContext.createAnalyser();
audioSrc.connect(analyser);

35. That we know it
works isn’t enough.

36. But…
what is sound?

37. Sound wave

38. Fourier
transformation

40. Volume

41. Amplitude

42. function canWeHearYou(analyser) {
const dataArray = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(dataArray);
}

43. function canWeHearYou(analyser) {
const dataArray = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(dataArray);
}
dataArray: [3, 25, 179, 5, …]

44. dataArray = [ , , ,]

45. 69-90 hz47-68 hz23-46 hz
Frequencies
0-22hz
Each bucket
contains the total
volume
of that frequency
range

46. 69-90 hz47-68 hz23-46 hz
Frequencies
0-22hz
Each bucket
contains the total
volume
of that frequency
range

47. function canWeHearYou(analyser) {
const dataArray = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(dataArray);
let sum = 0;
sum += dataArray.reduce((a, b) => a + b, 0);
return sum > 0;
}

48. Visualization
• Our DataArray gives us a
sequence of numbers
• We use Canvas to draw
them

49. <canvas id=“canvas”/>
draw(dataArray) {

50. <canvas id=“canvas”/>
draw(dataArray) {
const canvas = document.getElementById(‘canvas');

51. <canvas id=“canvas”/>
draw(dataArray) {
const canvas = document.getElementById(‘canvas');
const canvasCtx = canvas.getContext('2d');

52. <canvas id=“canvas”/>
draw(dataArray) {
const canvas = document.getElementById(‘canvas');
const canvasCtx = canvas.getContext('2d');
canvasCtx.beginPath();

53. <canvas id=“canvas”/>
draw(dataArray) {
const canvas = document.getElementById(‘canvas');
const canvasCtx = canvas.getContext('2d');
canvasCtx.beginPath();
canvasCtx.moveTo(0, 0);

54. draw(dataArray) {
const sliceWidth = CANVAS_WIDTH/dataArray.length;
let x = 0;
for(let y of dataArray) {
canvasCtx.lineTo(x, y);
x += sliceWidth;
}

55. draw(dataArray) {
const sliceWidth = CANVAS_WIDTH/dataArray.length;
let x = 0;
for(let y of dataArray) {
canvasCtx.lineTo(x, y);
x += sliceWidth;
}
canvasCtx.lineTo(canvas.width, 0);

56. Demo

57. ~80-15khz
What a mic can hear
~20-20khz
What a human can hear

58. Can you hear
the sound?

59. Audio sources
• computed mathematically (such as OscillatorNode)
• recordings from audio/video files (<audio/>, <video/>)
• WebRTC MediaStream

60. WebAudio
<audio id=“callingSound” src=“…”/>

61. WebAudio
<audio id=“callingSound” src=“…”/>
const audioContext = new AudioContext();

62. WebAudio
<audio id=“callingSound” src=“…”/>
const audioContext = new AudioContext();
const audioEl = document.getElementById('callingSound');

63. WebAudio
<audio id=“callingSound” src=“…”/>
const audioContext = new AudioContext();
const audioEl = document.getElementById('callingSound');
const audioSrc =
audioContext.createMediaElementSource(audioEl);

64. WebAudio
<audio id=“callingSound” src=“…”/>
const audioContext = new AudioContext();
const audioEl = document.getElementById('callingSound');
const audioSrc =
audioContext.createMediaElementSource(audioEl);
const analyser = audioContext.createAnalyser();

65. WebAudio
<audio id=“callingSound” src=“…”/>
const audioContext = new AudioContext();
const audioEl = document.getElementById('callingSound');
const audioSrc =
audioContext.createMediaElementSource(audioEl);
const analyser = audioContext.createAnalyser();
audioSrc.connect(analyser);

68. There are no
user problems.

69. Thank you!
confrere.com/test
Ingvild Indrebø
@IngvildIndrebo
ingvild@confrere.com