Mobile Device APIs

1. (Mobile) Device APIs James Pearce @jamespearce http://tripleodeon.com

2. What are Device APIs? “Client-side APIs that enable the development of Web Applications that interact with device hardware” ...and other capabilities outside of the browser’s traditional remit

3. Why are they important?• They’re not... ...if you’re happy with a web of documents• They are... ... if you want the the web to be competitive with native app platforms

4. HistoryCurrent Work Use Cases

5. WTAI Make Call<a href="wtai://wp/mc;16505551234"> Call us</a><a href="call.wmls#makeCall()"> Call us</a>// call.wmlsextern function makeCall() { WTAPublic.makeCall("16505551234");} http://www.wapforum.org/what/technical/wtai-30-apr-98.pdf

6. WTAI Call Control<a href="calls.wmls#setupCall()">Call us!</a><a href="calls.wmls#acceptCall()">Brring!</a><a href="wtai://cc/sd;1*2*3456">Enter code</a>// calls.wmlsextern function setupCall() { WTACallCont.setup("16505551234", 1);}extern function acceptCall() { WTACallCont.accept("1", 1);}

7. WTAI Messaging// sms.wmlsextern function sendMsg() { WTANetText.send("16505551234", "WML rocks");}extern function readMsg(i) { var sms = WTANetText.read(i); return WTANetText.getFieldValue( sms, "body" );}

8. WTAI Contacts// contacts.wmlsextern function addFriend(i, num, name) { WTAPhoneBook.write(i, num, name);}// return structs of contact detailsextern function getFriendById(i) { return WTAPhoneBook.read("i", i);}extern function getFriendByName(name) { return WTAPhoneBook.read("t", name);}

9. I may be laboring the point But that was 1998... ... and you still can’t do any of this with a contemporary mobile browser

10. BONDI“enables web based content to access nativedevice capability, intermediated through arobust, but ﬂexible security framework” http://www.omtp.org/1.11

11. Messagingvar sms = bondi.messaging.createSMS({ to: ["16505551234"], body: "Home soon"});var mms = bondi.messaging.createMMS({ to: ["16505551234"], subject: "I will be...", body: "...home soon"});

12. Messagingbondi.messaging.sendSMS( function () {}, // success function () {}, // error sms, true);bondi.messaging.sendMMS( function () {}, // success function () {}, // error mms, true);

13. Messagingbondi.messaging.subscribeToSMS( function () {}, // success function () {}, // error function () { // listener alert("They replied!"); }, { from: "16505551234" // filter }, true // exclusive);

14. Contactsbondi.pim.contact.getAddressBooks( function(books) { // success var contact = books[0].createContact({ name: James Pearce, emails: [{email:j@tripleodeon.com}], phoneNumbers: [{number:16505551234}] }); }, function () {} // error);

15. Telephonybondi.telephonyLog.findLogEntries( function () {}, // success function () {}, // error, [ bondi.telephonyLog.MISSED_CALLS_FOLDER, bondi.telephonyLog.RECEIVED_CALLS_FOLDER ], { phoneNumber: "16505551234" }); * Yes, it’s true: you can’t actually make or receive calls

16. App Launcherbondi.applauncher.launchApplication( function () {}, // success function () {}, // error "tel:16505551234");// mailto:...// sms:...// file:...

17. On one hand...• Security-conscious architecture• Comprehensive API support• Consistent API patterns• Conformance speciﬁcations• Reasonable test coverage http://www.omtp.org/1.11/apis

18. On the other...

19. bondi.*deviceapis.*

20. WAC 2.0• Accelerometer • File system• Calendar • Geolocation• Camera • Messaging• Contacts • Orientation• Core • Tasks• Device interaction • Viewport• Device status • Webview http://specs.wacapps.net/

21. Which is all awesome Except...

22. Device APIs Working Group “to create client-side APIs that enable the development of Web Applications and Web Widgets that interact with devices services such as Calendar, Contacts, Camera, etc” Joint efforts with WebRTC Working Group, WebApps Working Group http://www.w3.org/2009/dap/

23. W3C Device APIs• Media Capture(HTML) • Vibration• Media Capture(gUM) • Calendar• Contact • Permissions• Messaging • Menu• Battery Status • Gallery• Network Information • System info• Sensor • Media Stream

24. Media Capture (HTML)<input type="file" accept="image/*" /><!-‐-‐ The accept attribute is a hint. Specification suggests that it can be used to offer alternative picker UIs.-‐-‐> Public working draft, http://www.w3.org/TR/2011/WD-html-media-capture-20110414

25. Brad Lassey, September 2008

26. Media Capture (HTML)<input type="file" accept="image/*" capture="camera"/><!-‐-‐ Again, a hint. Valid options are: camera, camcorder, microphone, filesystem-‐-‐>

27. Media Capture (HTML)<input type="file" accept="image/*" capture="camera" id="photo"/>var photo = document.getElementById(photo);photo.files[0].getFormatData( function (data) {}, // success function () {} // error);// file is a MediaFile, extending File// data is a MediaFileData

28. MediaFileDatainterface MediaFileData { attribute DOMString codecs; attribute unsigned long bitrate; attribute unsigned long height; attribute unsigned long width; attribute float duration;}; ...seems to be under discussion

29. Media capture (gUM)navigator.getUserMedia( { audio: true, video: false }, function (stream) {}, // success function () {} // error);// stream is a LocalMediaStream Editors draft, http://dev.w3.org/2011/webrtc/editor/getusermedia.html

30. LocalMediaStreamvar recorder = stream.record();// recorder is a MediaStreamRecorderrecorder.getRecordedData( function (file) {} // success);// file is a FileURL.createObjectURL(stream);// a Blob URI for, say <video src=... />

31. Messagingnavigator.sendMessage( "sms:16505551234?" + "body=JavaScript%20says%20hi", [], // attachments as File array function () {}, // success function () {} // error);// also mms: and mailto:// oh... were you looking for receipt? Editors draft, http://dev.w3.org/2009/dap/messaging/

32. Sensorsnavigator.findSensors().onsuccess = function() { this.result.forEach(function (sensor) { // ... });};var sensor = new SensorConnection(Temperature);sensor.addEventListener(sensordata, function(e) { if(e.data > 20.0) { window.console.log(Hawt); }});sensor.startWatch({ interval: 2000 }); Editors draft, http://dev.w3.org/2009/dap/system-info/Sensors.html

33. Sensors• Temperature, ºC• AmbientLight, Lux• AmbientNoise, dbA• MagneticField, uTesla• Proximity, cm• AtmPressure, kP• RelHumidity, %

34. Contactsnavigator.contacts.find( [ // fields name, emails ], function (contacts) {}, // success function () {}, // error { // find options filter: James, multiple: true }); Editors draft, http://w3c-test.org/dap/contacts/

35. Contactinterface Contact { readonly attribute DOMString id; attribute DOMString? displayName; attribute ContactName? name; attribute DOMString? nickname; attribute ContactField[]? phoneNumbers; attribute ContactField[]? emails; attribute ContactAddress[]? addresses; attribute ContactField[]? ims; attribute ContactOrganization[]? organizations; attribute Date? birthday; attribute DOMString? note; attribute ContactField[]? photos; attribute DOMString[]? categories; attribute ContactField[]? urls;};

36. Cool, but... “The ability to add and update contact information is not a function of the API provided in this speciﬁcation.”

37. Holy Contacts Editing WTFvar vcard = BEGIN:VCARDrn + VERSION:3.0rn + UID: + contact.id + rn + N:Pearce;Jamesrn + FN:James Pearcern + END:VCARD;document.getElementById(vcard).href = data:text/x-‐vcard;charset=utf-‐8, + encodeURIComponent(vcard);

38. WebAPI “We are aiming at providing all the necessary APIs to build a basic HTML5 phone experience within the next 3-6 months” https://wiki.mozilla.org/WebAPI

39. WebAPI• Telephony • Embedded browser• Messaging • Battery• Contacts • Device Status• Camera • Settings• Network • Mouse Lock• USB • Vibrator

40. Hopefully W3C++• “Our proposal might or might not look like it for the moment”

41. Telephony!var tcall = navigator.telephony.dial("16505551234");tcall.readyState; // dialing, ringing, busy, connecting, // connected, disconnecting, disconnected, // incomingtcall.onconnected = function () {}; // onbusy, ondisconnected etctcall2.answer(); http://www.youtube.com/watch?v=RuIQskGD3u0#t=19s

42. Contacts “We need read/write”

43. USBvar usb = new MozUSBManager();usb.addEventListener("attachdevice", function () {});usb.addEventListener("detachdevice", function () {});usb.claimDevice({ deviceClass: ..., deviceSubClass: ...});

44. We deserve Device APIs...• As forward-looking as WTAI• As well-structured as BONDI• As comprehensive as WAC• As respectable as W3C DAP• As agile as WebAPI

45. And we’ve got...

46. PhoneGap• Accelerometer • Events• Camera • File• Capture • Geolocation• Compass • Media• Connection • Notiﬁcation• Contacts • Storage• Device

47. Use Cases

48. Useless Demos https://github.com/jamesgpearce/compios5

49. Fixing layout bugs, FMLwindow.addEventListener("deviceorientation", function (e) { orientation = Math.abs(w.orientation); rotation = Math.abs(e.gamma); if (rotation > 8 && orientation === 0) { if (enabled) { disableZoom(); } else { restoreZoom(); } } }, false); http://vimeo.com/adactio/iosbug https://github.com/scottjehl/iOS-Orientationchange-Fix

50. Camera• Proﬁle pictures• Review sites• Photo sharing• Recognition • Bar-codes • Text • Faces

51. Filters!$(img).vintage({ vignette: { black: 0.8, white: 0.2 }, noise: 20, screen: { red: 12, ... }, desaturate: 0.05}); https://github.com/rendro/vintageJS

52. Augmented reality? http://fhtr.org/JSARToolKit/

53. Contacts & calendaring• Add business details to address book• Web-based messaging• Facial recognition again...• Presence• Context-sensitivity of content

54. Messaging• Sharing• Notiﬁcations• Ofﬂine transport fallback• Frictionless web sign-on

55. Expect more

56. I still want... • Audio processing • Bluetooth • RFID & NFC • Both cameras • Memory & power • Native recognition

57. window.addEventListener( appointmentdue, function(appt) { if (device.near(WORK)) { siri.remind( contacts.get(Peter), calendar.getDetails(appt) ); } }, false); PS: does not work

58. James Pearce @jamespearcehttp://tripleodeon.com