COMP 4026 Lecture4: Processing and Advanced Interface Technology
â˘
3 likesâ˘900 views
Lecture 4 from the 2016 COMP 4026 course on Advanced Human Computer Interaction taught at the University of South Australia. Taught by Mark Billinghurst, and containing material about Processing and various advanced Human Computer Interfaces.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to COMP 4026 Lecture4: Processing and Advanced Interface Technology
Similar to COMP 4026 Lecture4: Processing and Advanced Interface Technology (20)
More from Mark Billinghurst
More from Mark Billinghurst (20)
Recently uploaded
Recently uploaded (20)
COMP 4026 Lecture4: Processing and Advanced Interface Technology
- 1. LECTURE 4: PROCESSING AND ADVANCED INTERFACES COMP 4026 â Advanced HCI Semester 5 - 2016 Mark Billinghurst University of South Australia August 18th 2016
- 2. RECAP
- 3. App Inventor â˘âŻ http://appinventor.mit.edu/ â˘âŻ http://appinventor.org/ â˘âŻ Visual Programming for Android Apps â˘âŻ Features â˘âŻ Access to Android Sensors â˘âŻ Multimedia output â˘âŻ Drag and drop web based interface â˘âŻ Designer view â app layout â˘âŻ Blocks view â program logic/control
- 6. Interactive Coding - Processing âŞâŻProgramming tool for Artists/Designers âŞâŻhttp://processing.org âŞâŻEasy to code, Free, Open source, Java based âŞâŻ2D, 3D, audio/video support âŞâŻProcessing For Android âŞâŻhttp://wiki.processing.org/w/Android âŞâŻStrong Android support, builds .apk file
- 8. Buchenau, M., & Suri, J. F. (2000, August). Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 424-433). ACM.
- 9. Experience Prototyping The experience of even simple artifacts does not exist in a vacuum but, rather, in dynamic relationship with other people, places and objects. Additionally, the quality of peopleâs experience changes over time as it is influenced by variations in these multiple contextual factors.
- 10. Role Playing
- 11. littleBits - http://littlebits.cc/ â˘âŻ Plug and play hardware components â˘âŻ Sensors, input, output â˘âŻ Rapid design with hardware
- 13. Interaction Design Process Evaluate (Re)Design Identify needs/ establish requirements Build an interactive version Final Product
- 14. When to evaluate? â˘âŻOnce the product has been developed â˘âŻ pros : rapid development, small evaluation cost â˘âŻ cons : rectifying problems â˘âŻDuring design and development â˘âŻ pros : find and rectify problems early â˘âŻ cons : higher evaluation cost, longer development design implementationevaluation redesign & reimplementation design implementation
- 15. Four evaluation paradigms â˘âŻâquick and dirtyâ â˘âŻusability testing (lab studies) â˘âŻfield studies â˘âŻpredictive evaluation
- 16. Characteristics of approaches Usability testing Field studies Predictive Users do task natural not involved Location controlled natural anywhere When prototype early prototype Data quantitative qualitative problems Feed back measures & errors descriptions problems Type applied naturalistic expert
- 17. Evaluation approaches and methods Method Usability testing Field studies Predictive Observing x x Asking users x x Asking experts x x Testing x Modeling x
- 18. CASE STUDY
- 19. Interaction Design Process Evaluate (Re)Design Identify needs/ establish requirements Build an interactive version Final Product
- 20. MOBILE AUGMENTED REALITY FOR SPATIAL NAVIGATION Sharon Brosnan 0651869 Bachelor of Science in Digital Media Design
- 21. STORYBOARD
- 23. POST IT PROTOTYPING First Dra. Camera View with 3D Second Dra. Third Dra. â˘âŻ Selec,on highlighted in blue â˘âŻ Home bu<on added for easy naviga,on to main menu
- 25. WIKITUDE â˘âŻ Popular augmented reality browser for mobile devices â˘âŻ Mapping â˘âŻ Point of Interest abilities â˘âŻ Multiplatform â˘âŻ Shows the points of interest of Bunratty Folk Park â˘âŻ Markers can be selected in and an information pop-up appears
- 26. VIDEO PROTOTYPE !⯠Flexible tool for capturing the use of an interface !⯠Elaborate simula,on of how the naviga,onal aid will work !⯠Does not need to be realis,c in every detail !⯠Gives a good idea of how the ďŹnished system will work
- 27. MORE PROCESSING
- 30. Processing - Notes â˘âŻ Language of Interaction â˘âŻ Physical Manipulation â˘âŻ Input using code â˘âŻ Mouse Manipulation â˘âŻ Presence, location, image â˘âŻ Haptic interfaces and multi-touch â˘âŻ Gesture â˘âŻ Voice and Speech
- 32. Basic Parts of a Sketch /* Notes comment */! //set up global variables! float moveX = 50;! ! //Initialize the Sketch! void setup (){! }! ! //draw every frame! void draw(){! }!
- 33. Sample Drawing int m = 0;! float s = 0;! ! void setup(){! size(512,512);! background(255);! }! ! void draw (){! fill(255,0,0);! ellipse(mouseX,mouseY,s,s);! }! ! void mouseMoved(){! s = 40 + 20*sin(++m/10.0f);! }!
- 35. Drawing â˘âŻ draw() gets called as fast as possible, unless a frameRate is specified â˘âŻ stroke() sets color of drawing outline â˘âŻ fill() sets inside color of drawing â˘âŻ mousePressed is true if mouse is down â˘âŻ mouseX, mouseY - mouse position !void draw() { ! !stroke(255); ! !if(mousePressed) {! ! !line(mouseX, mouseY, pmouseX, pmouseY);! ! !}! !}!
- 36. Processing and Drawing â˘âŻ Basic Shapes rect(x, y, width, height)! ellipse(x, y, width, height)! line(x1, y1, x2, y2), line(x1, y1, x2, y2, z1, z2)! â˘âŻ Filling shapes - fill( ) fill(int gray), fill(color color), fill(color color, int alpha)! â˘âŻ Curve â˘âŻ Draws curved lines â˘âŻ Vertex â˘âŻ Creates shapes (beginShape, endShape)
- 37. Vertex Demo void setup(){! size(400,400);! }! ! void draw(){! background(255);! fill(0);! beginShape();! vertex(0,0);! vertex(400,400);! vertex(mouseX,mouseY);! endShape();! }!
- 38. Curve Demo void setup(){! size(400,400);! }! ! void draw(){! background(255);! fill(0);! ! int xVal = mouseX*3-100;! int yVal = mouseY*3-100;! ! curve(xVal, yVal, 100, 100, 100, 300, xVal, yVal);! curve(xVal, yVal, 100, 300, 300, 300, xVal, yVal);! curve(xVal, yVal, 300, 300, 300, 100, xVal, yVal);! curve(xVal, yVal, 300, 100, 100, 100, xVal, yVal);! ! }!
- 39. Class and Objects â˘âŻsee http://processing.org/learning/objects/ â˘âŻObject â˘âŻ grouping of multiple related properties and functions â˘âŻObjects are defined by Object classes â˘âŻEg Car object â˘âŻ Data â˘âŻ colour, location, speed â˘âŻ Functions â˘âŻ drive(), draw()
- 40. Classes â˘âŻfour elements: name, data, constructor, and methods. â˘âŻName class myName { }! â˘âŻData â˘âŻ collection of class variables â˘âŻConstructor â˘âŻ run when object created â˘âŻMethods â˘âŻ class functions
- 41. Car Class
- 43. Class Usage // Step 1. Declare an object.! Car myCar;! ! void setup() { ! // Step 2. Initialize object.! myCar = new Car(); ! } ! ! void draw() { ! background(255); ! // Step 3. Call methods on the object. ! myCar.drive(); ! myCar.display(); ! }!
- 45. Constructing Objects â˘âŻ One Car Car myCar= new Car(); ! â˘âŻ Two Cars !// Creating two car objects ! !Car myCar1 = new Car(); ! !Car myCar2 = new Car(); ! â˘âŻ One car with initial values Car myCar = new Car(color(255,0,0),0,100,2); !
- 46. Modifying Constructor Car(color tempC, float tempXpos, float tempYpos, float tempXspeed) ! { ! c = tempC; ! xpos = tempXpos; ! ypos = tempYpos; ! xspeed = tempXspeed; ! }!
- 47. Mouse Interaction â˘âŻMouse position â˘âŻ mouseX, mouseY variables â˘âŻMouse Interaction â˘âŻ mousePressed() â˘âŻ mouseReleased() â˘âŻ mouseDragged() â˘âŻAdd in own code void mouseDragged(){! line(pmouseX, pmouseY, mouseX, mouseY);! }!
- 48. Keyboard Interaction â˘âŻCheck keyPressed variable in draw() method !void draw(){! ! !if(keyPressed){! ! ! !print(" you pressed " +key);! ! !}! }! â˘âŻUse keyPressed() method !void keyPressed(){! ! !print(" you're pressing "+key);! !}!
- 50. Importing Libraries â˘âŻ Can add functionality by Importing Libraries â˘âŻ java archives - .jar files â˘âŻ Include import code import processing.opengl.*;! â˘âŻ Popular Libraries â˘âŻ Minim - audio library â˘âŻ OCD - 3D camera views â˘âŻ Physics - physics engine â˘âŻ bluetoothDesktop - bluetooth networking
- 52. Graphical Controls â˘âŻUse ControlP5 Library â˘âŻ http://www.sojamo.de/libraries/controlP5/ â˘âŻAdd graphical controls â˘âŻ Buttons, sliders, etc â˘âŻ Support for OSC (Open Sound Controller) â˘âŻUse ControlP5 class import controlP5.*;! addButton(name, value, x, y, width, height);! â˘âŻEvent Handing
- 53. Interface Elements â˘âŻInterfascia â˘âŻhttp://www.superstable.net/interfascia/ â˘âŻGUI Library for Processing â˘âŻButtons â˘âŻCheck boxes â˘âŻTextfields â˘âŻProgress bar
- 55. Graphical Controls â˘âŻ Use ControlP5 Library â˘âŻ http://www.sojamo.de/libraries/controlP5/ â˘âŻ Add graphical controls â˘âŻ Buttons, sliders, etc â˘âŻ Support for OSC (Open Sound Controller) â˘âŻ Use ControlP5 class import controlP5.*;! addButton(name, value, x, y, width, height);! â˘âŻ Event Handing
- 57. Programming Graphics â˘âŻTransformations â˘âŻCreating motion and animation â˘âŻBitmaps and pixels â˘âŻTextures
- 58. 3D Graphics â˘âŻTwo options â˘âŻP3D Library â˘âŻOpenGL Library â˘âŻP3D â˘âŻSimple, integrated directly into processing â˘âŻLightweight 3D â˘âŻOpenGL â˘âŻFull graphics support â˘âŻComplex
- 59. Shapes â˘âŻ beginShape(SHAPE); â˘âŻ DefineVertices â˘âŻ SHAPES: QUADS, QUAD_STRIP,TRIANGLE_FAN â˘âŻ endShape(); â˘âŻ Eg: Quads !beginShape(QUADS);! !fill(0, 1, 1); vertex(-1, 1, 1);! !fill(1, 1, 1); vertex( 1, 1, 1);! !fill(1, 0, 1); vertex( 1, -1, 1);! !fill(0, 0, 1); vertex(-1, -1, 1);! !endShape();!
- 60. P3D Scene size(640, 360, P3D); ! background(0);! lights();! ! noStroke();! pushMatrix();! !translate(130, height/2, 0);! !rotateY(1.25);! !rotateX(-0.4);! !box(100);! popMatrix();! ! noFill();! stroke(255);! pushMatrix();! !translate(500, height*0.35, -200);! !sphere(280);! popMatrix();!
- 61. Vertices Demo â˘âŻRGB Cube â˘âŻ Vetices and vertex fills â˘âŻVertexDemo â˘âŻ Different types of quad strips â˘âŻ User defined drawing function
- 62. Transformations â˘âŻ Rotation ! !rotateX(a), rotateY(a * 2.0), rotateZ(a)! â˘âŻ Translation ! !translate(X,Y); translate(X,Y,Z);! â˘âŻ Scale â˘âŻ Push and Pop functions â˘âŻ Push - Saving current coordinate system â˘âŻ Pop â Restores previous coordinate system â˘âŻ Eg: PushPopCubes
- 63. OpenGL â˘âŻImport opengl library import processing.opengl.*;! â˘âŻCreate drawing context !void draw(){! ! !PGraphicsOpenGL pgl = (PGraphicsOpenGL) g;! ! !GL g = pgl.beginGL();! ! !//use GL methods! ! !pgl.endGL(); //end opengl calls! }!
- 64. 3D Model Loading â˘âŻ Using saito model loader â˘âŻ Loads obj files â˘âŻ http://code.google.com/p/saitoobjloader/downloads â˘âŻ Code fragment import saito.objloader.*;! OBJModel model ;! model = new OBJModel(this, "sa.obj", "absolute", TRIANGLES);!
- 66. OtherTools â˘âŻGLGraphics â˘âŻ http://glgraphics.sourceforge.net/ â˘âŻLibrary that extends the Processing OpenlGL renderer â˘âŻ opengl textures, â˘âŻ image post-processing filters, â˘âŻ 3D Models, and shaders in GLSL, â˘âŻ Cg and CgFX
- 68. Advanced Interface Technology â˘âŻWearable Computers â˘âŻAugmented Reality â˘âŻVirtual Reality â˘âŻInvisible Interfaces â˘âŻEnvironment Sensing â˘âŻPhysiological Sensing
- 69. Class Project 1.⯠Pick Advanced Technology 2.⯠Brainstorm use case 3.⯠Develop conceptual design 4.⯠Prototype interface/experience design 5.⯠Conduct user evaluation 6.⯠Repeat steps 3-5 7.⯠Write report
- 74. Major changes in computing
- 76. Wearable Computing âŞâŻ Computer on the body that is: âŞâŻ Always on âŞâŻ Always accessible âŞâŻ Always connected âŞâŻ Other attributes âŞâŻ Augmenting user actions âŞâŻ Aware of user and surroundings
- 78. Google Glass
- 81. Research Problems â˘âŻHardware â˘âŻ Power, networking, display â˘âŻUser Interaction â˘âŻ User input, speech, gesture, gaze, etc â˘âŻ Novel interaction methods â˘âŻSocial Acceptance â˘âŻ Privacy, social factors â˘âŻNovel Applications â˘âŻ Collaboration â˘âŻ Intelligent assistance
- 84. Augmented Reality Definition â˘âŻDefining Characteristics [Azuma 97] â˘âŻCombines Real andVirtual Images â˘âŻBoth can be seen at the same time â˘âŻInteractive in real-time â˘âŻThe virtual content can be interacted with â˘âŻRegistered in 3D â˘âŻVirtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 85. 2008 - CNN
- 86. â˘âŻ Put AR pictures here Augmented Reality Examples
- 88. Research Problems â˘âŻLow level hardware/software â˘âŻ Head mounted displays â˘âŻ Tracking systems â˘âŻUser Interaction â˘âŻ Gesture based interaction â˘âŻ Multimodal input (speech, gesture) â˘âŻNovel Applications â˘âŻ Face to face collaboration â˘âŻ Authoring tools
- 89. VIRTUAL REALITY
- 91. Virtual Reality â˘âŻImmersiveVR â˘âŻ Head mounted display, gloves â˘âŻ Separation from the real world
- 92. Occulus Rift â˘âŻ$300 USD â˘âŻ360 degree head tracking â˘âŻ100 degree field of view
- 93. Wearable Virtual Reality â˘âŻ Samsung Gear VR â˘âŻ See virtual reality on your phone â˘âŻ Using phone display, compass
- 95. AR vsVR
- 96. Research Problems â˘âŻLow level â˘âŻ Wide area tracking â˘âŻ Development tools â˘âŻUser Interaction â˘âŻ Intuitive input (gesture, controllers) â˘âŻ Avatar control and representation â˘âŻ Techniques for navigation/manipulation â˘âŻNovel Applications â˘âŻ Massive multi-user environments â˘âŻ Content capture and sharing
- 98. Early Examples â˘âŻ Interaction without devices: â˘âŻ BodySpace [Strachan 2007]: Functions to body position â˘âŻ Abracadabra [Harrison 2007]: Magnets on finger tips â˘âŻ GesturePad [Rekimoto 2001]: Capacitive sensing in clothing â˘âŻ Palm-based Interaction â˘âŻ Haptic Hand [Kohli 2005]: Using non-dominant hand in VR â˘âŻ Sixth Sense [Mistry 2009]: Projection on hand â˘âŻ Brainy Hand [Tamaki 2009]: Head worn projector/camera
- 99. Unobtrusive Input Devices âŞâŻ GesturePad âŞâŻ Capacitive multilayered touchpads âŞâŻ Supports interactive clothing
- 100. ImaginaryPhone â˘âŻ Gustafson, S., Holz, C., & Baudisch, P. [2011]
- 102. Transfer Learning
- 103. Invisible Interfaces â Gestures in Space â˘âŻ Gustafson, S., Bierwirth, D., & Baudisch, P. [2010] â˘âŻ Using a non-dominant hand stabilized interface.
- 105. Project Soli â˘âŻ Using Radar to support free-hand spatial input
- 106. Project Soli https://www.youtube.com/watch?v=0QNiZfSsPc0 https://www.youtube.com/watch?v=jWNebDDmuXc
- 107. Research Gaps â˘âŻFree-hand interfaces using relative input â˘âŻCombining invisible interface + mobile device â˘âŻMultimodal interaction â˘âŻ speech + gesture input â˘âŻAffordances and discoverability â˘âŻInteraction frameworks
- 108. Research Problems â˘âŻHardware â˘âŻ Power, networking, display â˘âŻUser Interaction â˘âŻ User input, speech, gesture, gaze, etc â˘âŻ Novel interaction methods â˘âŻSocial Acceptance â˘âŻ Privacy, social factors â˘âŻNovel Applications â˘âŻ Collaboration â˘âŻ Intelligent assistance
- 109. ENVIRONMENT SENSING
- 110. Environmental Sensor â˘âŻNew sensors track and capture real environment â˘âŻ Navigation, 3D modeling, user tracking â˘âŻDepth Sensors â˘âŻ Microsoft Kinect, Intel RealSense â˘âŻIntegrated Devices â˘âŻ Google Tango
- 111. Google Tango â˘âŻTablet based system â˘âŻAndroid OS â˘âŻMultiple sensors â˘âŻRGBD Sensor â˘âŻIR Structured light â˘âŻInertial sensors â˘âŻHigh end graphics â˘âŻNvidia tegra chip
- 112. Google Tango
- 113. Research Problems â˘âŻContent creation â˘âŻ Creating better 3D models â˘âŻ Segmenting objects â˘âŻUser Interaction â˘âŻ Interaction with real world â˘âŻ Interacting with multiple devices â˘âŻNovel Applications â˘âŻ AR notes/real world tagging â˘âŻ Social networking
- 115. Physiological Sensors â˘âŻSensing user state â˘âŻ Body worn devices â˘âŻMultiple possible sensors â˘âŻ Physical activity â˘âŻ Eye tracking, gaze â˘âŻ Heart rate â˘âŻ GSR â˘âŻ Breathing â˘âŻ Etc
- 116. Tobii Eye Tracker â˘âŻWearable eye tracking system â˘âŻ Natural data capture â˘âŻ Scene camera capture â˘âŻ Recording/streaming eye gaze, 60 Hz sampling
- 118. Research Problems â˘âŻUser Interaction â˘âŻ Implicit vs. Explicit interaction â˘âŻ Measuring cognitive â˘âŻSocial Acceptance â˘âŻ Privacy, social factors â˘âŻNovel Applications â˘âŻ Collaboration â˘âŻ Intelligent assistance