[PDF] the molecular control toolkit - Controlling 3D molecular graphics via gesture and voice

1. The Molecular Control Toolkit:
Controlling 3D Molecular Graphics via Gesture
and Voice
Quan Le
HCI Lab
17th Mar, 2015
Sabir, K. ; Garvan Inst. of Med. Res., Australia ; Stolte, C. ; Tabor, B. ; O'Donoghue, S.I.
IEEE Symposium on Biological Data Visualization 2013, Atlanta, Georgia, USA

2. Contents
 Introduction
 Outstanding
 Toolkit Overview
 Toolkit Architecture
 Toolkit Design
 Results
 Future Work
 Conclusion
2
Fig 1. Molecular Control Toolkit using Leap Motion and Aquaria

3. Introduction
3
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion
- Three-dimensional (3D) molecular graphics systems are widely
used in the life sciences.
- Biological systems requires use of molecular graphics to
examine atomic-resolution, protein, DNA, …
- Precise control 3D molecular structures can be challenging for
occasional users, event experts.

4. Introduction
4
Fig 3. Aquaria application is accessed by http://aquaria.ws
Fig 2. Molecular Control Toolkit
runs as a standalone Java
application
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion
- Molecular Control Toolkit supports multiple gestures and voice
recognition.
- Apply Toolkit to Aquaria molecular graphics system.

5. Outstanding
- The common interactions: translation, rotation, zooming, selection
of specific items, change display mode, …
- Current applications:
• In 1996, Pavlovic: using hand gesture and speech interface.
• Multi-touch display: applies six degrees of freedom (6DoF*), only uses finger
gestures.
• Sony’s EyeToy, Playstation Eye, uses audio gesture control.
• Nintendo Wii and Sony Move: requires a physical device to manipulate
objects in 3D space.
5
* 6DoF: Six degrees of freedom, comprising of three axes for rotation and three for translation.
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion

6. Outstanding
- Molecular Control Toolkit provides a full six degree of
freedom.
- Ability to redefine the center of rotation, integrated voice
commands.
- Lower cost, more intuitive to learn, interact and communicate.
6
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion

7. Six degrees of freedom (6DoF)
7
Fig 4. The ship movements on the wave
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion
The ship movements on the wave:
Translation:
1 – Moving up and down
2 – Moving left and right
3 – Moving forward and backward
Rotation:
4 – Turning left and right
5 – Tilting forward and backward
6 – Tilting side to side

8. Molecular Control Toolkit
- Supports two gesture devices: Leap Motion and Microsoft
Kinect to control and interact with a Java Application.
- Provide voice commands.
8
Fig 5. Molecular Control Toolkit interacts with Aquaria via Leap Motion and MS Kinect
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion

9. Toolkit Architecture
- Gesture Listener
- Device Connectors
- Gesture Dispatcher
- Voice Dispatcher
9
Fig 6. Molecular Control Toolkit architecture
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion
Toolkit Listeners
Connectors
& Dispatchers

10. Toolkit Design
• Connector for Kinect gesture recognition:
10
Fig 7. Support gesture controls for Kinect
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion

11. Toolkit Design
• Connector for Leap Motion gesture recognition:
11
Fig 8. Support gesture controls for Leap Motion
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion

12. Toolkit Design
• Connector for Kinect voice recognition:
- Microsoft Kinect Speech API.
- Recognize the words from a custom grammar dictionary.
12
Fig 9. Supported voice commands for
Molecular Control Toolkit
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion

13. Results
13
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion
Fig 10. Participants background experience
- 18 staff members from Medical Research in Sydney, Australia.
- 20 ~ 30 minutes for training.
- Three tasks: Rotating, Selecting and Zooming.

14. Results
14
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion
Fig 11. Pilot study comparing mouse/keyboard, Leap Motion and Kinect controls
for molecular graphics.

15. Future Work
15
- Voice command:
• The vocabulary should be expanded.
- Support other new gesture devices: Create Interactive Gesture
Camera, PrimeSense, …
- Provides gesture tracking algorithms for both Leap Motion and
Kinect at the same time.
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion

16. Conclusion
16
Demo: https://www.youtube.com/watch?v=XU_6TWM9ke8
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Result
Future work
Conclusion
- Molecular Control Toolkit is the first system designed to enable
life scientists to control molecular graphics using gestures and
voice commands.
- Supports Leap Motion and MS Kinect.

17. Thank you!
17