The document discusses using picture-driven computing as an assistive technology to improve accessibility. It describes using visual images instead of text inputs to allow indirect access for users with disabilities. The Sikuli platform is highlighted as a way to automate tasks by programming screenshots of graphical interfaces. Future work areas include more research on picture-driven approaches, evaluating Sikuli, and developing additional assistive technology scripts.

1. PICTURE-DRIVEN COMPUTING IN ASSISTIVE
TECHNOLOGY FOR ENHANCED ACCESSIBILITY
Silvio Bonfiglio1 - George Kourousias2
1. FIMI, a Barco Company, Saronno (Italy)
2. Synchrotron Radiation Facility, ELETTRA, Trieste (Italy)
1st International AEGIS Conference 7-8 October 2010, University of Seville
www.aegis-project.eu

2. Domain: Improved Accessibility in ICT
Why?
It may positively affect and improve our society
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

3. Two Paths to Accessibility
DIRECT ACCESS Universal Design (UD) – Design for All
i.e. a product that – without add-ons – can be experienced with similar ease of use by any user independently by the disability.
INDIRECT ACCESS Multi-modal, Personalized Accessible Design
i.e. a product that is made accessible for a specific category of users by incorporating as an add on a specific AT component
or an extra interaction modality.
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

4. Picture Driven Computing
We define as Picture-driven Computing the Computing paradigm that
has as core Data Abstraction the Visual Information in a 2-
dimensional spatial domain. This Visual Information is what intends
to be visible on a graphical Output device.
•We aim to introduce it as an Assistive Technology
•Adding a Modality to the GUI of the computer
•Accessibility through Indirect Access
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

5. Example: a Picture-Driven Search
•Using Picture objects instead of keywords
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

6. Sikuli, a Picture-Driven language
•Developed recently at MIT by T. Yeh, T.H. Chang and R.C.Miller for the search and
automation of GUI elements using screenshots
•Based on Sikuli-script , a scripting module that allows programmers to use
screenshots of GUI elements to control them programmatically without any API.
•Python
•Open Source & Multiplatform
•We use it as our main paradigm for Picture-Driven Development of AT
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

7. Sikuli: How it Works
A. a computer vision library
B. a mechanism to programmatically “dictate” Input actions
C. a mature underlying language (Python) for traditional programatic control
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

8. Sikuli: An Example
Goal: clicks all the “minimize window” buttons on the screen
Picture-Object Input: target image of the button
i) then the systems computer vision (A) will locate the x,y position of that on the screen
ii) (B) the “mouse control” will “send” the mouse on x,y, and
iii) through a for or while loop (C) will click (B) for all the targets that (A) has found.
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

9. Sikuli as an Accessibility Design enabler
We Identify the following 3 classes:
 Automation Scripts: where Sikuli script - following a predefined set of mouse
and keyboard actions- preforms a task that may be difficult otherwise to the user with a
disability.
 Event Raisers: where the system can keep monitoring the user's screen for
specific graphical objects and raise an event/alarm and consequently trigger a set of
programmatic actions.
 Bridge Systems: where applications can communicate with other software even if
an API is not provided.
+ Hybrid approaches
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

10. Future work
•Further research on Picture-driven Computing.
•Comparison of “indirect access” Accessibility Design versus the
“direct” one (Universal Design) for the computer as an AT-aid.
•Additional development of Sikuli AT Scripts.
•Evaluation of the technical issues and performance of Sikuli.
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville

11. Acknowledgments

12. further info:
silvio.bonfiglio@barco.com
george.kourousias@elettra.trieste.it
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
1st International AEGIS Conference 7-8 October 2010, University of Seville