In this work, we propose a virtual assistant that allows building models by means of voice commands. To demonstrate the generality of the approach, we describe three alternative strategies that apply voice-based support at three levels of detail: a fully-guided strategy; a pattern-based strategy; and an element-based strategy. We describe our implementation experience with the development of a design assistant covering the three strategies described above for OMG's IFML (Interaction Flow Modeling Language), in the context of user interaction design, including the integration with the Amazon Alexa assistant. We report our results that show how the assistant can bring advantages in terms of productivity. Presented at ICWE 2021. The paper describing the approach is available online at: https://link.springer.com/chapter/10.1007/978-3-030-74296-6_45

1. Voice-based Virtual Assistants for
User Interaction Modeling
Marco Brambilla
marco.brambilla@polimi.it
Davide Molinelli
davide.molinelli@mail.polimi.it

2. Web Interaction and Application Design
The development of a Web and
mobile or web application
requires a great effort in the
planning of the corresponding
abstract model

3. Web Interaction and Application Design
It represents one of the most
delicate steps in the whole
process of building and
implementing a system

4. Web Interaction and Application Design
It represents one of the most
delicate steps in the whole
process of building and
implementing a system
The development of a Web and
mobile or web application
requires a great effort in the
planning of the corresponding
abstract model
Without exaggerating, it could
determine the success or
failure of the application itself

5. Web Interaction and Application Design
It could determine the
success or failure of the
application itself

6. Model Development
Model development remains a non-deterministic process,
with lacking support, costly and error-prone
The construction of a model is still largely dependent
on the imagination and the skill of the developer
Making use of design patterns during the modeling process is not enough

7. Goals
• Speed up the development process and increase
the quality of the produced models, with the help
of tools able to generate, customize and detect
design patterns inside a model
• Guide the developer in the development process,
with the support of a voice assistant

8. With our approach, designers can build software models using voice
commands.
The method is general and independent from the modeling language and the
specific voice assistant.
We show the method at work on a use case: the specification of user
interactions
using the Interaction Flow Modeling Language (IFML), a standard by the OMG.
Goals

9. We integrate with
Edit.org
org the voice assistant developed by Amazon.
an online tool for the rapid prototyping of
web and mobile applications, based on
the Interaction Flow Modeling Language
(IFML)
Integration

10. Alexa, open Model Creator
Welcome to Model Creator
Interaction Cycle

11. IFMLEdit.org and Quick Prototyping

12. The demo illustrates three alternative strategies that apply voice-based
support at three levels of detail:
1. Fully-guided strategy
2. Pattern-based strategy
3. Element-based
strategy

13. Fully-guided Strategy
Produces complete designs starting from requirements,
without the need of looking into the modeling at all

14. Pattern-based Strategy
Specifies solutions by selecting design patterns
and combining them together

15. Element-based Strategy
Specifies precise modeling structures
with fine-grained granularity

16. Use Cases and Experiments

17. Use Cases
• E-commerce
• Blog
• Social Network
• Crowdsourcing

18. Evaluation Metric
Operation types:
• Insertion
• Connection
• Customization
• Binding
Number of operations needed to develop models

19. E-commerce
0
50
100
150
200
250
300
350
With Automatic Design Pattern Generator Without Automatic Design Pattern Generator
• The model built without
the support of the
Automatic Design
Pattern Generator
requires a total of 1443
operations
• The model built with the
support of the Automatic
Design Pattern
Generator requires a
total of 362 operations.

20. Blog
0
20
40
60
80
100
120
140
160
180
With Automatic Design Pattern Generator Without Automatic Design Pattern Generator
• The model built without
the support of the
Automatic Design
Pattern Generator
requires a total of 971
operations
• The model built with the
support of the Automatic
Design Pattern
Generator requires a
total of 236 operations.

21. Social Network
0
50
100
150
200
250
300
Master Detail Basic Search Sign Up and Log
In
Account
Content
Management
Friends Content
Management
Posts Pages
Content
Management
Support and
Aggregation
With Automatic Design Pattern Generator Without Automatic Design Pattern Generator
• The model built without
the support of the
Automatic Design
Pattern Generator
requires a total of 847
operations
• The model built with the
support of the Automatic
Design Pattern
Generator requires a
total of 331 operations.

22. Crowdsourcing
0
20
40
60
80
100
120
140
160
180
Multilevel
Master Detail
Restricted
Search
Sign Up and
Log In
Master
Account
Content
Management
Worker
Account
Content
Management
Tasks
Content
Management
Tasks Input
Data
Validation
Support and
Aggregation
With Automatic Design Pattern Generator Without Automatic Design Pattern Generator
• The model built without
the support of the
Automatic Design
Pattern Generator
requires a total of 735
operations
• The model built with the
support of the Automatic
Design Pattern
Generator requires a
total of 177 operations.

23. Percentage Results

24. Demo video
https://www.youtube.com/watch?v=00HoMz9Tq0A&t=47s

25. Voice-based Virtual Assistants for
User Interaction Modeling
Marco Brambilla, Davide Molinelli
Contacts:
Marco Brambilla @marcobrambi marco.brambilla@polimi.it
http://datascience.deib.polimi.it http://home.deib.polimi.it/marcobrambi