Human-Agent Interaction: Building Socially Intelligent Agents in Games

1. Human-Agent Interaction
Building Socially Intelligent Agents in Games
Rui Prada
Instituto Superior Técnico, Universidade de Lisboa
INESC-ID
Habilitation in Computer Science and Engineering, June 29, 2022

2. Human-Agent Interaction

3. HAI: Agency
Agents are subsets of machines that have agency
HAI is HCI with agents in the interaction
It is more than automation
Agents have autonomy
Regarding the performance of actions
To start actions
Regarding the definition and selection of goals

4. HAI: Mixed-initiative
Direct control Agency
Automation
User makes decisions
User performs actions
User makes decisions
Agent performs actions
Agent makes decisions
Agent performs actions
Shared control and adaptive autonomy

5. HAI: User Experience
Attributions of agency are subjective
Change with the familiarity of the system/agent
User experience and user acceptance are key factors in HAI
There is a tension between solving the users’ problems, without
bothering them, and the users’ experience
Problems with the lack of sense of control (and actual control)
And understanding the state of affairs
Explainable (transparent) AI

6. HAI: Delegation
Users delegate part of their tasks and goals to the agents
Agents fill the unspecified parts
Delegation must be aligned with the agents’ capabilities
User needs to get a good understanding of the space of constraints and
affordances of the interaction
Agent needs to be able to accept the delegation
The agent must be able to understand the user’s needs, preferences, and
requests
Reaching common ground and mutual understanding is key

7. HAI: Delegation
Agent has autonomy in the acceptance of the delegation
Agency does not assume a benevolent attitude from the agent
Agents may interact with different stakeholders that have different
needs, concerns and resources
The user may request something that the agent should not do
The agent performs a better action (overhelp) filling ill-specified details
and using knowledge that the user is not aware of

8. HAI: Trust
Delegation and agency makes trust a crucial issue
Trust at several levels
Competence. The belief that the agent has competence to perform
the actions
Predictability. The ability of the agent to perform and present
results according to the user’s expectations
Benevolence. The agent has positive intentions and is motivated to
help the user as best as possible
Integrity. The agent will follow principles that are morally
acceptable

9. HAI: Trust
Trust is built overtime and develops as users get deeper
understanding of the agent
There are several factors that affect the perception of trust
Human factors (e.g., expertise, preferences)
Agent factors (e.g., capabilities, appearance, transparency,
consistency)
Environmental factors (e.g., culture, tasks, reputation)

10. HAI: Trust
Trust is attributed at the first impression of the agent, and
even before interacting with it
A good HAI system must seek a good balance of trust
Overtrust leads to misuse of the system since it is built in false
expectations by the user
Distrust leads to the lack of use of the system and higher user’s
workload
The agent may need to trust the human as well

11. HAI as a Social System
Delegation implies a social structure where the user always
maintain authority
The goal of HAI is to create human-machine systems that
make use of the best human capabilities combined with the
best machine capabilities

12. HAI as a Social System
HAI has multi-lateral delegation
HAI may have several agents in human-agent mixed-networks
Diverse social structure and social order
Agents as social actors that can take any social role, not being
just intelligent tools or slaves
Subordinates, Teammates and Managers
Competitors (representing others)

13. HAI as a Social System
Diverse social roles lead to diverse social interactions (beyond
delegation)
Sharing information, reaching common ground, coordinating
actions, reaching agreement, persuasion, negotiation, co-creation,…
Coordination at different levels
Content, protocol (e.g., conversation), relational, emotional
Mutual understanding at different levels
Preferences, goals, capabilities, decision policies, emotions

14. Challenges in HAI
Methodological Challenges. Social roles and interactions in the core
(design and technical), assessing and measuring success
Situation Awareness Challenges. Reaching common ground,
understanding the context and others (the users)
Interaction Dynamics Challenges. Long-term, group dynamics, using
the 5 senses
Societal Challenges. Social responsibility and ethics, novel applications

15. Building
Socially
Intelligent
Agents
Generated by AI
https://creator.nightcafe.studio/

16. Building Socially Intelligent Agents
Create interactive autonomous agents that act according to social
context, user actions, and designers' goals
Social context
Who: roles and relations
What: activity and tasks
Where: places and locations
When: time and events
Combined with
Volition: goals, needs

17. Social Importance Dynamics
Social Importance (SI) as a key concept in social interactions
Precondition for acceptable behaviour
Claim and confer dynamics
(Kemper Status-power Interaction Dynamics)
Difference groups/cultures give social importance to different
things
(Hofstede Model of Culture)
Samuel Mascarenhas, João Dias, Rui Prada, Ana Paiva “A Dimensional Model for Cultural Behaviour in Virtual Agents” in Applied Artificial Intelligence,
vol. 24 (6), pp. 552-574, July 2010. Taylor & Francis.
Samuel Mascarenhas, Nick Degens, Ana Paiva, Rui Prada, Gert Jan Hofstede, Adrie Beulens, Ruth Aylett: “Modeling culture in intelligent virtual agents:
From theory to implementation” in Autonomous Agents and Multi-Agent Systems. pp. 1-32, 2015. Springer.

18. Social Importance Dynamics
Actions invoke SI claims
Agents attribute SI to others
An action performed by agent a is
accepted by agent b if b confers
enough SI to a to support the
action claim
The rules for SI claim and conferral
differ across groups
E.g.: the SI given to strangers is
different across cultures
Traveller: a game for Cultural Training

19. Traveller: Study
Cross-cultural study 2x2
Participants: Portugal (collectivistic) vs The Netherlands
(individualistic)
Culture parameterization: collectivistic agents vs
individualistic agents
Samuel Mascarenhas, João Dias, Rui Prada, Ana Paiva “A Dimensional Model for Cultural Behaviour in Virtual Agents” in Applied Artificial Intelligence,
vol. 24 (6), pp. 552-574, July 2010. Taylor & Francis.
Samuel Mascarenhas, Nick Degens, Ana Paiva, Rui Prada, Gert Jan Hofstede, Adrie Beulens, Ruth Aylett: “Modeling culture in intelligent virtual agents:
From theory to implementation” in Autonomous Agents and Multi-Agent Systems. pp. 1-32, 2015. Springer.

20. Traveller: Results
Dutch participants complained that the collectivistic agents were too
distant
Dutch participants offered more often a drink to the individualistic
agents. Portuguese offered more to the collectivistic agents
Portuguese participants had a significantly higher opinion of the
collectivistic agents than the Dutch
Both countries had a similarly opinion of the individualistic agents
Samuel Mascarenhas, João Dias, Rui Prada, Ana Paiva “A Dimensional Model for Cultural Behaviour in Virtual Agents” in Applied Artificial Intelligence,
vol. 24 (6), pp. 552-574, July 2010. Taylor & Francis.
Samuel Mascarenhas, Nick Degens, Ana Paiva, Rui Prada, Gert Jan Hofstede, Adrie Beulens, Ruth Aylett: “Modeling culture in intelligent virtual agents:
From theory to implementation” in Autonomous Agents and Multi-Agent Systems. pp. 1-32, 2015. Springer.

21. Social Power
Social power mediates social interactions
Influence towards change
Influence is a function of the power exerted and the
resistance to change
Social power as dynamics of social importance
Power = SI conferred to the actor
Resistance = SI claim of the change
Gonçalo Pereira, Rui Prada, Pedro A. Santos: “Integrating social power into the decision-making of cognitive agents” in Artificial Intelligence. vol. 241,
pp. 1-44, December 2016. Elsevier.

22. Social Power: SAPIENT Model
Social power has different sources
(French and Raven)
Reward
Coercive
Legitimate
Expert
Referent
Power strategies highlight the sources
in the context
Social Theatre: a game for training conflict resolution
Gonçalo Pereira, Rui Prada, Pedro A. Santos: “Integrating social power into the decision-making of cognitive agents” in Artificial Intelligence. vol. 241,
pp. 1-44, December 2016. Elsevier.

23. Social Theatre: Results
Within groups study with 30 people
Playing the game with agents using social power dynamics
Playing the game with agents using scripted behaviour - fixed
accepting rates: 95% (desired role), 40%(undesirable role)
Participants attributed significantly higher social power
awareness to the agents that used the model
The interaction experience was better as well with those
agents
Gonçalo Pereira, Rui Prada, Pedro A. Santos: “Integrating social power into the decision-making of cognitive agents” in Artificial Intelligence. vol. 241,
pp. 1-44, December 2016. Elsevier.

24. Social Identity
Self and others’ identity shape the behaviour, social identity
changes with context
Dynamic Identity Model for Agents (DIMA)
Agents redefine their (and others) identity either as unique
individuals or as members of a social group
Base on the salience of the social identity in the context
Joana Dimas, Phil Lopes, Rui Prada: “One for all, all for one: Agents with social identities” in proceedings of CogSci‘2013 - 35th Annual Meeting of the
Cognitive Science Society, pp. 2195-2200, Berlin, Germany. August 2013. CSS.
Joana Dimas, Rui Prada: “Dynamic Identity Model for Agents” in Multi-Agent-Based Simulation XIV: International Workshop, MABS 2013, Saint Paul, USA,
May 2013, Revised Selected Papers, Shah Jamal Alam, H. Van Dyke Parunak (Eds.). Lecture Notes in Computer Science, pp. 37-52. 2014. Springer Berlin
Heidelberg.

25. The DIMA Model
Social identity salience
Fit: comparative and normative
Accessibility: past experiences
Agents will use the salient identities
(characteristics and memberships)
and the value of the salience for
decision making Volcano Island: a public good game

26. Volcano Island: Study
Between groups 2X2 with 216 participants
Social identity cues: with social identity cues (avatar using a
common colour) or without
Playing partners: 3 human players vs playing with 2 agents
Jorge Peña, Jannath Ghaznavi, Nicholas Brody, Rui Prada, Carlos Martinho, Pedro A. Santos, Hugo Damas, Joana Dimas (2019): “Effects of Human vs.
Computer-Controlled Characters and Social Identity Cues on Enjoyment Mediation Effects of Presence, Similarity, and Group Identification”. Journal of
Media Psychology: Theories, Methods, and Applications, 31(1): 35–47. Hogrefe.

27. Volcano Island: Results
Social identity cues increased group identification, enjoyment
and social presence
Participants reported more enjoyment while playing with
other humans in the condition with no social identity cues
But enjoyment was higher when playing with agents when
social identity was salient
Jorge Peña, Jannath Ghaznavi, Nicholas Brody, Rui Prada, Carlos Martinho, Pedro A. Santos, Hugo Damas, Joana Dimas (2019): “Effects of Human vs.
Computer-Controlled Characters and Social Identity Cues on Enjoyment Mediation Effects of Presence, Similarity, and Group Identification”. Journal of
Media Psychology: Theories, Methods, and Applications, 31(1): 35–47. Hogrefe.

28. Socially Situated Cognition
Social meaning of objects
E.g., An apple can be food, a gift, a toy, a weapon, …
Social categorization and social identity
The agents and their social groups in a given context
Social affordances
What you can do with the agents and objects in the context
Socially affordable
What is acceptable
Diogo Rato, Samuel Mascarenhas, Rui Prada (2021). Towards Social Identity in Socio-Cognitive Agents. Sustainability special issue on AI and Interaction
Technologies for Social Sustainability, 13(20): 11390. MDPI.

29. Social AI agents in Minecraft
Context
Time, location, agents
Social practices
Activated by context
Social roles
Locations have social properties
Expected activity
Ownership
Agents have categories/identities
Define relevant social practices

30. Socially Aware Conversational Agents
Maria Inês Lobo, Diogo Rato, Rui Prada, Frank Dignum (2021). Socially Aware Interactions: From Dialogue Trees to Natural Language Dialogue Systems.
Proceedings of CONVERSATIONS 2021 - the 5th International Workshop on Chatbot Research, Lecture Notes in Computer Science. Oslo, Norway.
November 23-24. Springer.
Doctor appointment: goal-oriented interview
Conversation as social knowledge
Simple conversation beats as social
practices
Context filters acceptable practices
Who (social roles, relations)
State of conversation
Goals
Practices define sentences available
to use

31. Application
to
Social
Robots
Generated by AI
https://replicate.com/pixray/text2image

32. Social Power for Persuasive Robots
Mojgan Hashemian; Ana Paiva; Samuel Mascarenhas; Pedro A. Santos; Rui Prada: “The power to persuade: a study of social power in human-robot
interaction” in proceedings of RO-MAN’19 - the 28th IEEE International Conference on Robot and Human Interactive Communication, pp. 1-8, New Delhi, India,
October 2019. IEEE.
Mojgan Hashemian, Marta Couto, Samuel Mascarenhas, Ana Paiva, Pedro A. Santos, Rui Prada (2020) “Investigating Reward/Punishment Strategies in the
Persuasiveness of Social Robots” Proceedings of RO-MAN 2020 - the 29th International Conference on Robot and Human Interactive Communication, 863-868.
Naples, Italy, August 31 – September 4. IEEE.
Robot tries to make the user
select a coffee (lower raking)
Using different strategies
(reward, expert, coercion)
The robot was able to persuade
No difference was found on the
effects of the strategies

33. André Tiago Pereira, Rui Prada, Ana Paiva: “Improving social presence in human-agent interaction” in proceedings of CHI’2014 - 32nd annual ACM conference
on Human Factors in Computing Systems, pp. 1449-1458, Toronto, Canada. April 2014. ACM.
Robotic Social Player
Contextual gaze
Emotional reactions to game
events
Interpersonal relations
Increased the social
presence of the artificial
player
Emys the Risk player

34. HRI Group-based Emotions
Displaying emotions as individual or group
Determine the cognitive unit for the emotional appraisal
Positive effects for group identification, trust and likability
Filipa Correia, Samuel Mascarenhas, Rui Prada, Francisco S. Melo, Ana Paiva: “Group-based emotions in teams of humans and robots” in proceedings of
HRI'18 - International Conference on Human-Robot Interaction, pp. 261-269, Chicago, IL, USA, March 2018. ACM/IEEE.

35. Social Robots as Team Leaders
Leadership types
Transactional (TA): focus on task
Transformational (TF): focus on people
108 people (Portuguese companies) 36
teams of 3
Productivity: higher for TA
Engagement: higher for TF
Role Ambiguity: no sig. difference
Trust: no sig. difference
Sara L. Lopes, José Bernardo Rocha, Aristides I. Ferreira, Rui Prada (2021) “Social robots as leaders: leadership styles in human robot teams” Proceedings of
RO-MAN 2021 - the 30th International Conference on Robot and Human Interactive Communication, 258-263. Vancouver, BC, Canada, August 8-12. IEEE.

36. Application
to
Serious
Games
Generated by AI
https://huggingface.co/spaces/dalle-mini/dalle-mini

37. https://fatima-toolkit.eu
Samuel Mascarenhas, Manuel Guimarães, Rui Prada, João Dias, Pedro A. Santos, Kam Star, Ben Hirsh, Ellis Spice, and Rob Kommeren “A Virtual Agent Toolkit for
Serious Games Developers” in proceedings of 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-7. IEEE, August 2018.

38. Space Modules Inc. Sports Team Manager
Social AI agents with FAtiMA Toolkit

39. Com@Viver
A game to promote bystanders’ empathy in cyberbullying
situations
Paula C. Ferreira, Ana Margarida Veiga Simão, Ana Paiva, Carlos Martinho, Rui Prada, Aristides Ferreira, Francisco
Santos “Exploring empathy in cyberbullying with serious games” Computers & Education, 166:104155, 2021. Elsevier

40. Com@Viver: Results
Longitudinal study (5 sessions) with 221, 7th and 8th grade students (in
3 schools)
3 groups: play the game (G), read a paper version (P), did not play (N)
Positive difference in terms of cognitive empathy
More empathic concerns towards the cyberbullying situations
Higher levels of empathic reactions towards victims
Paula C Ferreira, Ana Margarida Veiga Simão, Ana Paiva, Carlos Martinho, Rui Prada, Aristides Ferreira, Francisco
Santos “Exploring empathy in cyberbullying with serious games” Computers & Education, 166:104155, 2021. Elsevier

41. Conclusions
In HAI, agents have agency, knowledge can capabilities that take the
control from the user
This can lead to misunderstandings, trust issues and, therefore, misuse
of the HAI system
Challenging from the AI perspective, as the agent needs proper social
skills to understand the user needs and the context
Challenging from the HCI perspective need to incorporate the lack of
control in the interactions and accommodate designer intentions
Prominent applications of HAI are social robotics and games

42. Future work
The automatic generation of groups of agents that function well
as social groups, or a society
Explore crowdsourced mechanisms (e.g., games) to teach social
skills to the agents
Agents that can engage in a partnership with users, for example,
to play collaborative games, such as Geometry Friends1, and
perform group tasks (teamwork) together with users
Design and development tools for the creation and testing of HAI
systems
1https://gaips.inesc-id.pt/geometryfriends/

43. Acknowledgements
Samuel Mascarenhas
André Pereira
Iolanda Leite
Gonçalo Pereira
Joana Dimas
Diogo Rato
Manuel Guimarães
Mojgan Hashemian
Filipa Correia
Sara Lopes
Inês Lobo
José Rocha