TITAN (EU R&I project) is dedicated to fighting disinformation, and is doing so through the co-creation of an AI-based engine that encourages citizens’ critical thinking to help them better identify false information online. Whilst AI offers exciting and unprecedented opportunities to deliver social good, the project team recognises it can entail ethical dangers too, from usability and understanding issues, to biases and data quality. To help counter any ethical implications, and enlighten the process of developing a trustworthy, ethically, and societally acceptable AI tool, TITAN has adopted a citizen-centric approach to understand requirements and needs. Citizens from across Europe have been involved in a co-creation process where their values, concerns and requirements were discussed, the results of which will become the foundation for the digital development of TITAN. The first round of citizen co-creation was conducted during five-hour workshop's with 30 citizens in each session, in 8 different European countries - Denmark, Ireland, Italy, Spain, Lithuania, Belgium, Bulgaria and Slovakia. The co-creation framework created by TITAN partner the not-for-profit Danish Board of Technology consists of tried and tested engagement and co-creation methodologies. Read more about the techniqies adopted at https://www.titanthinking.eu/post/building-together-insights-from-8-countries-co-creating-the-titan-ai-tool-to-fight-disinformation

1. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
TITAN Citizen Co-Creation
Presentation of Findings from the Citizen Co-Creation Workshops (D3.1),
by DBT
Sissel Fibecker Ladegaard, Marie Hoff
10th of May
The research leading to this result has received funding
from the European Union’s Horizon 2020 Programme
under Grant Agreement (GA) #101070658 and by UK
Research and innovation under the UK governments
Horizon funding guarantee grant numbers 10040483 and
10055990.

2. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Agenda
MAIN FINDINGS QUESTIONS TO THE
FINDINGS
DEBATE ON KEY
FINDINGS

3. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
• Session 1: Timeline Exercise
• Session 2: News Bit Exercise
• Session 3: Scenario Game
• Session 4: Pre-mortem Exercise
Methods of Co-creation Workshops
Workshop:

4. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
• 8 countries
• 215 citizens participated
• Diversity across age, gender,
educational background,
current occupation, and
geographical zone.
• Gave insight into a broad
perspective of opinions of
European citizens.

5. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Next Step of the Co-creation Process:
Stakeholder Workshops
• Workshop #1:
How to enable informed consent in AI tools
• Workshop #2:
How to give the users of AI tools control of their data and how it influences the ethics
and trust in AI
• Workshop #3:
How to adapt an AI learning tool to citizen’s needs

6. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358

7. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358

8. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Disinformation Signals
• List of information signals based on session 2
• Show patterns of disinformation signals
• Overlap between signals of true information and
disinformation
• Evaluation dependent on personal perception,
bias, and cultural context
Recommendation: That guidance and training of
the user’s ability to identify disinformation signals
can consider personal differences

9. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Disinformation Signals – Examples
• One example is from news bit 5 (see Annex 2),
about the finding of drugs an antibiotic in milk
in Italy: The name of the website “greenme” was
understood in two different ways. Some citizens
argue that the news bit has a climate
focus/political agenda, and their hidden agenda
could be that people should not drink cows’ milk
(possibly disinformation),while another citizen
argue that because the name of the website
contained the word “green” it gave creditability
to the source (possibly true news).
It is true news
Continuation of list from previous slide

10. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
How Citizens Share News and Informaiton
• Share in private conversations with friends, family, or colleagues
• Share in private conversations in offline conversations
• Share in online private messenger apps, like WhatsApp and Telegram, and email
• Across all eight countries, some do not share any news or information online
Recommendation: TITAN needs to consider how and when to best interact with the
users to make them aware of the risk of sharing disinformation. Consider to focus
on the moment of consuming and less on the moment of sharing

11. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
How Citizens Share News and Informaiton
– Examples
• Don´t want monitoring of private online conversations vs. hindering spread of
disinformation in private messenger channels
• Sharing publicly online when:
- relevant/trending information at the current moment
- promote a topic (sometimes work related)
- share information to refute misinformation

12. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Citizens view on Data
• Citizens demand control over data and transparency about how their data is used
• Considered when personal data is needed and give the users a choice of what data
to give to the system, to get specific functionalities
• Have transparency in; storage of data, security of data, who have access (data
controller), what is their interest in the data
• Consider to offer a version where users is not obligated to share personal data
Recommendation: Consider the benefits and disadvantages with respectively local
and central storage, explain why one option of storage of data is decided.
Allow different individual choices of what kind of data the user give to the system in
exchange for the service they can receive.

13. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Citizens View on Data – Examples
57% prefers not to be obligated
to share personal data
29% wanted a personalised profile
that track their behaviour online
47% of the citizens preferred
local storage and 53%
preferred central storage

14. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Citizens View on User Experience (UX)
• A system that can adapt to different users needs:
- Balance the contradiction of many preferring a passive and generic system,
but expecting a (personalised) version that can adapt to different users’ needs
- Preferring a passive and generic system is grounded in citizens wanting
control and choice regarding their data
- Explaining how certain data can improve the functionalities of the system, can
appeal to more citizens using a personalised system
- Both an active and passive system can enhance a critical thinking mindset in
news consumption
Recommendation: Service offered in several languages, and consider age
differences, time and attention aspects, and accommodating users wish for more/
less personalization – will be key to attract users.

15. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Citizens View on UX – Examples
• Time
• Attention
• Language

16. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Coaching and Learning Perspective
• Some citizens perceived TITAN as a fact checking tool, for some a desire for
others a fear that it will be so
• Balance: having an easy to use and at-hand tool to help citizens fact-check
information and having a learning tool
• Consider how the chatbot is utilized to not become an obstacle for the users
Recommendation: Clearly communicate that TITAN is a coach that offers a
learning element and consider if citizens’ suggestions on learning perspectives
can be implemented in designing the tool. Transparent communication is key
(TITAN is not oppressing free speech or creating echo chambers)

17. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Coaching and Learning - Debate
• What have already been considered?
• What does the users have in common across all use cases?
• How do we ensure learning elements in the TITAN services?
• What is the unique selling point for TITAN? (makes it attractive to use)

18. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Creating Trustworthiness
• TITAN must build trust to the citizens from the start:
- A communication effort within the system and in the dissemination of TITAN
- The topics of AI and disinformation are associated with negative connotations
- Communicate about how the system works
- Citizens testing the tool can help spread the word about the tool and
recommend friends and family to use the system
Recommendation: Consider and communicate how TITAN is different from other
services and what the unique selling points of TITAN are - a human-centred, ethical,
and trustworthy AI tool. The above suggestions give recommendation on how to
implement trustworthiness in the system.

19. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Creating Trustworthiness - Debate
• What concrete initiatives in TITAN will create a trustworthy AI tool?
• How do we measure trustworthiness in the development process? (what
factors enabling trust do we work with?)
• Can we do more in our communicaiton about TITAN to create trust towards
citizens?
- Action point: Have a common language within the consortium about
what we call the TITAN tool/system/coach (and elements within)?

20. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Conclusion:
Recommondations to the TITAN system
• Accommodate the complexity of disinformation signals
• Mitigate the concerns that the citizens have expressed about data sharing and storage
• Develop a tool according to the citizens desires (acceptance in society):
- Be transparent about data – need, use, and storage
- Be adaptable for the different users’ individuals needs and preferences
- The coaching aspect must be transparent in its functionality and be user-friendly
- Create trustworthiness by clearly communicating the reasoning behind the tool
and its design

21. Fonden Teknologirådet ∙ Arnold Nielsens Boulevard 68E ∙ DK - 2650 Hvidovre
www.tekno.dk ∙ tekno@tekno.dk ∙ T. +45 33 32 05 03 ∙ CVR: 34577358
Last Questions?