Abstract: The ubiquitous adoption of social media has set high expectations for emergency services to serve the public online. However, the information overload of social media requests to seek help challenges the emergency services. This work introduces a human-AI collaboration framework to assist emergency services in effectively responding to online serviceable requests. In particular, this talk describes specific solutions to two problems in this framework. The first problem is the challenge of how to mitigate potential human errors when giving annotation feedback to the active learning model in the system that we address by proposing a psychological theory-inspired technique. The second is the challenge of how to optimally select how many and how often to present the request for request while accounting for the dynamic constraints of the busy service personnel that we address by proposing an optimization technique.   Biography: Hemant Purohit, Ph.D. is an assistant professor in the Department of Information Sciences & Technology at Volgenau School of Engineering, George Mason University. His research interest is to design intelligent systems for augmenting human capabilities of real-time information processing at a workplace, particularly public services and NGOs, by using methods of social & web mining, semantic computing, and human-AI collaboration. He applies this research in disaster informatics for assisting communities toward resilience from natural hazards, societal crises (e.g. violence and stereotyping), and man-made crises including cyber attacks. Purohit has received many awards for disaster informatics work including 2014 ITU Young-Innovator award from the United Nations agency on information and communication technologies for an opensource technology concept for disaster management. He was an invited academic member of the DHS Science & Technology Directorate's subcommittee on Social Media Working Group for Emergency Services and Disaster Management. He has given several talks, tutorials, and lectures on social computing for public services as well as organized workshops. His work has been published in prestigious conferences and journals and he currently serves on the editorial board of the Elsevier journal of Information Processing & Management. His research is supported by various national and international agencies including the U.S. National Science Foundation.   Contact: hpurohit@gmu.edu | http://ist.gmu.edu/~hpurohit

1. Human-AI Collaboration for Real-time Data
Processing Systems at Emergency Services
Hemant Purohit, Ph.D.
Humanitarian Informatics Lab (Human_Info_Lab)
Dept. of Information Sciences & Technology
Mar 5, 2021 @hemant_pt | hpurohit@gmu.edu
Grants:
• IIS #1657379, IIS #1815459
PhD Students:
Rahul Pandey & Yasas Senarath
Special Thanks:
Guest Lecture for CSCE 791: Seminar in Advances in Computing
University of South Carolina

2. Human-AI Collaboration for Next-Generation Emergency Services
Outline
¨ Summary of research thrusts
¨ Focus: social media & city services during crises
¨ Problem 1. Modeling human errors in human-in-the-loop AI
system design
¨ Problem 2. Human workload-aware serviceability ranking
system design
¨ Future directions
2

3. Human-AI Collaboration for Next-Generation Emergency Services
Broad Research Area
3
¨ Human-centered Computing
n Sebe (2010) – “integrating human sciences (e.g. social & cognitive)
and computer science (e.g. machine learning) methods
for the design of computing systems with a human focus,
which should consider the personal, social, and cultural contexts
in which such systems are deployed”
My focus on
Social Media Mining &
Semantic Text Analytics
for real-time processing
systems at city services

4. Human-AI Collaboration for Next-Generation Emergency Services
Lab’s Research Thrusts
4
¨ [Natural Crises] Social Media Mining for Crisis Communication
¤ Extracting actionable posts in a new crisis using transfer & active learning
¤ Ranking serviceable requests for help on social media
¤ Human workload-aware ranking system design
¨ [Societal Crises] Semantic Analysis for Human Behavior Modeling
¤ Defining intent behind harmful behaviors on social media: Stereotyping, Hate
¤ Mining malicious stereotypical behavior against women for negative social construction
¤ Identifying factors affecting diffusion and mitigation of hate and disinformation
¨ [Cyber Crises] Text Comprehension Modeling for Cyber Defense
¤ Manipulating text comprehensibility to generate deceptive content
¤ Estimating believability for deceptive content

5. Human-AI Collaboration for Next-Generation Emergency Services
Outline
¨ Summary of research thrusts
¨ Focus: social media & city services during crises
¨ Problem 1. Modeling human errors in human-in-the-loop AI
system design
¨ Problem 2. Human workload-aware serviceability ranking
system design
¨ Future directions
5

6. Human-AI Collaboration for Next-Generation Emergency Services
Current Work at EM Services
6
World
Events
EM Response &
Decision Making
Human Workers
Information
Processing
CURRENT
Reliable but
small-scale
Accurate but
high workload
Data
Collection

7. Human-AI Collaboration for Next-Generation Emergency Services
Motivation
7
When traditional call-for-help EM services are overwhelmed ..
Source: https://www.usatoday.com/story/news/nation-now/2017/08/27/desperate-help-flood-victims-houston-turn-twitter-rescue/606035001/
Help
Offering
Help
Seeking
People resolving
to Social Media
How to
discover?

8. Human-AI Collaboration for Next-Generation Emergency Services
Future of Work at EM Services
8
World
Events
EM Response &
Decision Making
Human Worker +
AI agent
Data
Collection
Information
Processing
FUTURE
Noisy but
Large-scale
Faster but
Inaccurate

9. Human-AI Collaboration for Next-Generation Emergency Services
Future of Work at EM Services
9
World
Events
EM Response &
Decision Making
Human Worker +
AI agent
Data
Collection
Information
Processing
FUTURE
How to improve
AI Mental Model
with Worker
Mental Model?
Noisy but
Large-scale
Faster but
Inaccurate

10. Human-AI Collaboration for Next-Generation Emergency Services
Matching Mental Models of Human & AI
Agent: How to design Human-in-the-loop AI system?
10
Human Worker +
AI Agent
Information Processing Tasks
1. Filtering
• Classification Problem
2. Prioritization
• Ranking Problem
..
1. Adapt to
classify relevant
items in a
data stream
2. Adapt to rank
top-K items for
human
intervention

11. Human-AI Collaboration for Next-Generation Emergency Services
Human-in-the-loop AI System Design:
Awareness for human factors
11
Human Worker +
AI Agent
Information Processing Tasks
1. Filtering
• Classification Problem
2. Prioritization
• Ranking Problem
..
1. Active Learning
for Relevancy
Classification
Depends on
Annotator
Reliability

12. Human-AI Collaboration for Next-Generation Emergency Services
Human-in-the-loop AI System Design:
Awareness for human factors
12
Human Worker +
AI Agent
Information Processing Tasks
1. Filtering
• Classification Problem
2. Prioritization
• Ranking Problem
..
2. Adaptive
Top-K ranking
alerts for
human
Affects
Human
Workload
1. Active Learning
for Relevancy
Classification
Depends on
Annotator
Reliability

13. Human-AI Collaboration for Next-Generation Emergency Services
Outline
¨ Summary of research thrusts
¨ Focus: social media & city services during crises
¨ Problem 1. Modeling human errors in human-in-the-loop AI
system design
¨ Problem 2. Human workload-aware serviceability ranking
system design
¨ Future directions
13

14. Human-AI Collaboration for Next-Generation Emergency Services
Human-in-the-loop AI System Design:
Awareness for human factors
14
Human Worker +
AI Agent
1. Active Learning
for Relevancy
Classification
Depends on
Annotator
Reliability
What if
system
causes
human
errors?

15. Human-AI Collaboration for Next-Generation Emergency Services
Problem 1: How to Reduce Annotator Errors
15
Understanding potential human error causes using psychology theories
¨ Annotator burnout(Marshall and Shipman, 2013)
¨ Cognitive bias for answer positions(Burghardt, Hogg, and Lerman, 2018)
¨ Human error in execution(Reason, 1990; Zhang et al., 2004)
Mistakes
Errors due to incorrect or incomplete
knowledge
Faulty heuristics
Slips
Errors in the presence of correct and
complete knowledge
Loss of activation
[Pandey, Castillo, & Purohit, ASONAM’19]

16. Human-AI Collaboration for Next-Generation Emergency Services
Problem 1: How to Reduce Annotator Errors
16
Ø Hypothesis: Serial ordering of instances given to the human
annotator may cause him/her errors due to a mistake or slip.
{c4, c1, c2, c3, c1, c3, c4, c1, c4, c1, c4, c2, c1, c4, c1, c2, c4, c2, c4, c3}
How likely an
annotator would
make error on this
3rd occurrence due
to the potential
decay in memory?
Instance class
Motivation: Memory Decay, Ebbinghaus Curve(Ebbinghaus,2013)
[Pandey, Castillo, & Purohit, ASONAM’19]

17. Human-AI Collaboration for Next-Generation Emergency Services
Problem 1: How to Reduce Annotator Errors
17
Type of Error Potential Cause Mitigation Approach
Slips induced by
time constraints
• Concept forgotten • Show reminder for concept
examples
Mistakes induced by
serial ordering
• Concept not acquired yet
or forgotten
• Show frequent learning
examples
Slips induced by
serial ordering
• Presence of
a high-availability concept or
a low-availability concept
• Limit extreme divergence
from base rate
Preliminary framework to study human factors in active learning
Ø Hypothesis: Serial ordering of instances given to the human
annotator may cause him/her errors due to a mistake or slip.
[Pandey, Castillo, & Purohit, ASONAM’19]

18. Human-AI Collaboration for Next-Generation Emergency Services
Problem 1: How to Reduce Annotator Errors
18
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1st 2nd 3rd
Average
Error
Position
¨ Crowdsourcing annotation
testing experiment
¤ 20 ordered instances per
schedule with specific
class positions
¤ 6 such schedules
¤ 10 human annotator per
task
p-value
0.005
Annotation Schedule: {c4, c1, c2, c3, c1, c3, c4, c1, c4, c1, c4, c2, c1, c4, c1, c2, c4, c2, c4, c3}
Forgetting or Memory Decaying
behavior exists
[Pandey, Castillo, & Purohit, ASONAM’19]

19. Human-AI Collaboration for Next-Generation Emergency Services
Problem 1: How to Reduce Annotator Errors
¨ Sigmoid function to
model error probability
for an ordered instance
¨ Lab annotation testing
¤ 3 human annotator
¤ 800 ordered instances
with the induced error
19
[Pandey, Castillo, & Purohit, ASONAM’19]
Forgetting or Memory Decaying
behavior resembles sigmoid function.

20. Human-AI Collaboration for Next-Generation Emergency Services
Problem 1: How to Reduce Annotator Errors
Generate
Annotation-Schedule
Sample instances
Minimizing
human memory
decaying score
Maximizing
streaming model
performance
1. Sample instances from decision boundary
range of active learning model
• Prediction probability in [30%, 70%]
2. Maintain a class label Cdiscarded for each
interval to avoid samples predicted with
Cdiscarded labels
• Choose Cdiscarded based on
• If the class is appearing too frequent
• If the class is adding noise to the
streaming model
20
Solution: Error-avoiding Annotation Schedule to augment both human & model performance
[Pandey, Castillo, & Purohit, ASONAM’19]

21. Human-AI Collaboration for Next-Generation Emergency Services
21
[Pandey, Castillo, & Purohit, ASONAM’19]
Solution: Error-avoiding Annotation Schedule to augment both human & model performance
Problem 1: How to Reduce Annotator Errors

22. Human-AI Collaboration for Next-Generation Emergency Services
Problem 1: How to Reduce Annotator Errors
22
Human Error-Mitigating Sampling Algorithm outperforms in most cases!
[Pandey, Castillo, & Purohit, ASONAM’19]

23. Human-AI Collaboration for Next-Generation Emergency Services
Outline
¨ Summary of research thrusts
¨ Focus: social media & city services during crises
¨ Problem 1. Modeling human errors in human-in-the-loop AI
system design
¨ Problem 2. Human workload-aware serviceability ranking
system design
¨ Future directions
23

24. Human-AI Collaboration for Next-Generation Emergency Services
Human-in-the-loop AI System Design:
Awareness for human factors
24
Human Worker +
AI Agent
2. Adaptive
Top-K ranking
alerts for
human Affects
Human
Workload
Can you
increase
human
control or
agency?

25. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2: How to Create Human Workload-
aware Serviceability Ranking System
25
Image: https://blog.bufferapp.com/twitter-timeline-algorithm
BEYOND TIME & CREDIBILITY,
RANK BY
Serviceability
Can you
increase my
control or
agency?
End user
(Servicer)

26. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2: Workload-aware Serviceability
Ranking: Designing for human-AI collaboration
26
¨ Problem: how many and how often generate the request
alerts to respond for a human servicer (cause him workload!)
High Recall can cause
more work for a
time-crunched
Servicer!
Low Recall can cause
missing important
requests for a
Servicer!
RECALL (Machine/System metric)
WORKLOAD
(Human metric)
Ineffective
Inefficient
Worst
Desired
Optimal
Solution
[Purohit, Castillo, Imran, & Pandey, WI’18]

27. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2: Workload-aware Serviceability
Ranking: Designing for human-AI collaboration
27
Streaming
Requests
tij corresponds to
time period - when
to check requests,
e.g., 10 mins.
Row k corresponds
to the selection of
top-k ranked
requests to check
Ranked Requests Performance Metrics Estimation Dynamic Policy Selection
A cell tuple
corresponds to the
attainable
(Recall, Workload)
Choose a config,
e.g., k=10, tij=30,
and (R,W) = (90,20)
[Purohit, Castillo, Imran, & Pandey, WI’18]

28. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2: Workload-aware Serviceability
Ranking: Approach summary
28
Serviceability
Categorization
and Ranking
Ranking-
Workload
(RW) Matrix
Generation
Optimal RW
Policy Selection

29. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2. Serviceability Model: using Qualitative
Knowledge extracted from domain guides
29
Explicit
Request
E(m)
Answerable
Query
A(m)
Sufficiently
Detailed
D(m)
Correctly
Addressed
C(m)
Serviceability(m) = f ( E(m), A(m), D(m), C(m) )
Explicitly asks for a
resource or service
Explicitly asks a question
that can be answered
Sent to organization or
person who could have
resources or provide the
service, an alarm, or
could answer questions
Specifying contextual
information: time (when),
location (where),
quantity (how much),
resource (which)
[Purohit, Castillo, Imran, &
Pandey, ASONAM’18]

30. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2. Serviceability Model: Quantifying
Characteristics
(Anonymized) Message Explicit Answer-
able
Addressed Detailed
@account1 please, governor, post a phone # for
specific info in our local areas
4.3 4.3 3.3 3.7
@account2 is thr parking at McMahon for volunteer? 4.0 5.0 5.0 5.0
@account3 how can I help 1.3 4.3 4.3 1.0
@account4 Plz pray for these families 1.7 1.0 1.0 1.0
@account5 been working in #LAFlood shelter, we
actively monitor SM for feedback
1.0 1.0 2.0 2.0
“@account7 No matter where in the world ur
followers live, you can donate from link Plz RT
1.0 1.0 1.0 1.0
¨ E(m), A(m), C(m), D(m) : Likert Scale Functions [score:1-5]
30
Illustration Table: Average scores of Likert ratings by crowd annotators

31. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2. Serviceability Model: Learning-to-Rank
System Design
31
[Purohit, Castillo, Imran, & Pandey, ASONAM’18]

32. Human-AI Collaboration for Next-Generation Emergency Services
Serviceability Model: Examples of resulting
ranked requests
33
Ranked Messages by T (text)+I (Inferred) Modeling Scheme
TOP-2
[Sandy]
- @_USER_ please, governor, post a website or phone# where we can get
specific info for our local areas
- @_USER_ Queens trains aren’t being addressed at all. When can v expect any
service updates for the NQR trains?
BOTTOM-2
[Sandy]
- @_USER_ Romney not going2like that gov christie is being nice about Obama’s
leadership
- @_USER_ HILARIOUS! That’s much needed laughter, I am sure.
TOP-2
[Alberta]
- @_USER_ can you tell me if sanitary pumps are running yet in elbow park?
#yycflood
- @_USER_ plz text with what you need & address. Lots of volunteers in mission
BOTTOM-2
[Alberta]
- @_USER_ thank u calgary police
- @_USER_ Tx for ur time!!

33. Human-AI Collaboration for Next-Generation Emergency Services
Workload-aware Serviceability Ranking:
Designing for human-AI collaboration
34
Streaming
Requests
Ranked Requests Performance Metrics Estimation Dynamic Policy Selection
[Purohit, Castillo, Imran, & Pandey, WI’18]
Image: https://commons.wikimedia.org/wiki/File:Front_pareto.svg
Pareto Optimization
à Given the lack of user
preference apriori, rely
on for non-dominated
sorting
Ranking-Workload (RW) Matrix

34. Human-AI Collaboration for Next-Generation Emergency Services
Workload-aware Serviceability Ranking:
Ranking-Workload (RW) Matrix
35
¨ Define RW Matrix to model the relationship between human &
machine performances for a request-set 𝑥𝑖𝑗 in time 𝑡𝑖𝑗
n 𝑅𝑊 (𝑘, 𝑡𝑖𝑗) = ⟨ 𝑀(𝑅(𝑥𝑖𝑗)), 𝑤 𝑡𝑖𝑗, 𝑘 ⟩
¤ Machine Performance Metric (for a ranking system 𝑅(𝑥𝑖𝑗)): 𝑀(𝑅(𝑥𝑖𝑗))
n Recall@k, e.g., no. of relevant requests in top-k
n Precision@k
¤ Human Performance Metric: 𝑤(𝑡𝑖𝑗, 𝑘)
n Cognitive Load, e.g. hourly rate of requests to read
n Time-on-Task
[Purohit, Castillo, Imran, & Pandey, WI’18]

35. Human-AI Collaboration for Next-Generation Emergency Services
Workload-bound Serviceability Ranking:
Pareto-Optimal RW Policy Selection
36
Given the lack of user
preference apriori, rely on
Pareto Optimization[Ross, 1973]
for the non-dominated
selection
Can you
recommend
me to choose?
Image: https://commons.wikimedia.org/wiki/File:Front_pareto.svg

36. Human-AI Collaboration for Next-Generation Emergency Services
Workload-bound Serviceability Ranking:
Experimental Setup
37
¨ Used relevancy data as alerts from 6 crisis events in our prior work,
where relevancy is ‘serviceability’[Purohit, Castillo, Imran, & Pandey, ASONAM18] of a
message for response
Event (Year, start day – end day) Tweets Relevant Irrelevant
Hurricane Sandy (2012, 10/27-11/07) 1,153 40% 60%
Oklahoma Tornado (2013, 05/20-05/29) 1,513 48% 52%
Alberta Floods (2013, 06/16-06/16) 2,727 28% 72%
Nepal Earthquake (2015, 04/15-05/15) 2,222 18% 82%
Louisiana Floods (2016, 10/11-10/31) 1,369 34% 66%
Hurricane Harvey (2017, 08/29-09/15) 12,742 20% 80%

37. Human-AI Collaboration for Next-Generation Emergency Services
Workload-bound Serviceability Ranking:
Experimental Setup
38
¨ Compared two algorithms for recommending RW policy:
¤ Periodic algorithm
n process requests posted in the time window of past H (24) hrs.
n generate top-k ranking and a RW matrix at the beginning of
every hour (e.g., 7am, 8am)
¤ Near-Realtime algorithm
n process requests posted in the time window of past G (60) mins.
n generate top-k ranking and a RW matrix at the beginning of
every minute (e.g., 7:01am, 7:02am)

38. Human-AI Collaboration for Next-Generation Emergency Services
Workload-bound Serviceability Ranking:
Experiment 1 – RW trade-off validation analysis
39
[Purohit, Castillo, Imran, & Pandey, WI’18]
Multiple
Recall values
for a given
workload
budget!

39. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2. Workload-aware Serviceability
Ranking: Experiment – Greedy-recall baseline comparison
40
Pareto-optimal
Periodic RW
recommendations
give lower workload
in contrast to max.
recall-based policy.

40. Human-AI Collaboration for Next-Generation Emergency Services
Problem 2. Workload-aware Serviceability
Ranking: Experiment – Greedy-workload baseline
41
Pareto-optimal
Periodic RW
recommendations
give higher recall in
contrast to min.
workload-based
policy.

41. Human-AI Collaboration for Next-Generation Emergency Services
Conclusion: Lessons, Limitations, and Future Work
42
¤ A human-AI collaboration approach can help in scalable
stream data processing for the emergency services
n Combining Human Factors + AI systems
¤ Lessons learned:
n Serviceability characteristics of information capture the notion of relevance
and serviceability for social media requests to online public services.
n Workload-aware serviceability ranking provides a Human-AI Collaboration
design to seamlessly incorporate user choices in the system design.

42. Human-AI Collaboration for Next-Generation Emergency Services
Conclusion: Lessons, Limitations, and Future Work
43
¨ Limitations & opportunities:
¤ Serviceability model
n Study non-English language request messages
n Explore multiple as opposed to single platform based datasets
n Twitter vs. Forum
n Include indirectly addressed requests (i.e. not starting with @user)
¤ Human-AI collaboration
n Extend the human performance metrics in the Ranking-Workload matrix
n Incorporate bias of the performance metrics in the RW matrix
n Adapt the workload-aware serviceability approach to other domains

43. Human-AI Collaboration for Next-Generation Emergency Services
44
Applications:
CitizenHelper-Adaptive Tool: Expert-augmented Streaming Analytics
System for Emergency Services and Humanitarian Organizations
[Pandey & Purohit, ASONAM’18]

44. Human-AI Collaboration for Next-Generation Emergency Services
Applications:
Human-Annotation for Crowdsourcing Work
45
Concept for class c2
not acquired yet
– Mistakes
Imbalanced
presence of class c1
– Slips

45. Human-AI Collaboration for Next-Generation Emergency Services
Applications:
Working with CERTs
46
Assisting regional CERT organizations for rapid social media filtering
for COVID-19 response using the tool developed under NSF CRII
project, CitizenHelper Tool, leading to a new NSF RAPID grant!

46. Human-AI Collaboration for Next-Generation Emergency Services
Future Work: Human-AI Collaboration at
Workplaces of Various City Services
47
Q2.
How to classify
relevant
content in
online streams
in a new event
domain?
[ECML’20, ASONAM’20,
SBP-BRiMS’18]
Q3.
How to rank &
semantically group
serviceable,
actionable request
content?
[SNAM’20, ASONAM’18]
Q4.
How many & when
to present requests
to a worker with
dynamic workload?
[ASONAM’18, WI’18]
Data
Stream
City
Service
Worker
Filtering Prioritization Human-Machine
Interaction
Q1.
How to sample &
order instances
for human
annotation, to
improve labeled
data quality?
[ASONAM’19, IJHCS (under
review)]
Human
Annotation
opportunity for fundamental research in AI with Human-Centered Computing
CitizenHelper
Tool

47. Human-AI Collaboration for Next-Generation Emergency Services
More about our research:
http://ist.gmu.edu/~hpurohit/informatics-lab.html
CONTACT: hpurohit@gmu.edu
Acknowledgement:
Image sources, collaborators (especially Prof. Carlos Castillo, Prof. Valerie Shalin, Dr. Muhammad Imran);
U.S. DHS Science & Technology SMWGESDM Researcher-Practitioner Subgroup (especially Steve Peterson),
Human_Info_lab students as well as sponsors:
Questions?
48
Primary grants that supported this work:
• IIS #1657379, IIS #1815459