Automatically Rank Social Media Requests for Emergency Services using Serviceability Model - ASONAM18

Social-EOC: Serviceability Model To Rank Social Media
Requests for Emergency Operation Centers
Hemant Purohit, Rahul Pandey
Humanitarian, Semantics &
Informatics Lab (Human_Info_Lab)
The 2018 IEEE/ACM International Conference on
Advances in Social Networks Analysis and Mining (ASONAM-2018)
Barcelona, Spain
Aug 29, 2018
Carlos Castillo
Web Science & Social
Computing Research Group
Muhammad Imran
Crisis Computing & Social
Computing Group
@hemant_pt @chaToX @mimran15 @gizmowiki

Serviceability Model for Social Media Services, ASONAM-18
Outline
¨ Introduction
n Social media & web streams during disasters
n Problem: filter and rank requests for help
¨ Background and Related Work
n Services in emergency management
n Social computing for emergency management
n Mining request / help-seeking intent
¨ Social-EOC: Serviceability Model
n Qualitative and quantitative formulation
n Social-EOC based Learning-to-Rank system
¨ Experiments
n Conversational data collection
n Learning-to-Rank evaluation
¨ Results
¨ Discussion and Future Work
2

Social Media & Web Use in Disasters
3
When traditional call systems are exhausted ..
Source: https://www.usatoday.com/story/news/nation-now/2017/08/27/desperate-help-flood-victims-houston-turn-twitter-rescue/606035001/

Mining Social & Web Data: Innovation
Opportunity for Emergency Services
4
But ..
WHAT & HOW
to filter?

Mining Social & Web Data: Relevancy issue
5

Mining Social & Web Data: Ranking issue
(Anonymized) Message Serviceable Degree
@_USER_ I am 9 ft above current water levels,
why am I told to evacuate Grand Lakes now?
Please advise.
serviceable
@_USER_ If there has been no rain since
yesterday, why is water not draining?
Serviceable but lacks details
@_USER_ Thank God you are working on this.
Let us chat when things settle down
not serviceable
¨ Illustration from Hurricane Harvey 2017
6

Problem
7
¨ Can we filter/classify and prioritize/rank serviceable
social media requests for emergency services?
¨ Scope:
¤ Study requests directly sent to the accounts of services
¤ Analyze the requests during a disaster event

Mining Social & Web Data: Challenges
¨ Variable Actionability
¤ Subjective relevance of messages, often task-
dependent
¨ Insufficient Samples
¤ Limited context representativeness of request samples
to directly learn ranking
8

Solution: Our Contributions
9
¨ Novel generalizable model (Social-EOC) to define serviceability
characteristics of a service request message on social media
¨ Learning-to-rank system based on the Social-EOC model
¤ Inference of the serviceability characteristics
¤ Classification and ranking of serviceable requests
¨ Evaluation of the Social-EOC model with the datasets from six
crisis events

Outline
¨ Introduction
n Mining request/ help-seeking intent
¨ Experiments
¨ Results
10

Services in Emergency Management (EM)
¨ Incident-Command System (ICS) model for the response coordination
11
Public Info. Officer (PIO)
provides the information
services [FEMA; Hughes & Palen,
JHSEM 2012]
FEMA: https://training.fema.gov/ programs/pio/ Source: https://en.wikipedia.org/wiki/Incident_Command_System

Social Computing for EM: Crisis Informatics
12
¨ Extensive literature on social media during disasters [Castillo,
Cambridge Press 2016; Imran et al., CSUR 2015]
¤ Addressed a variety of problems [c.f. Tutorial: Castillo, Diaz & Purohit, SDM 2014]
n Data collection and filtering[e.g., Olteanu et al., ICWSM 2014]
n Modeling actionable help behavior [e.g., Purohit et al., SocialCom 2015]
n Summarization [e.g., Rudra et al., HyperText 2016]
n Information diffusion [e.g., Starbird & Palen, ISCRAM 2010]
n Information and source credibility [e.g., Castillo et al., WWW 2011]
n Visual analytics [e.g., Kumar et al., ICWSM 2011]
n ..
¨ Open problem: finding formal characteristics of relevant
social media requests that must be prioritized

Mining Request/Seeking Intent
13
REQUEST MEDIUM TYPE FOCUS
Emails
Finding and ranking messages to reply with
different interpersonal communication
behavior for information seeking
[Yang et al., SIGIR 2017; Lampert et al., HLT 2010]
Q&A Forums
Generic seeking behavior for all types of
users and often not targeted towards
answers from a specific organization/group
[Mai, Emerald 2016; Vasilescu et al., CSCW 2014]
Social Media Chats
Finding explicit or implicit requests for help
during disaster relief and users who could
answer queries [Purohit et al., First Monday 2013;
He et al., WebSci 2017; Ranganath et al., TKDE 2017;
Sachdeva & Kumaraguru, CSCW 2017]

Outline
¨ Introduction
n Mining requests / help-seeking intent
¨ Experiments
¨ Results
14

Social-EOC Serviceability Model:
Qualitative Domain Knowledge based on FEMA PIO Guide
15
Explicit
Request
E(m)
Answerable
Query
A(m)
Sufficiently
Detailed
D(m)
Correctly
Addressed
C(m)
Serviceability(m) = f ( E(m), A(m), D(m), C(m) )
Explicitly asks for a
resource or service
Explicitly asks a question
that can be answered
Sent to organization or
person who could have
resources or provide the
service, an alarm, or
could answer questions
Specifying contextual
information: time (when),
location (where),
quantity (how much),
resource (which)

Serviceability Model: Quantifying Characteristics
(Anonymized) Message Explicit Answer-
able
Addressed Detailed
@account1 please, governor, post a phone # for
specific info in our local areas
4.3 4.3 3.3 3.7
@account2 is thr parking at McMahon for volunteer? 4.0 5.0 5.0 5.0
@account3 how can I help 1.3 4.3 4.3 1.0
@account4 Plz pray for these families 1.7 1.0 1.0 1.0
@account5 been working in #LAFlood shelter, we
actively monitor SM for feedback
1.0 1.0 2.0 2.0
“@account7 No matter where in the world ur
followers live, you can donate from link Plz RT
1.0 1.0 1.0 1.0
¨ E(m), A(m), C(m), D(m) : Likert Scale Functions [score:1-5]
16
Illustration Table: Average scores of Likert ratings by crowd annotators

Learning-to-Rank System Design: Using Social-
EOC Model
17

Learning-to-Rank System Design: Steps
18
1. Collecting conversation chains of seed messages
n A seed message mentions a targeted account of official services (e.g.,
@fboem for ‘Fort Bend County OEM’) identified using gov./news reports
2. Rating serviceability characteristics of a request
n Also additional category ‘Other’: advertisements, jokes, ..
n Annotation by crowdsourcing workers
3. Creating gold standard of serviceable requests
n Binary classes: Serviceable (relevant), Not-Serviceable (Not relevant)
n Annotation by Emergency Management professionals
4. Learning to classify and rank serviceable requests
n SVM-Rank
n Relevance grade levels: binary

Outline
¨ Introduction
¨ Related Work
n Mining requests / help-seeking intent
¨ Experiments and Results
n Results
19

Experiments
20
¨ Data
¤ Collected Twitter messages related to 6 crisis events:
n Big: Hurricane Harvey 2017, Nepal Earthquake 2015, Alberta Floods 2013
n Small: Louisiana Floods 2016, Oklahoma Tornado 2013, Hurricane Sandy 2012
¤ Crawl Twitter conversation chains for seed messages

Experiments
21
¨ Feature (f.) extraction for Learning-to-Rank method
¤ Generic f.: number of words, hashtags, mentions, and URLs in a tweet
¤ Text f.: tf-idf with Bag-of-Words representation
¤ Manual serviceable f.: average annotation rating between 1 to 5
for each serviceable characteristic
¤ Inferred serviceable f.: 0 or 1
n Binary classifier for each characteristic (rating 1-2: negative, 3-5: positive
class}
¨ Evaluation metric for top-K ranked results
¤ Normalized Discounted Cumulative Gain (NDCG)
¤ NDCG @10

Experiments: Evaluation Schemes
22
LTR models based on varied features (f.):
¨ [T]: Text f. and generic f. (baseline)
¨ [T+I]: T and inferred serviceable f. from the model built on the
same event data
¤ [T + I_all]: T and inferred serviceable f. from the model built
on the all events data
¤ [T + I_cross]: T and inferred serviceable f. from the model
built on the cross events data (all except the current)
¤ [T_cross + I_cross]: both T and inferred serviceable f. from the
model built on the cross events data (all except the current)
¨ [T+M]: T + manual serviceable f.

Results
1. Models with inferred
serviceable features
are generally better
than the baseline.
2. Performance varies in
cases of small datasets.
n less training examples
n imbalanced
23

Results
3. Cross-event models
perform well.
n especially for smaller
datasets
4. Serviceability
characteristics based
features are among the
best discriminators.
n among the top-5 features,
identified using χ2 test
24

Discussion: Examples of Resulting Ranked Requests
25
Ranked Messages by T (text)+I (Inferred) Modeling Scheme
TOP-2
[Sandy]
- @_USER_ please, governor, post a website or phone# where we can get
specific info for our local areas
- @_USER_ Queens trains aren’t being addressed at all. When can v expect any
service updates for the NQR trains?
BOTTOM-2
[Sandy]
- @_USER_ Romney not going2like that gov christie is being nice about Obama’s
leadership
- @_USER_ HILARIOUS! That’s much needed laughter, I am sure.
TOP-2
[Alberta]
- @_USER_ can you tell me if sanitary pumps are running yet in elbow park?
#yycflood
- @_USER_ plz text with what you need & address. Lots of volunteers in mission
BOTTOM-2
[Alberta]
- @_USER_ thank u calgary police
- @_USER_ Tx for ur time!!

Discussion: Future Application for Prioritization of
Streaming Messages
26
Serviceability
Source: https://blog.bufferapp.com/twitter-timeline-algorithm
BEYOND TIME & CREDIBILITY,
RANK BY

Discussion: Lessons, Limitations and Future Work
27
¨ Lessons learned:
n Serviceability characteristics capture the notion of relevance and
serviceability for social media requests to organizational services.
n Experimented for binary relevance (serviceability) grades, but the method is
extensible for multiple grade levels.
¨ Limitations & opportunities for future work
n Study non-English language request messages
n Explore multiple as opposed to single platform based datasets
n Include indirectly addressed request messages (i.e. not starting with @user)
n Generate optimal request alerts to respond
n Our new work: Purohit, Castillo, Imran, Pandey (2018, to appear). Ranking of Social
Media Alerts with Workload Bounds in Emergency Operation Centers. Web Intelligence.

Conclusion
¨ Demonstrated that social media requests have some common
core characteristics for helping rank the serviceable requests
for organizational services.
¨ Presented a novel qualitative-quantitative model for
serviceable request characteristics for emergencies and
demonstrated its use by Learning-to-Rank (LTR) methodology.
¨ Evaluated LTR systems based on inclusion of serviceability
characteristics features across six disaster events and noted a
superior performance (gain up to 25% in nDCG@10 and nDCG@5).
28

PAPER: http://ist.gmu.edu/~hpurohit/informatics-
lab/papers/serviceability_ranking_disasters_ASONAM18_final.pdf
CONTACT: hpurohit@gmu.edu
Acknowledgement:
image sources, collaborators (especially Profs. Amit Sheth, Valerie Shalin, & TK Prasad at Kno.e.sis Center;
U.S. DHS Science & Technology SMWGESDM Researcher-Practitioner Subgroup), Human_Info_lab Alumni
(Yogen, Sharan) as well as sponsors:
Questions?
29
Grants:
IIS #1657379,
IIS #1815459
La Caixa project:
LCF/PR/PR16/11110009

Automatically Rank Social Media Requests for Emergency Services using Serviceability Model - ASONAM18

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Automatically Rank Social Media Requests for Emergency Services using Serviceability Model - ASONAM18

Similar to Automatically Rank Social Media Requests for Emergency Services using Serviceability Model - ASONAM18 (20)

More from Hemant Purohit

More from Hemant Purohit (7)

Recently uploaded

Recently uploaded (20)

Automatically Rank Social Media Requests for Emergency Services using Serviceability Model - ASONAM18