SlideShare a Scribd company logo
1 of 30
Download to read offline
R by example: mining Twitter for consumer
attitudes towards airlines

presented at the

Boston Predictive Analytics
MeetUp
by


Jeffrey Breen
President
Cambridge Aviation Research

jbreen@cambridge.aero

June 2011




  Cambridge Aviation Research   • 245 First Street • Suite 1800 • Cambridge, MA 02142 • cambridge.aero




                                                                                                         © Copyright 2010 by Cambridge Aviation Research. All rights reserved.
Airlines top customer satisfaction... alphabetically




http://www.theacsi.org/                                     3
Actually, they rank below the Post
    Office and health insurers




                                     4
which gives us plenty to listen to
                               Completely unimpressed with @continental or @united.
RT @dave_mcgregor:       Poor communication, goofy reservations systems and
Publicly pledging to                       all to turn my trip into a mess.
never fly @delta again.
The worst airline ever.
U have lost my patronage     @united #fail on wifi in red carpet clubs (too
forever due to ur            slow), delayed flight, customer service in red
incompetence                 carpet club (too slow), hmmm do u see a trend?



@United Weather delays may not be your fault,
but you are in the customer service business.
It's atrocious how people are getting treated!
We were just told we are delayed 1.5        @SouthwestAir I know you don't make the
hrs & next announcement on @JetBlue -      weather. But at least pretend I am not a
“We're selling headsets.” Way to           bother when I ask if the delay will make
capitalize on our misfortune.                                    miss my connection
         @SouthwestAir
     I hate you with every            Hey @delta - you suck! Your prices
    single bone in my body          are over the moon & to move a flight
   for delaying my flight by         a cpl of days is $150.00. Insane. I
   3 hours, 30mins before I              hate you! U ruined my vacation!
    was supposed to board.
             #hate
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                14
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                15
Searching Twitter in one line
R’s XML and RCurl packages make it easy to grab web data, but Jeff
Gentry’s twitteR package makes searching Twitter almost too easy:

> # load the package
> library(twitteR)
> # get the 1,500 most recent tweets mentioning ‘@delta’:
> delta.tweets = searchTwitter('@delta', n=1500)




See what we got in return:              A “list” in R is a collection of
                                        objects and its elements may be
> length(delta.tweets)                  named or just numbered.
[1] 1500
> class(delta.tweets)
[1] "list"
                                        “[[ ]]” is used to access elements.
Examine the output
Let’s take a look at the first tweet in the output list:

    > tweet = delta.tweets[[1]]
                                       tweet is an object of type “status”
                                       from the “twitteR” package.
    > class(tweet)
    [1] "status"
    attr(,"package")                   It holds all the information about
    [1] "twitteR"                      the tweet returned from Twitter.



The help page (“?status”) describes some accessor methods like
getScreenName() and getText() which do what you would expect:

    > tweet$getScreenName()
    [1] "Alaqawari"
    > tweet$getText()
    [1] "I am ready to head home. Inshallah will try to get on the earlier
    flight to Fresno. @Delta @DeltaAssist"
Extract the tweet text
R has several (read: too many) ways to apply functions iteratively.
•The plyr package unifies them all with a consistent naming convention.
•The function name is determined by the input and output data types. We
have a list and would like a simple array output, so we use “laply”:

> delta.text = laply(delta.tweets, function(t) t$getText() )


> length(delta.text)[1] 1500
> head(delta.text, 5)
[1] "I am ready to head home. Inshallah will try to get on the earlier
flight to Fresno. @Delta @DeltaAssist"
[2] "@Delta Releases 2010 Corporate Responsibility Report - @PRNewswire
(press release) : http://tinyurl.com/64mz3oh"
[3] "Another week, another upgrade! Thanks @Delta!"
[4] "I'm not able to check in or select a seat for flight DL223/KL6023 to
Seattle tomorrow. Help? @KLM @delta"
[5] "In my boredom of waiting realized @deltaairlines is now @delta
seriously..... Stil waiting and your not even unloading status yet"
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                19
Estimating Sentiment

There are many good papers and resources describing methods to
estimate sentiment. These are very complex algorithms.



For this tutorial, we use a very simple algorithm which assigns a score by
simply counting the number of occurrences of “positive” and “negative”
words in a tweet. The code for our score.sentiment() function can be
found at the end of this deck.


Hu & Liu have published an “opinion lexicon” which categorizes
approximately 6,800 words as positive or negative and which can be
downloaded.


            Positive: love, best, cool, great, good, amazing
            Negative: hate, worst, sucks, awful, nightmare
                                                                        20
Load sentiment word lists
1. Download Hu & Liu’s opinion lexicon:


   http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html


2. Loading data is one of R’s strengths. These are simple text files,
though they use “;” as a comment character at the beginning:

   > hu.liu.pos = scan('../data/opinion-lexicon-English/positive-
   words.txt', what='character', comment.char=';')

   > hu.liu.neg = scan('../data/opinion-lexicon-English/negative-
   words.txt', what='character', comment.char=';')



3. Add a few industry-specific and/or especially emphatic terms:

   > pos.words = c(hu.liu.pos, 'upgrade')         The c() function
   > neg.words = c(hu.liu.neg, 'wtf', 'wait',     combines objects
     'waiting', 'epicfail', 'mechanical')         into vectors or lists
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                22
Algorithm sanity check
    > sample = c("You're awesome and I love you",
          "I hate and hate and hate. So angry. Die!",
          "Impressed and amazed: you are peerless in your achievement of
          unparalleled mediocrity.")
    > result = score.sentiment(sample, pos.words, neg.words)
    > class(result)
                                   data.frames hold tabular data so they
    [1] "data.frame"
                                   consist of columns & rows which can
    > result$score
                                   be accessed by name or number.
    [1]   2 -5   4
                                   Here, “score” is the name of a column.


So, not so good with sarcasm. Here are a couple of real tweets:

    > score.sentiment(c("@Delta I'm going to need you to get it together.
    Delay on tarmac, delayed connection, crazy gate changes... #annoyed",
    "Surprised and happy that @Delta helped me avoid the 3.5 hr layover I
    was scheduled for. Patient and helpful agents. #remarkable"),
    pos.words, neg.words)$score
    [1] -4   5
Accessing data.frames
Here’s the data.frame just returned from score.sentiment():
   > result
       score                                                                                     text

   1           2                                                         You're awesome and I love you

   2      -5                                                   I hate and hate and hate. So angry. Die!

   3           4 Impressed and amazed: you are peerless in your achievement of unparalleled mediocrity.



Elements can be accessed by name or position, and positions can be
ranges:
   > result[1,1]
   [1] 2
   > result[1,'score']
   [1] 2
   > result[1:2, 'score']
   [1]         2 -5
   > result[c(1,3), 'score']
   [1] 2 4
   > result[,'score']
   [1]         2 -5     4
Score the tweets
To score all of the Delta tweets, just feed their text into
score.sentiment():

    > delta.scores = score.sentiment(delta.text, pos.words,     Progress bar
    neg.words, .progress='text')                                provided by
    |==================================================| 100%   plyr

Let’s add two new columns to identify the airline for when we
combine all the scores later:
    > delta.scores$airline = 'Delta'
    > delta.scores$code = 'DL’
Plot Delta’s score distribution
R’s built-in hist() function will create and plot histograms of your data:
    > hist(delta.scores$score)
The ggplot2 alternative
ggplot2 is an alternative graphics package which generates more refined
graphics:
   > qplot(delta.scores$score)
Lather. Rinse. Repeat
To see how the other airlines fare, collect & score tweets for other
airlines.


Then combine all the results into a single “all.scores” data.frame:

    > all.scores = rbind( american.scores, continental.scores, delta.scores,
    jetblue.scores, southwest.scores, united.scores, us.scores )



                                                rbind() combines
                                                rows from
                                                data.frames, arrays,
                                                and matrices
Compare score distributions
   ggplot2 implements “grammar of graphics”, building plots in layers:
       > ggplot(data=all.scores) + # ggplot works on data.frames, always
            geom_bar(mapping=aes(x=score, fill=airline), binwidth=1) +
            facet_grid(airline~.) + # make a separate plot for each airline
            theme_bw() + scale_fill_brewer() # plain display, nicer colors




ggplot2’s faceting
capability makes it
easy to generate the
same graph for
different values of a
variable, in this case
“airline”.
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                30
Ignore the middle
Let’s focus on very negative (<-2) and positive (>2) tweets:
    > all.scores$very.pos = as.numeric( all.scores$score >= 2 )
    > all.scores$very.neg = as.numeric( all.scores$score <= -2 )


For each airline ( airline + code ), let’s use the ratio of very positive to
very negative tweets as the overall sentiment score for each airline:
    > twitter.df = ddply(all.scores, c('airline', 'code'), summarise,
    pos.count = sum( very.pos ), neg.count = sum( very.neg ) )
    > twitter.df$all.count = twitter.df$pos.count + twitter.df$neg.count
    > twitter.df$score = round( 100 * twitter.df$pos.count /
                 twitter.df$all.count )

Sort with orderBy() from the doBy package:
        > orderBy(~-score, twitter.df)
Any relation to ACSI’s airline scores?




http://www.theacsi.org/index.php?option=com_content&view=article&id=147&catid=&Itemid=212&i=Airlines

                                                                                                       18
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                33
Scrape, don’t type
XML package provides amazing readHTMLtable() function:
    > library(XML)
    > acsi.url = 'http://www.theacsi.org/index.php?
    option=com_content&view=article&id=147&catid=&Itemid=212&i=Airlines'
    > acsi.df = readHTMLTable(acsi.url, header=T, which=1,
    stringsAsFactors=F)
    > # only keep column #1 (name) and #18 (2010 score)
    > acsi.df = acsi.df[,c(1,18)]
    > head(acsi.df,1)
                         10
    1 Southwest Airlines 79



Well, typing metadata is OK, I guess... clean up column names, etc:

    > colnames(acsi.df) = c('airline', 'score')              NA (as in “n/a”) is
    > acsi.df$code = c('WN', NA, 'CO', NA, 'AA', 'DL',       supported as a
                       'US', 'NW', 'UA')                     valid value
    > acsi.df$score = as.numeric(acsi.df$score)              everywhere in R.
Game Plan
Search Twitter for
airline mentions &
collect tweet text            Score sentiment for    Summarize for each
                                  each tweet              airline
 Load sentiment
   word lists


                                                           Compare Twitter
                                                         sentiment with ACSI
                                                           satisfaction score
                       Scrape ACSI web site for
                     airline customer satisfaction
                                 scores




                                                                                35
Join and compare
merge() joins two data.frames by the specified “by=” fields. You can
specify ‘suffixes’ to rename conflicting column names:

    > compare.df = merge(twitter.df, acsi.df, by='code',
        suffixes=c('.twitter', '.acsi'))




Unless you specify “all=T”, non-matching rows are dropped (like a SQL
INNER JOIN), and that’s what happened to top scoring JetBlue.


With a very low score, and low traffic to boot, soon-to-disappear
Continental looks like an outlier. Let’s exclude:
    > compare.df = subset(compare.df, all.count > 100)
an actual result!
ggplot will even run lm() linear
(and other) regressions for you
 with its geom_smooth() layer:

> ggplot( compare.df ) +
geom_point(aes(x=score.twitter,
y=score.acsi,
color=airline.twitter), size=5) +
geom_smooth(aes(x=score.twitter,
y=score.acsi, group=1), se=F,
method="lm") +
theme_bw() +
opts(legend.position=c(0.2,
0.85))




                                         37
                                         21
http://www.despair.com/cudi.html
R code for example scoring function
    score.sentiment = function(sentences, pos.words, neg.words, .progress='none')
{
	   require(plyr)
	   require(stringr)
	
	   # we got a vector of sentences. plyr will handle a list or a vector as an "l" for us
	   # we want a simple array of scores back, so we use "l" + "a" + "ply" = laply:
	   scores = laply(sentences, function(sentence, pos.words, neg.words) {
	   	
	   	      # clean up sentences with R's regex-driven global substitute, gsub():
	   	      sentence = gsub('[[:punct:]]', '', sentence)
	   	      sentence = gsub('[[:cntrl:]]', '', sentence)
	   	      sentence = gsub('d+', '', sentence)
	   	      # and convert to lower case:
	   	      sentence = tolower(sentence)

	   	     # split into words. str_split is in the stringr package
	   	     word.list = str_split(sentence, 's+')
	   	     # sometimes a list() is one level of hierarchy too much
	   	     words = unlist(word.list)

	   	     # compare our words to the dictionaries of positive & negative terms
	   	     pos.matches = match(words, pos.words)
	   	     neg.matches = match(words, neg.words)
	
	   	     # match() returns the position of the matched term or NA
	   	     # we just want a TRUE/FALSE:
	   	     pos.matches = !is.na(pos.matches)
	   	     neg.matches = !is.na(neg.matches)

	   	     # and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum():
	   	     score = sum(pos.matches) - sum(neg.matches)

	   	      return(score)
	   }, pos.words, neg.words, .progress=.progress )

	   scores.df = data.frame(score=scores, text=sentences)
	   return(scores.df)
}                                                                                          39

More Related Content

What's hot

Selection sort and insertion sort
Selection sort and insertion sortSelection sort and insertion sort
Selection sort and insertion sortMay Ann Mendoza
 
Bruteforce algorithm
Bruteforce algorithmBruteforce algorithm
Bruteforce algorithmRezwan Siam
 
Sentiment analysis using naive bayes classifier
Sentiment analysis using naive bayes classifier Sentiment analysis using naive bayes classifier
Sentiment analysis using naive bayes classifier Dev Sahu
 
Heuristic Search Techniques Unit -II.ppt
Heuristic Search Techniques Unit -II.pptHeuristic Search Techniques Unit -II.ppt
Heuristic Search Techniques Unit -II.pptkarthikaparthasarath
 
how to calclute time complexity of algortihm
how to calclute time complexity of algortihmhow to calclute time complexity of algortihm
how to calclute time complexity of algortihmSajid Marwat
 
Clustering in artificial intelligence
Clustering in artificial intelligence Clustering in artificial intelligence
Clustering in artificial intelligence Karam Munir Butt
 
8 query processing and optimization
8 query processing and optimization8 query processing and optimization
8 query processing and optimizationKumar
 
Asymptotic notations
Asymptotic notationsAsymptotic notations
Asymptotic notationsEhtisham Ali
 
Data mining in social network
Data mining in social networkData mining in social network
Data mining in social networkakash_mishra
 
Boolean,vector space retrieval Models
Boolean,vector space retrieval Models Boolean,vector space retrieval Models
Boolean,vector space retrieval Models Primya Tamil
 
Infix to Prefix (Conversion, Evaluation, Code)
Infix to Prefix (Conversion, Evaluation, Code)Infix to Prefix (Conversion, Evaluation, Code)
Infix to Prefix (Conversion, Evaluation, Code)Ahmed Khateeb
 
3.1 clustering
3.1 clustering3.1 clustering
3.1 clusteringKrish_ver2
 
ANN(Artificial Neural Networks) Clustering Algorithms
ANN(Artificial  Neural Networks)  Clustering AlgorithmsANN(Artificial  Neural Networks)  Clustering Algorithms
ANN(Artificial Neural Networks) Clustering AlgorithmsAnuj Kumar Pathak
 
CS6007 information retrieval - 5 units notes
CS6007   information retrieval - 5 units notesCS6007   information retrieval - 5 units notes
CS6007 information retrieval - 5 units notesAnandh Arumugakan
 

What's hot (20)

Selection sort and insertion sort
Selection sort and insertion sortSelection sort and insertion sort
Selection sort and insertion sort
 
Bruteforce algorithm
Bruteforce algorithmBruteforce algorithm
Bruteforce algorithm
 
Problem Solving
Problem Solving Problem Solving
Problem Solving
 
Data For Datamining
Data For DataminingData For Datamining
Data For Datamining
 
Sentiment analysis using naive bayes classifier
Sentiment analysis using naive bayes classifier Sentiment analysis using naive bayes classifier
Sentiment analysis using naive bayes classifier
 
Heuristic Search Techniques Unit -II.ppt
Heuristic Search Techniques Unit -II.pptHeuristic Search Techniques Unit -II.ppt
Heuristic Search Techniques Unit -II.ppt
 
how to calclute time complexity of algortihm
how to calclute time complexity of algortihmhow to calclute time complexity of algortihm
how to calclute time complexity of algortihm
 
Segment tree
Segment treeSegment tree
Segment tree
 
Clustering in artificial intelligence
Clustering in artificial intelligence Clustering in artificial intelligence
Clustering in artificial intelligence
 
8 query processing and optimization
8 query processing and optimization8 query processing and optimization
8 query processing and optimization
 
Data Mining: Association Rules Basics
Data Mining: Association Rules BasicsData Mining: Association Rules Basics
Data Mining: Association Rules Basics
 
Asymptotic notations
Asymptotic notationsAsymptotic notations
Asymptotic notations
 
Data mining in social network
Data mining in social networkData mining in social network
Data mining in social network
 
Boolean,vector space retrieval Models
Boolean,vector space retrieval Models Boolean,vector space retrieval Models
Boolean,vector space retrieval Models
 
Infix to Prefix (Conversion, Evaluation, Code)
Infix to Prefix (Conversion, Evaluation, Code)Infix to Prefix (Conversion, Evaluation, Code)
Infix to Prefix (Conversion, Evaluation, Code)
 
3.1 clustering
3.1 clustering3.1 clustering
3.1 clustering
 
Sentiment analysis
Sentiment analysisSentiment analysis
Sentiment analysis
 
Digital Search Tree
Digital Search TreeDigital Search Tree
Digital Search Tree
 
ANN(Artificial Neural Networks) Clustering Algorithms
ANN(Artificial  Neural Networks)  Clustering AlgorithmsANN(Artificial  Neural Networks)  Clustering Algorithms
ANN(Artificial Neural Networks) Clustering Algorithms
 
CS6007 information retrieval - 5 units notes
CS6007   information retrieval - 5 units notesCS6007   information retrieval - 5 units notes
CS6007 information retrieval - 5 units notes
 

Viewers also liked

Sentiment Analysis via R Programming
Sentiment Analysis via R ProgrammingSentiment Analysis via R Programming
Sentiment Analysis via R ProgrammingSkillspeed
 
Sentiment Analysis in R
Sentiment Analysis in RSentiment Analysis in R
Sentiment Analysis in REdureka!
 
How to Win Machine Learning Competitions ?
How to Win Machine Learning Competitions ? How to Win Machine Learning Competitions ?
How to Win Machine Learning Competitions ? HackerEarth
 
Tweets Classification using Naive Bayes and SVM
Tweets Classification using Naive Bayes and SVMTweets Classification using Naive Bayes and SVM
Tweets Classification using Naive Bayes and SVMTrilok Sharma
 
Twitter sentiment-analysis Jiit2013-14
Twitter sentiment-analysis Jiit2013-14Twitter sentiment-analysis Jiit2013-14
Twitter sentiment-analysis Jiit2013-14Rachit Goel
 
Sentiment analysis of tweets
Sentiment analysis of tweetsSentiment analysis of tweets
Sentiment analysis of tweetsVasu Jain
 
Introduction to Sentiment Analysis
Introduction to Sentiment AnalysisIntroduction to Sentiment Analysis
Introduction to Sentiment AnalysisJaganadh Gopinadhan
 
How Sentiment Analysis works
How Sentiment Analysis worksHow Sentiment Analysis works
How Sentiment Analysis worksCJ Jenkins
 
Sentiment Analysis in Twitter
Sentiment Analysis in TwitterSentiment Analysis in Twitter
Sentiment Analysis in TwitterAyushi Dalmia
 
Tutorial of Sentiment Analysis
Tutorial of Sentiment AnalysisTutorial of Sentiment Analysis
Tutorial of Sentiment AnalysisFabio Benedetti
 
Sentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSumit Raj
 
The Who What Where When And Why Of Social Media Lead Generation
The Who What Where When And Why Of Social Media Lead GenerationThe Who What Where When And Why Of Social Media Lead Generation
The Who What Where When And Why Of Social Media Lead GenerationAbhishek Shah
 
Social Media Secrets
Social Media SecretsSocial Media Secrets
Social Media SecretsGuy Kawasaki
 
Zenpayroll Pitch Deck Template
Zenpayroll Pitch Deck TemplateZenpayroll Pitch Deck Template
Zenpayroll Pitch Deck TemplateJoseph Hsieh
 
AdPushup Fundraising Deck - First Pitch
AdPushup Fundraising Deck - First PitchAdPushup Fundraising Deck - First Pitch
AdPushup Fundraising Deck - First Pitchadpushup
 
AppVirality.com - Investor Pitch Deck
AppVirality.com - Investor Pitch DeckAppVirality.com - Investor Pitch Deck
AppVirality.com - Investor Pitch DeckLaxman Papineni
 
How Wealthsimple raised $2M in 2 weeks
How Wealthsimple raised $2M in 2 weeksHow Wealthsimple raised $2M in 2 weeks
How Wealthsimple raised $2M in 2 weeksWealthsimple
 
500’s Demo Day Batch 16 >> Podozi
500’s Demo Day Batch 16 >>  Podozi500’s Demo Day Batch 16 >>  Podozi
500’s Demo Day Batch 16 >> Podozi500 Startups
 
Swipes pitch deck for Beta Pitch 2013 Finals in Berlin
Swipes pitch deck for Beta Pitch 2013 Finals in BerlinSwipes pitch deck for Beta Pitch 2013 Finals in Berlin
Swipes pitch deck for Beta Pitch 2013 Finals in BerlinSwipes App
 

Viewers also liked (20)

Sentiment Analysis via R Programming
Sentiment Analysis via R ProgrammingSentiment Analysis via R Programming
Sentiment Analysis via R Programming
 
Sentiment Analysis in R
Sentiment Analysis in RSentiment Analysis in R
Sentiment Analysis in R
 
How to Win Machine Learning Competitions ?
How to Win Machine Learning Competitions ? How to Win Machine Learning Competitions ?
How to Win Machine Learning Competitions ?
 
Tweets Classification using Naive Bayes and SVM
Tweets Classification using Naive Bayes and SVMTweets Classification using Naive Bayes and SVM
Tweets Classification using Naive Bayes and SVM
 
Twitter sentiment-analysis Jiit2013-14
Twitter sentiment-analysis Jiit2013-14Twitter sentiment-analysis Jiit2013-14
Twitter sentiment-analysis Jiit2013-14
 
Sentiment analysis of tweets
Sentiment analysis of tweetsSentiment analysis of tweets
Sentiment analysis of tweets
 
Introduction to Sentiment Analysis
Introduction to Sentiment AnalysisIntroduction to Sentiment Analysis
Introduction to Sentiment Analysis
 
How Sentiment Analysis works
How Sentiment Analysis worksHow Sentiment Analysis works
How Sentiment Analysis works
 
Sentiment Analysis in Twitter
Sentiment Analysis in TwitterSentiment Analysis in Twitter
Sentiment Analysis in Twitter
 
Tutorial of Sentiment Analysis
Tutorial of Sentiment AnalysisTutorial of Sentiment Analysis
Tutorial of Sentiment Analysis
 
Sentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSentiment Analysis of Twitter Data
Sentiment Analysis of Twitter Data
 
The Who What Where When And Why Of Social Media Lead Generation
The Who What Where When And Why Of Social Media Lead GenerationThe Who What Where When And Why Of Social Media Lead Generation
The Who What Where When And Why Of Social Media Lead Generation
 
Social Media Secrets
Social Media SecretsSocial Media Secrets
Social Media Secrets
 
SteadyBudget's Seed Funding Pitch Deck
SteadyBudget's Seed Funding Pitch DeckSteadyBudget's Seed Funding Pitch Deck
SteadyBudget's Seed Funding Pitch Deck
 
Zenpayroll Pitch Deck Template
Zenpayroll Pitch Deck TemplateZenpayroll Pitch Deck Template
Zenpayroll Pitch Deck Template
 
AdPushup Fundraising Deck - First Pitch
AdPushup Fundraising Deck - First PitchAdPushup Fundraising Deck - First Pitch
AdPushup Fundraising Deck - First Pitch
 
AppVirality.com - Investor Pitch Deck
AppVirality.com - Investor Pitch DeckAppVirality.com - Investor Pitch Deck
AppVirality.com - Investor Pitch Deck
 
How Wealthsimple raised $2M in 2 weeks
How Wealthsimple raised $2M in 2 weeksHow Wealthsimple raised $2M in 2 weeks
How Wealthsimple raised $2M in 2 weeks
 
500’s Demo Day Batch 16 >> Podozi
500’s Demo Day Batch 16 >>  Podozi500’s Demo Day Batch 16 >>  Podozi
500’s Demo Day Batch 16 >> Podozi
 
Swipes pitch deck for Beta Pitch 2013 Finals in Berlin
Swipes pitch deck for Beta Pitch 2013 Finals in BerlinSwipes pitch deck for Beta Pitch 2013 Finals in Berlin
Swipes pitch deck for Beta Pitch 2013 Finals in Berlin
 

Similar to R by example: mining Twitter for consumer attitudes towards airlines

Sentiment analysis on airlines
Sentiment analysis on airlinesSentiment analysis on airlines
Sentiment analysis on airlinesPiyush Srivastava
 
Final Presentation
Final PresentationFinal Presentation
Final PresentationLove Tyagi
 
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...Amazon Web Services
 
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...PAPIs.io
 
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docxDashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docxsimonithomas47935
 
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...AWS Germany
 
Exploring Equilibrium Login to the computer and open a web browser.docx
Exploring Equilibrium Login to the computer and open a web browser.docxExploring Equilibrium Login to the computer and open a web browser.docx
Exploring Equilibrium Login to the computer and open a web browser.docxssuser454af01
 
(BDT302) Real-World Smart Applications With Amazon Machine Learning
(BDT302) Real-World Smart Applications With Amazon Machine Learning(BDT302) Real-World Smart Applications With Amazon Machine Learning
(BDT302) Real-World Smart Applications With Amazon Machine LearningAmazon Web Services
 
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter AnalysisIBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter AnalysisTorsten Steinbach
 
Building an Autonomous Data Layer
Building an Autonomous Data LayerBuilding an Autonomous Data Layer
Building an Autonomous Data LayerMartyPitt1
 
Flutter Festivals IIT Goa Session 2
Flutter Festivals IIT Goa Session 2Flutter Festivals IIT Goa Session 2
Flutter Festivals IIT Goa Session 2SEJALGUPTA44
 
"R & Text Analytics" (15 January 2013)
"R & Text Analytics" (15 January 2013)"R & Text Analytics" (15 January 2013)
"R & Text Analytics" (15 January 2013)Portland R User Group
 
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI servicesAWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI servicesAmazon Web Services
 
Command Query Responsibility Segregation at Enterprise Scale
Command Query Responsibility Segregation at Enterprise ScaleCommand Query Responsibility Segregation at Enterprise Scale
Command Query Responsibility Segregation at Enterprise ScaleTechWell
 
Databaseconcepts
DatabaseconceptsDatabaseconcepts
Databaseconceptsdilipkkr
 
Demystifying Machine Learning on AWS
Demystifying Machine Learning on AWSDemystifying Machine Learning on AWS
Demystifying Machine Learning on AWSAmazon Web Services
 

Similar to R by example: mining Twitter for consumer attitudes towards airlines (20)

Sentiment analysis on airlines
Sentiment analysis on airlinesSentiment analysis on airlines
Sentiment analysis on airlines
 
Final Presentation
Final PresentationFinal Presentation
Final Presentation
 
The Value of Twitter
The Value of TwitterThe Value of Twitter
The Value of Twitter
 
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
AWS January 2016 Webinar Series - Building Smart Applications with Amazon Mac...
 
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
 
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docxDashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
DashboardUMUC DATA 620Assignment 3.1Your NameProfessorDateProbl.docx
 
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
Real-World Smart Applications with Amazon Machine Learning - AWS Machine Lear...
 
Exploring Equilibrium Login to the computer and open a web browser.docx
Exploring Equilibrium Login to the computer and open a web browser.docxExploring Equilibrium Login to the computer and open a web browser.docx
Exploring Equilibrium Login to the computer and open a web browser.docx
 
(BDT302) Real-World Smart Applications With Amazon Machine Learning
(BDT302) Real-World Smart Applications With Amazon Machine Learning(BDT302) Real-World Smart Applications With Amazon Machine Learning
(BDT302) Real-World Smart Applications With Amazon Machine Learning
 
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter AnalysisIBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
 
Building an Autonomous Data Layer
Building an Autonomous Data LayerBuilding an Autonomous Data Layer
Building an Autonomous Data Layer
 
Flutter Festivals IIT Goa Session 2
Flutter Festivals IIT Goa Session 2Flutter Festivals IIT Goa Session 2
Flutter Festivals IIT Goa Session 2
 
REST APIs, Girls Who Code
REST APIs, Girls Who CodeREST APIs, Girls Who Code
REST APIs, Girls Who Code
 
"R & Text Analytics" (15 January 2013)
"R & Text Analytics" (15 January 2013)"R & Text Analytics" (15 January 2013)
"R & Text Analytics" (15 January 2013)
 
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI servicesAWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
AWS Summit Singapore 2019 | Accelerating ML Adoption with Our New AI services
 
Command Query Responsibility Segregation at Enterprise Scale
Command Query Responsibility Segregation at Enterprise ScaleCommand Query Responsibility Segregation at Enterprise Scale
Command Query Responsibility Segregation at Enterprise Scale
 
Databaseconcepts
DatabaseconceptsDatabaseconcepts
Databaseconcepts
 
Blind sql injection
Blind sql injectionBlind sql injection
Blind sql injection
 
Blind sql injection
Blind sql injectionBlind sql injection
Blind sql injection
 
Demystifying Machine Learning on AWS
Demystifying Machine Learning on AWSDemystifying Machine Learning on AWS
Demystifying Machine Learning on AWS
 

More from Jeffrey Breen

Tapping the Data Deluge with R
Tapping the Data Deluge with RTapping the Data Deluge with R
Tapping the Data Deluge with RJeffrey Breen
 
Getting started with R & Hadoop
Getting started with R & HadoopGetting started with R & Hadoop
Getting started with R & HadoopJeffrey Breen
 
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)Jeffrey Breen
 
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....Jeffrey Breen
 
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2Jeffrey Breen
 
Big Data Step-by-Step: Infrastructure 1/3: Local VM
Big Data Step-by-Step: Infrastructure 1/3: Local VMBig Data Step-by-Step: Infrastructure 1/3: Local VM
Big Data Step-by-Step: Infrastructure 1/3: Local VMJeffrey Breen
 
Move your data (Hans Rosling style) with googleVis + 1 line of R code
Move your data (Hans Rosling style) with googleVis + 1 line of R codeMove your data (Hans Rosling style) with googleVis + 1 line of R code
Move your data (Hans Rosling style) with googleVis + 1 line of R codeJeffrey Breen
 
Accessing Databases from R
Accessing Databases from RAccessing Databases from R
Accessing Databases from RJeffrey Breen
 
Grouping & Summarizing Data in R
Grouping & Summarizing Data in RGrouping & Summarizing Data in R
Grouping & Summarizing Data in RJeffrey Breen
 
R + 15 minutes = Hadoop cluster
R + 15 minutes = Hadoop clusterR + 15 minutes = Hadoop cluster
R + 15 minutes = Hadoop clusterJeffrey Breen
 
FAA Aviation Forecasts 2011-2031 overview
FAA Aviation Forecasts 2011-2031 overviewFAA Aviation Forecasts 2011-2031 overview
FAA Aviation Forecasts 2011-2031 overviewJeffrey Breen
 

More from Jeffrey Breen (12)

Tapping the Data Deluge with R
Tapping the Data Deluge with RTapping the Data Deluge with R
Tapping the Data Deluge with R
 
Getting started with R & Hadoop
Getting started with R & HadoopGetting started with R & Hadoop
Getting started with R & Hadoop
 
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
 
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
 
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
Big Data Step-by-Step: Infrastructure 2/3: Running R and RStudio on EC2
 
Big Data Step-by-Step: Infrastructure 1/3: Local VM
Big Data Step-by-Step: Infrastructure 1/3: Local VMBig Data Step-by-Step: Infrastructure 1/3: Local VM
Big Data Step-by-Step: Infrastructure 1/3: Local VM
 
Move your data (Hans Rosling style) with googleVis + 1 line of R code
Move your data (Hans Rosling style) with googleVis + 1 line of R codeMove your data (Hans Rosling style) with googleVis + 1 line of R code
Move your data (Hans Rosling style) with googleVis + 1 line of R code
 
Accessing Databases from R
Accessing Databases from RAccessing Databases from R
Accessing Databases from R
 
Reshaping Data in R
Reshaping Data in RReshaping Data in R
Reshaping Data in R
 
Grouping & Summarizing Data in R
Grouping & Summarizing Data in RGrouping & Summarizing Data in R
Grouping & Summarizing Data in R
 
R + 15 minutes = Hadoop cluster
R + 15 minutes = Hadoop clusterR + 15 minutes = Hadoop cluster
R + 15 minutes = Hadoop cluster
 
FAA Aviation Forecasts 2011-2031 overview
FAA Aviation Forecasts 2011-2031 overviewFAA Aviation Forecasts 2011-2031 overview
FAA Aviation Forecasts 2011-2031 overview
 

Recently uploaded

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 

Recently uploaded (20)

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 

R by example: mining Twitter for consumer attitudes towards airlines

  • 1. R by example: mining Twitter for consumer attitudes towards airlines presented at the Boston Predictive Analytics MeetUp by Jeffrey Breen President Cambridge Aviation Research jbreen@cambridge.aero June 2011 Cambridge Aviation Research • 245 First Street • Suite 1800 • Cambridge, MA 02142 • cambridge.aero © Copyright 2010 by Cambridge Aviation Research. All rights reserved.
  • 2. Airlines top customer satisfaction... alphabetically http://www.theacsi.org/ 3
  • 3. Actually, they rank below the Post Office and health insurers 4
  • 4. which gives us plenty to listen to Completely unimpressed with @continental or @united. RT @dave_mcgregor: Poor communication, goofy reservations systems and Publicly pledging to all to turn my trip into a mess. never fly @delta again. The worst airline ever. U have lost my patronage @united #fail on wifi in red carpet clubs (too forever due to ur slow), delayed flight, customer service in red incompetence carpet club (too slow), hmmm do u see a trend? @United Weather delays may not be your fault, but you are in the customer service business. It's atrocious how people are getting treated! We were just told we are delayed 1.5 @SouthwestAir I know you don't make the hrs & next announcement on @JetBlue - weather. But at least pretend I am not a “We're selling headsets.” Way to bother when I ask if the delay will make capitalize on our misfortune. miss my connection @SouthwestAir I hate you with every Hey @delta - you suck! Your prices single bone in my body are over the moon & to move a flight for delaying my flight by a cpl of days is $150.00. Insane. I 3 hours, 30mins before I hate you! U ruined my vacation! was supposed to board. #hate
  • 5. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 14
  • 6. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 15
  • 7. Searching Twitter in one line R’s XML and RCurl packages make it easy to grab web data, but Jeff Gentry’s twitteR package makes searching Twitter almost too easy: > # load the package > library(twitteR) > # get the 1,500 most recent tweets mentioning ‘@delta’: > delta.tweets = searchTwitter('@delta', n=1500) See what we got in return: A “list” in R is a collection of objects and its elements may be > length(delta.tweets) named or just numbered. [1] 1500 > class(delta.tweets) [1] "list" “[[ ]]” is used to access elements.
  • 8. Examine the output Let’s take a look at the first tweet in the output list: > tweet = delta.tweets[[1]] tweet is an object of type “status” from the “twitteR” package. > class(tweet) [1] "status" attr(,"package") It holds all the information about [1] "twitteR" the tweet returned from Twitter. The help page (“?status”) describes some accessor methods like getScreenName() and getText() which do what you would expect: > tweet$getScreenName() [1] "Alaqawari" > tweet$getText() [1] "I am ready to head home. Inshallah will try to get on the earlier flight to Fresno. @Delta @DeltaAssist"
  • 9. Extract the tweet text R has several (read: too many) ways to apply functions iteratively. •The plyr package unifies them all with a consistent naming convention. •The function name is determined by the input and output data types. We have a list and would like a simple array output, so we use “laply”: > delta.text = laply(delta.tweets, function(t) t$getText() ) > length(delta.text)[1] 1500 > head(delta.text, 5) [1] "I am ready to head home. Inshallah will try to get on the earlier flight to Fresno. @Delta @DeltaAssist" [2] "@Delta Releases 2010 Corporate Responsibility Report - @PRNewswire (press release) : http://tinyurl.com/64mz3oh" [3] "Another week, another upgrade! Thanks @Delta!" [4] "I'm not able to check in or select a seat for flight DL223/KL6023 to Seattle tomorrow. Help? @KLM @delta" [5] "In my boredom of waiting realized @deltaairlines is now @delta seriously..... Stil waiting and your not even unloading status yet"
  • 10. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 19
  • 11. Estimating Sentiment There are many good papers and resources describing methods to estimate sentiment. These are very complex algorithms. For this tutorial, we use a very simple algorithm which assigns a score by simply counting the number of occurrences of “positive” and “negative” words in a tweet. The code for our score.sentiment() function can be found at the end of this deck. Hu & Liu have published an “opinion lexicon” which categorizes approximately 6,800 words as positive or negative and which can be downloaded. Positive: love, best, cool, great, good, amazing Negative: hate, worst, sucks, awful, nightmare 20
  • 12. Load sentiment word lists 1. Download Hu & Liu’s opinion lexicon: http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html 2. Loading data is one of R’s strengths. These are simple text files, though they use “;” as a comment character at the beginning: > hu.liu.pos = scan('../data/opinion-lexicon-English/positive- words.txt', what='character', comment.char=';') > hu.liu.neg = scan('../data/opinion-lexicon-English/negative- words.txt', what='character', comment.char=';') 3. Add a few industry-specific and/or especially emphatic terms: > pos.words = c(hu.liu.pos, 'upgrade') The c() function > neg.words = c(hu.liu.neg, 'wtf', 'wait', combines objects 'waiting', 'epicfail', 'mechanical') into vectors or lists
  • 13. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 22
  • 14. Algorithm sanity check > sample = c("You're awesome and I love you", "I hate and hate and hate. So angry. Die!", "Impressed and amazed: you are peerless in your achievement of unparalleled mediocrity.") > result = score.sentiment(sample, pos.words, neg.words) > class(result) data.frames hold tabular data so they [1] "data.frame" consist of columns & rows which can > result$score be accessed by name or number. [1] 2 -5 4 Here, “score” is the name of a column. So, not so good with sarcasm. Here are a couple of real tweets: > score.sentiment(c("@Delta I'm going to need you to get it together. Delay on tarmac, delayed connection, crazy gate changes... #annoyed", "Surprised and happy that @Delta helped me avoid the 3.5 hr layover I was scheduled for. Patient and helpful agents. #remarkable"), pos.words, neg.words)$score [1] -4 5
  • 15. Accessing data.frames Here’s the data.frame just returned from score.sentiment(): > result score text 1 2 You're awesome and I love you 2 -5 I hate and hate and hate. So angry. Die! 3 4 Impressed and amazed: you are peerless in your achievement of unparalleled mediocrity. Elements can be accessed by name or position, and positions can be ranges: > result[1,1] [1] 2 > result[1,'score'] [1] 2 > result[1:2, 'score'] [1] 2 -5 > result[c(1,3), 'score'] [1] 2 4 > result[,'score'] [1] 2 -5 4
  • 16. Score the tweets To score all of the Delta tweets, just feed their text into score.sentiment(): > delta.scores = score.sentiment(delta.text, pos.words, Progress bar neg.words, .progress='text') provided by |==================================================| 100% plyr Let’s add two new columns to identify the airline for when we combine all the scores later: > delta.scores$airline = 'Delta' > delta.scores$code = 'DL’
  • 17. Plot Delta’s score distribution R’s built-in hist() function will create and plot histograms of your data: > hist(delta.scores$score)
  • 18. The ggplot2 alternative ggplot2 is an alternative graphics package which generates more refined graphics: > qplot(delta.scores$score)
  • 19. Lather. Rinse. Repeat To see how the other airlines fare, collect & score tweets for other airlines. Then combine all the results into a single “all.scores” data.frame: > all.scores = rbind( american.scores, continental.scores, delta.scores, jetblue.scores, southwest.scores, united.scores, us.scores ) rbind() combines rows from data.frames, arrays, and matrices
  • 20. Compare score distributions ggplot2 implements “grammar of graphics”, building plots in layers: > ggplot(data=all.scores) + # ggplot works on data.frames, always geom_bar(mapping=aes(x=score, fill=airline), binwidth=1) + facet_grid(airline~.) + # make a separate plot for each airline theme_bw() + scale_fill_brewer() # plain display, nicer colors ggplot2’s faceting capability makes it easy to generate the same graph for different values of a variable, in this case “airline”.
  • 21. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 30
  • 22. Ignore the middle Let’s focus on very negative (<-2) and positive (>2) tweets: > all.scores$very.pos = as.numeric( all.scores$score >= 2 ) > all.scores$very.neg = as.numeric( all.scores$score <= -2 ) For each airline ( airline + code ), let’s use the ratio of very positive to very negative tweets as the overall sentiment score for each airline: > twitter.df = ddply(all.scores, c('airline', 'code'), summarise, pos.count = sum( very.pos ), neg.count = sum( very.neg ) ) > twitter.df$all.count = twitter.df$pos.count + twitter.df$neg.count > twitter.df$score = round( 100 * twitter.df$pos.count / twitter.df$all.count ) Sort with orderBy() from the doBy package: > orderBy(~-score, twitter.df)
  • 23. Any relation to ACSI’s airline scores? http://www.theacsi.org/index.php?option=com_content&view=article&id=147&catid=&Itemid=212&i=Airlines 18
  • 24. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 33
  • 25. Scrape, don’t type XML package provides amazing readHTMLtable() function: > library(XML) > acsi.url = 'http://www.theacsi.org/index.php? option=com_content&view=article&id=147&catid=&Itemid=212&i=Airlines' > acsi.df = readHTMLTable(acsi.url, header=T, which=1, stringsAsFactors=F) > # only keep column #1 (name) and #18 (2010 score) > acsi.df = acsi.df[,c(1,18)] > head(acsi.df,1) 10 1 Southwest Airlines 79 Well, typing metadata is OK, I guess... clean up column names, etc: > colnames(acsi.df) = c('airline', 'score') NA (as in “n/a”) is > acsi.df$code = c('WN', NA, 'CO', NA, 'AA', 'DL', supported as a 'US', 'NW', 'UA') valid value > acsi.df$score = as.numeric(acsi.df$score) everywhere in R.
  • 26. Game Plan Search Twitter for airline mentions & collect tweet text Score sentiment for Summarize for each each tweet airline Load sentiment word lists Compare Twitter sentiment with ACSI satisfaction score Scrape ACSI web site for airline customer satisfaction scores 35
  • 27. Join and compare merge() joins two data.frames by the specified “by=” fields. You can specify ‘suffixes’ to rename conflicting column names: > compare.df = merge(twitter.df, acsi.df, by='code', suffixes=c('.twitter', '.acsi')) Unless you specify “all=T”, non-matching rows are dropped (like a SQL INNER JOIN), and that’s what happened to top scoring JetBlue. With a very low score, and low traffic to boot, soon-to-disappear Continental looks like an outlier. Let’s exclude: > compare.df = subset(compare.df, all.count > 100)
  • 28. an actual result! ggplot will even run lm() linear (and other) regressions for you with its geom_smooth() layer: > ggplot( compare.df ) + geom_point(aes(x=score.twitter, y=score.acsi, color=airline.twitter), size=5) + geom_smooth(aes(x=score.twitter, y=score.acsi, group=1), se=F, method="lm") + theme_bw() + opts(legend.position=c(0.2, 0.85)) 37 21
  • 30. R code for example scoring function score.sentiment = function(sentences, pos.words, neg.words, .progress='none') { require(plyr) require(stringr) # we got a vector of sentences. plyr will handle a list or a vector as an "l" for us # we want a simple array of scores back, so we use "l" + "a" + "ply" = laply: scores = laply(sentences, function(sentence, pos.words, neg.words) { # clean up sentences with R's regex-driven global substitute, gsub(): sentence = gsub('[[:punct:]]', '', sentence) sentence = gsub('[[:cntrl:]]', '', sentence) sentence = gsub('d+', '', sentence) # and convert to lower case: sentence = tolower(sentence) # split into words. str_split is in the stringr package word.list = str_split(sentence, 's+') # sometimes a list() is one level of hierarchy too much words = unlist(word.list) # compare our words to the dictionaries of positive & negative terms pos.matches = match(words, pos.words) neg.matches = match(words, neg.words) # match() returns the position of the matched term or NA # we just want a TRUE/FALSE: pos.matches = !is.na(pos.matches) neg.matches = !is.na(neg.matches) # and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum(): score = sum(pos.matches) - sum(neg.matches) return(score) }, pos.words, neg.words, .progress=.progress ) scores.df = data.frame(score=scores, text=sentences) return(scores.df) } 39

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. delta.tweets[1] \n\n&quot;0533370677: RT @g: To the @Delta attendant at gate D39 in ATL: your attitude isn&apos;t the best, but you did get me on the earlier flight. Thank you. #ncslatl&quot;\n\n&amp;#x2018;original tweet has two parts: screenname and the text; we need to separate out the text&amp;#x201D;\n
  8. &amp;#x2018;This is one tweet, next will batch process the set&amp;#x2019;\n
  9. &amp;#x2018;can see here the text has been extracted&amp;#x2019;\n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n