Review Analysis: an Approach to Leveraging User-Generated Content in the Context of Retail: What are customers really thinking? What are they looking for specifically when shopping for a product? And if they are satisfied with their purchase, what is the main reason? Today’s technology offers many different avenues for customers to express themselves, set their expectations in writing, and share their opinion, frustration or satisfaction regarding all kinds of products and services. ‘Leaving a review’ has become an integral part of the purchase process. Through reviews, customers are volunteering invaluable information that can be turned into insights that would help drive business decisions (if you are a retailer), or help you make a successful purchase (if you are a customer). Yet the amount of data available to make these decisions is oftentimes extremely large, and it might be difficult for a human to read and synthesize all that has been said about their product of interest. Review analysis and opinion mining offer solutions to automate the analysis of customer feedback through large-scale machine learning, natural language processing and sentiment analysis, and allow retailers to better understand their customers… as well as their data. In this talk, I will present the various ways in which machine learning techniques can be used to extract the most significant features for a given category of products. I will then dig into a process aiming at identifying the sentiments relative to these features, and a useful way to aggregate this information into insights that are both usable and readable by any user. I will end with mentions to some of the challenges met when trying to extract objective information from a data source likely tainted with human subjectivity in an ever-changing market.

1. Review Analysis:
An Approach to Leveraging User-Generated Content in
the Context of Retail
Jennifer Prendki, Principal Data Scientist
Walmart Global e-Commerce
California, USA
The Machine Learning Conference, San Francisco, CA
11/11/2016

2. Outline
• Business motivation
• Algorithm Pipeline
• Feature Space Computation
• Sentiment Capture
• Real-Life Examples and Results
• Future Work and Conclusions
2
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

3. Business Motivation
3
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

4. Business Motivation
4
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

5. Business Motivation
5
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

6. Business Motivation
6
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

7. Business Motivation
“ I bought this for my daughter to
do her college work on, it's been
great, no problems so far. “
[SuperMom72]
“ Works like a charm, would
definitely recommend to anyone
on a budget. “
[Vamsy]
“ Fast CPU but slow disk drive
slows everything down. ”
[TalonBay]
“ I don't do gaming or downloading movies or music, so for those
folks I can't speak to the performance. But for surfing the web,
checking email, etc., this computer will save you time for watching
the little ball spin!”
[Anonymous]
7
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

8. Review Analysis: A Current Landscape
• Sentiment analysis
• Best known use case: Social Media Analysis/Tweets
Why tweets?  shorter, condensed, highly sentimental content
• Movie review analysis:
Kaggle: Analysis of the ‘Rotten Tomatoes’ Dataset
• Regarding product review analysis
• Little to no papers regarding product review analysis at commercial scale
• Shortage of work regarding combination of topic modeling and sentiment
analysis
8
Our research: Combine feature computation and sentiment analysis to
summarize reviewers’ opinions about a specific product
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

9. Algorithm Pipeline
9
Product 𝛼
Product 𝛽
Product 𝛾
Review
Review
Review
Review
Review
Review
Fc
Category C
Feature
Space
Computation
F 𝛼
F 𝛽
F 𝛾
Feature
Space
Reduction
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

10. Algorithm Pipeline
10
Product 𝛼
Product 𝛽
Product 𝛾
Review
Review
Review
Review
Review
Review
Category C
F 𝛼
F 𝛽
F 𝛾
Sentiment
Sentiment
Sentiment
Sentiment
Sentiment
Sentiment
Sentiment
Computation
For Each Review
… Sentiment
Computed For
Relevant Features
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

11. Algorithm Pipeline
11
Product 𝛼
Product 𝛽
Product 𝛾
Review
Review
Review
Review
Review
Review
Category C
Sentiment
Sentiment
Sentiment
Sentiment
Sentiment
Sentiment
𝜎𝑡, 𝛼, 𝑓
𝜎𝑡, 𝛽, 𝑓
𝜎𝑡, 𝛾, 𝑓
∀ 𝑡 ∈ 𝜏
∀ 𝑓 ∈ F 𝛼
∀ 𝑡 ∈ 𝜏
∀ 𝑓 ∈ Fβ
∀ 𝑡 ∈ 𝜏
∀ 𝑓 ∈ F 𝛾
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions
Sentiment
Computation
For Each Review
… Sentiment
Computed For
Relevant Features

12. Feature Space Computation
• Textual reviews go through a careful process:
• TF/TF-Idf transform on documents
• Stop words removal, stemming, part-of-speech selection
• Spell-checking
• etc.
• ‘Synonym’ computation
• Can be done using Word Embedding (glove, word2vec)
• Can be done building synonym graph using dictionary/Wikipedia
• Is complex and tricky, context-sensitive, unsupervised
12
In short: Creating synonym sets is difficult, and challenging as an online algorithm
In short: Preprocessing crucial to extracting relevant features
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

13. [0-1]
Intensity of negativity
in sentence
{'neg': 0.0, 'neu': 0.58, 'pos': 0.42, 'compound': 0.4404}
[0-1]
Intensity
of neutrality
in sentence
[0-1]
Intensity
of positivity
in sentence
[-1,1]
Combination of positive and
negative sentiments.
Allows positive and negative to
‘compensate’ one another
Sentiment Capture with Vader
VADER: Valence Aware Dictionary and sEntiment
Reasoner
• Is a Python sub-module found of the nltk module
• Is a lexicon and rule-based sentiment analysis tool
• Is specifically attuned to sentiments expressed in social media
• Is fully open-sourced, developed and licensed by MIT
13
Sentiment is not boolean
posneg neu
Sentiment as
a PDF
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

14. Sentiment Capture with Vader
14
”This computer is good deal.” “This computer is a bad deal.”
pos 0.42 0.0
neu 0.58 0.533
neg 0.0 0.467
compound 0.4404 -0.5423
”This computer is
not powerful.”
“This computer is
not that powerful.”
“This computer is not
powerful, but I like it
anyways.”
“This computer is not
that powerful, but I
like it anyways.”
pos 0.0 0.0 0.0 0.252
neu 0.632 0.682 0.618 0.619
neg 0.368 0.318 0.382 0.129
compound -0.3252 -0.3252 -0.5157 0.3786
 Vader is sensitive to adverbs, punctuation, case, emoticons and nuances…
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

15. Sentiment Capture with Vader
15
Product A
“The design and picture quality are amazing!”
I love it! Just perfect for people on a budget. And it is
beautifully designed!!
“Pretty good, but I am not a fan of the design.”
“I don’t think it’s possible to find better for the price”
design
Product B
“I just HATE the design!!”
“Okay computer. Wish I read the other reviews first.”
design
picture
quality
picture
quality
+ 0.39 + 0.39
+ 0.56
~ 0.48
- 0.62
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

16. Sentiment Capture with Vader
16
Product A
design
Product B
design
picture
quality
picture
quality
design
picture quality
battery
value
design
battery
CPU
processor
0.48
0.39
0.46 0.62
0.49 NA
NA NA
0.75
0.87NANA
0.43 NA NA
3
1
4
2
1
0.39
0.60
NA
NA
1
1
1
NA
NA
NA
NA
0.62
✍
Scraping Summarizing Sentiment Intensity
+ 0.39
+ 0.56
~ 0.49
- 0.62
+ 0.39 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

17. Results Discussion: Real Life Example
17
Product BProduct A
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

18. Results Discussion: Real Life Example
18
screen
design
quality
performance
NEG NEU POSscreen
design
performance
quality
screen
design
quality
performance
Some dissatisfaction with overall quality
Reviewers are rather
happy with keyboard
Weight is better for product
A than for product B
Customers satisfied with keyboard,
display, screen, design, …Product B’s weakness is battery life The product’s features
are well documented
Product A
Product B
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

19. Results Discussion: Real Life Example
19
This laptop exceeds my expectations. It's fast,
it's powerful, it's compact and great to travel
with.
“The screen is amazing and the keyboard
too. the weight is so light, it's become my
portable.”
It’s durable, good keyboard, decent screen,
and a good battery life.
Plasticky build quality but holds up with my
rough and tough handling. Is is surprisingly
light. Keyboard is the best but it tales a bit of
getting used to[…]
Very light to carry and the carbon color gives
an elegant finishing touch. Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions
Product A
11 reviews, ~ 53 words per review

20. Results Discussion: Real Life Example
The design is what caught my eye. Everything
about this laptop is okay, except the battery
life.
Overall a great laptop with good display and
build quality, solid performance and sleek
design the only major concern is battery life.
The touch screen is absolutely first rate […],
and the back-lit keyboard has just the right
feel.
This is the best computer I've ever owned.
[…]. I love the backlit keyboard, the easily
adjustable resolution and the long battery life.
Pros: great screen, keyboard feels nice, best
touchpad, very fast, extremely light, built
durable
Cons: battery life is less than competitors
[…].
It's really light weight yet really durable. I love
the keyboard and mouse pad.
28
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions
Product B
40 reviews, ~ 83 words per review

21. Conclusion and Future Work
Work in Progress
• Synonym computation: work in progress
• Observed bias in sentiment, needs particular attention
• Alternative when no/little reviews exist?
Potential future applications
• Offer a snapshot of product reviews to customers
• Assist customers in finding similar items with enhanced feature(s)
• Process seller satisfaction information/rating
• Customer email processing, determine subject of request automatically
21
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

22. References
[1] Gensim
https://radimrehurek.com/gensim/models/word2vec.html
[2] GloVe
http://nlp.stanford.edu/projects/glove/
[3] Wordnet
https://wordnet.princeton.edu/
[4] nltk.stem
http://www.nltk.org/api/nltk.stem.html
[5] nltk.vader
Paper: VADER: A Parsimonious Rule-based Model for Sentiment
Analysis of Social Media Text, C.J. Hutto, Eric Gilbert
Code: http://www.nltk.org/_modules/nltk/sentiment/vader.html
22
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

23. Questions?
23

24. Back-Up Slides

25. Sentiment Capture with Vader
25
>>> sentence1 = ”This computer is a good deal."
{'neg':0.0,'neu':0.58,'pos':0.42,'compound':0.4404}
>>> sentence2 = “This computer is a very good deal.”
{'neg':0.0,'neu':0.61,'pos':0.39,'compound':0.4927}
>>> sentence3 = “This computer is a very good deal!!”
{'neg':0.0,'neu':0.57,'pos':0.43,'compound':0.5827}
>>> sentence4 = “This computer is a very good deal!! :-)”
{'neg':0.0,'neu':0.441,'pos':0.559,'compound':0.7462}
>>> sentence5 = “This computer is a VERY good deal!! :-)”
{'neg':0.0,'neu':0.393,'pos':0.607,'compound':0.8287}
>>> sentence6 = “This computer is a very bad deal!! :-(”
{'neg':0.588,'neu':0.412,'pos':0.0,'compound':-0.7987}
Adverb
addition
Punctuation
addition
Emoticon addition
Case
enhancement
Inverse polarity
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

26. Ratings vs Sentiment Analysis
26
Ratings (number of stars)
Average sentiment from
text review
negativeneutralpositive
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

27. Ratings vs Sentiment Analysis
27
Good reviews
Bad reviews
user bias =
𝑛 𝑝𝑜𝑠 − 𝑛(𝑛𝑒𝑔)
𝑛 𝑝𝑜𝑠 + 𝑛(𝑛𝑒𝑔)
where:
pos = number of prior good reviews
neg = number of prior bad reviews
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions

28. Review Bias
28
• Where is subjectivity coming from?
• Language bias / gender bias / etc.
• Vader package biases due to development specificity?
(remember: originally developed for social media)
• Incentivized customers/reviewers
• Why is it important to correct for it?
• Filtering/sorting with ratings doesn’t work as well as expected
• Possible options
• Filter reviews with large bias
• Weight results
• Re-center the output of Vader to fit our definition of ’neutrality’
In short: Biases in both ratings and textual sentiment, both need attention
 Business
Motivation
 Algorithm
 Feature Space
Computation
 Sentiment
Capture
 Real-Life
Examples
 Future Work and
Conclusions