Presentation to the Chicago chapter of the American Statistical Association, November 20, 2020, by Seth Grimes.

1. NLP 2020: What Works and What's Next
Recent Advances in Natural Language Processing
Seth Grimes
Alta Plana Corporation
@SethGrimes
November 20, 2020

4. Disclaimer
I use A LOT of commercial product materials in the
slides that follow. These are illustrations and not
recommendations.

5. Natural Language Processing
Natural Language Understanding (NLU)
• OCR, language detection, tokenization, parsing
• Information extraction: parts of speech, chunks , entities,
aspects, topics/themes, relations, attributes, events, …
• Speech processing
Natural Language Generation (NLG)
NLU + NLG together, for example:
• Summarization
• Machine translation
• Conversational interfaces
• Question answering

6. Functions
https://gradientflow.com/2020nlpsurvey/

8. Empirical Methods in Natural Language Processing (EMNLP2020)

9. Early Days (1958)
Transcribing
Encoding
Abstracting
Who needs to know?
Who knows what?
What is known?
Hans Peter Luhn
“A Business Intelligence System”
IBM Journal, October 1958

10. “Statistical information
derived from word frequency
and distribution is used by the
machine to compute a relative
measure of significance, first
for individual words and then
for sentences. Sentences scoring
highest in significance are
extracted and printed out to
become the auto-abstract.”
-- H.P. Luhn, The Automatic
Creation of Literature Abstracts,
IBM Journal, 1958.

11. “All models are wrong, but some are useful.”
-- George Box

12. +17 years
https://en.wikipedia.org/wiki/Document-term_matrix

13. Skipping A Lot of Stuff…
Rules
Taxonomies
Booleans
…

14. Word2Vec (2013)
https://code.google.com/p/word2vec/
https://developers.google.com/machine-learning/crash-course/embeddings/translating-to-a-lower-dimensional-space
“You shall know a
word by the
company it
keeps.”
– J.R. Firth, 1957

15. Word2Vec: Key Concepts
Continuous bag-of-
words (CBOW)
predicts a word from
a window of
surrounding words.
Skip-gram uses a
word to predict a
window of
surrounding words.

16. Doc2Vec (2014)
https://arxiv.org/abs/1405.4053

17. Sense2Vec (2015)
https://arxiv.org/abs/1511.06388
“Sense2vec (Trask
et. al, 2015) is a
new twist on
word2vec that lets
you learn more
interesting, detailed
and context-
sensitive word
vectors.”

19. Encoder-
Decoder
Architecture
Here, machine
translation:
https://leonoverweel.com/projects/2019/nlu-coursework/

20. Transformers (2017)
https://arxiv.org/abs/1706.03762
2020:
https://arxiv.org/pdf/1910.03771.pdf

21. BERT (2018)
https://arxiv.org/abs/1810.04805
https://arxiv.org/pdf/1910.03771.pdf

22. Transfer Learning
https://pennylane.ai/qml/demos/tutorial_quantum_transfer_learning.html

23. Transfer Learning
https://pennylane.ai/qml/demos/tutorial_quantum_transfer_learning.html

24. Pre-Trained Models
https://www.semanticscholar.org/paper/Semi-supervised-sequence-tagging-with-bidirectional-
Peters-Ammar/0bb4cadc80c0afaf29c57518dc9c06f8fcfa5f38#extracted

25. http://www.cs.cmu.edu/~yifengt/courses/machine-learning/slides/lecture9-topics-nlp-v1.0.pdf

26. https://pair-code.github.io/lit/

27. Back To The Garden

28. NLP Libraries

29. https://blog.rasa.com/rasa-nlu-in-depth-part-1-intent-classification/

32. Hugging Face
Model Hub

33. Hugging Face Pipeline Example

34. Hugging Face Pipeline Examples

35. Cloud Services

36. Amazon Comprehend Medical
https://aws.amazon.com/comprehend/medical/
“Amazon Comprehend Medical is a natural language processing service that
makes it easy to use machine learning to extract relevant medical information from
unstructured text. Using Amazon Comprehend Medical, you can quickly and
accurately gather information, such as medical condition, medication, dosage,
strength, and frequency from a variety of sources like doctors’ notes, clinical trial
reports, and patient health records. Amazon Comprehend Medical can also link the
detected information to medical ontologies such as ICD-10-CM or RxNorm so it
can be used easily by downstream healthcare applications.”

37. AWS Comprehend:
Ontology Linking
https://aws.amazon.com/blogs/aws/new-amazon-comprehend-medical-adds-ontology-linking/

38. Training

39. Services and Solutions: Examples

41. https://www.clarabridge.com/customer-experience-dictionary/text-analytics

42. https://www.qualtrics.com/experience-management/research/text-analysis/

43. Emotion AI

44. The Expressions
Charles Le Brun

46. “If we want computers to be genuinely
intelligent, to adapt to us and to interact
naturally with us, then they will need to ability
to recognize and express emotions, and to
have what has come to be called ‘emotional
intelligence.’”
-- Rosalind Picard, Affective Computing, 1997
Affective Computing

47. 1997

48. « Le cœur a ses raisons que la raison ne connaît
point. »
“The heart has its reasons that reason knows
nothing of.”
-- Blaise Pascal

49. Emotion models: Plutchik

50. What about Sentiment Analysis? Text sourced…
https://dl.acm.org/doi/10.1145/945645.945658,
October 2003
https://dl.acm.org/doi/10.1561/1500000011,
January 2008

51. Bing Liu, 2010

52. Bing Liu, 2015, 2020

53. Emotion AI
“Emotion mining is the science of detecting, analyzing, and
evaluating humans' feelings towards different events, issues,
services, or any other interest.”
Emotion synthesis enhances the ability of a machine to provide
meaningful, contextual responses, by conveying an appropriate
emotional state through words, voice, and expression.
Emotion induction aims to evoke a certain emotional response or
affective state.

54. Emotion
in Text:
Lexicons

55. Emotion in Text
https://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm

56. “Parsing Text for Emotion Terms:
Analysis & Visualization Using R”
Emotion in
Text: Parsing,
Stats
https://datascienceplus.com/parsing-text-for-emotion-terms-analysis-visualization-using-r/

57. Bing Liu, 2020

58. Emotion in Text

59. Words aren’t
everything, and
they’re not
enough:
• Idiom
• Medium/media
• Personality
• Communicative
act
• Social act
(sharing etc.)
• Linguisitics…

60. Conversation: Text and Voice

62. https://blog.rasa.com/conversational-ai-your-guide-to-five-levels-of-ai-assistants-in-enterprise/
(2018)

63. Voice Bots
Human =
Good
https://voicebot.ai/2019/11/27/alexa-is-learning-to-speak-emotionally/
https://voicebot.ai/2019/11/21/correct-call-to-action-recall-by-users-is-twice-
as-high-for-human-voices-as-synthetic-for-voice-apps/

64. Speech https://www.phon.ucl.ac.uk/courses/spsci/iss/week9.php
“Voice cues are commonly divided into those related to: (a) fundamental
frequency (F0, a correlate of the perceived pitch), (b) vocal perturbation (short-
term variability in sound production), (c) voice quality (a correlate of the
perceived ‘timbre’), (d) intensity (a correlate of the perceived loudness), and (e)
temporal aspects of speech (e.g., speech rate), as well as various combinations of
these aspects (e.g., prosodic features).”
http://www.scholarpedia.org/article/Speech_emotion_analysis

65. Voice Rendering
https://www.sciencedirect.com/science/article/abs/pii/S0167639317303187

66. Voice Rendering…via Markup
https://developer.amazon.com/en-US/blogs/alexa/alexa-skills-kit/2019/11/new-alexa-emotions-and-speaking-styles

67. Bot Emotion
https://www.youtube.com/watch?v=MUdB7xkChaA

68. “Conversational Intelligence & Behavioral Prediction Insights from
Voice: Our Oliver engine offers ASR+ with a sophisticated layer of
emotion recognition metrics & behavioral KPIs, not only from what is
being said but also from the how it is said.”

69. NLP 2020: What Works and What's Next
Recent Advances in Natural Language Processing
Seth Grimes
Alta Plana Corporation
@SethGrimes
November 20, 2020