Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Next Generation of AI-powered Search

What does it really mean to deliver an "AI-powered Search" solution? In this talk, we’ll bring clarity to this topic, showing you how to marry the art of the possible with the real-world challenges involved in understanding your content, your users, and your domain. We'll dive into emerging trends in AI-powered Search, as well as many of the stumbling blocks found in even the most advanced AI and Search applications, showing how to proactively plan for and avoid them. We'll walk through the various uses of reflected intelligence and feedback loops for continuous learning from user behavioral signals and content updates, also covering the increasing importance of virtual assistants and personalized search use cases found within the intersection of traditional search and recommendation engines. Our goal will be to provide a baseline of mainstream AI-powered Search capabilities available today, and to paint a picture of what we can all expect just on the horizon.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

The Next Generation of AI-powered Search

  1. 1. Trey Grainger Chief Algorithms Officer Lucidworks Next Generation of The
  2. 2. What is ?
  3. 3. AI? Machine Learning? Data Science? Neural Search? AI-Powered Search? Deep Learning?
  4. 4. http://aiPoweredSearch.com ... is my new book!
  5. 5. Keyword Search Knowledge Graph User Intent Personalized Search Semantic Search Domain-aware Matching The Goal of AI-powered Search Content Understanding Domain Understanding Collaborative Recommendations User Understanding
  6. 6. image credit: SAS Institute, 1998 Definitions, circa 1998
  7. 7. AI-powered Search
  8. 8. AI-powered Search Question / Answer Systems Virtual Assistants • Signals Boosting Models • Learning to Rank • Semantic Search • Collaborative Filtering • Personalized Search • Content Clustering • NLP / Entity Resolution • Semantic Knowledge Graphs • Document Classification • etc. • Neural Search • Word Embeddings • Vector Search • Image / Voice Search • etc. • Question / Answer Systems • Virtual Assistants • Chatbots • Rules-based Relevancy • etc.
  9. 9. Proudly built with open-source tech at its core: Apache Solr & Apache Spark Personalizes search with applied machine learning Proven on the world’s biggest information systems
  10. 10. Query Intent (query classification, semantic query parsing, semantic knowledge graphs, concept expansion, automatic query rewrites, clustering, personalization, question/answer systems, virtual assistants) Search Intelligence Spectrum Basic Keyword Search (inverted index, tf-idf, bm25, multilingual text analysis, query formulation, etc.) Taxonomies / Entity Extraction (entity recognition, taxonomies, ontologies, business rules, synonyms, etc.) Automated Relevancy Tuning (Signals boosting, AB testing / multi- armed bandits testing, signals-based back-testing, genetic algorithms, Learning to Rank) Self-learning
  11. 11. People to People People to Information People to Insights Search + AI=
  12. 12. Watson: “You appeared to [see a good deal] which was quite invisible to me” Sherlock: “Not invisible but unnoticed, Watson. You did not know where to look, and so you missed all that was important.” The Adventures of Sherlock Holmes, ADVENTURE III. A CASE OF IDENTITY, Sir. Oliver Conan Doyle
  13. 13. Head? Pipe? Coat Collar? Back of Hat? Hat? Smoke? Nose? Abstract Concept of Detective with Pipe Specific hypothesis from Experience (leveraging social cue that this is probably a well-known answer) Detective (Deerstalker) Hat! Final Answer + conceptual context
  14. 14. Natural Language Understanding
  15. 15. Ok, Google… Is Agave Nectar good for you?
  16. 16. How can we help
  17. 17. = = = =
  18. 18. Significance of Feedback Loops User Searches User Sees Results User takes an action Users’ actions inform system improvements
  19. 19. Signals Boosting Popularized Relevancy Learning to Rank Generalized Relevancy Collaborative Filtering Personalized Relevancy
  20. 20. Signals Boosting Popularized Relevancy Learning to Rank Generalized Relevancy Collaborative Filtering Personalized Relevancy
  21. 21. Signal Collection & Processing User Searches User Sees Results User takes an action Users’ actions inform system improvements User Query Results Alonzo ipad doc10, doc22, doc12, … Elena printer doc84, doc2, doc17, … Ming ipad doc10, doc22, doc12, … … … … User Action Document Alonzo click doc22 Elena click doc17 Ming click doc12 Alonzo purchase doc22 Ming click doc22 Ming purchase doc22 Elena click doc2 … … … ipad ⌕ Signal Processing & Machine Learning Learned Relevance Models
  22. 22. ipad
  23. 23. ipad
  24. 24. Crowdsourced Relevancy Crowdsourced Biases
  25. 25. Still Available through Query Variations Manual Override By Facebook
  26. 26. ipad Bananas
  27. 27. ipad Bananas
  28. 28. Risks Potential Solutions Reinforces Current Biases • Learning to Rank • Use-case-specific Click Models • User-group bias detection • Inject conceptual diversity Subject To Manipulation • Session Filtering • Quality vs. Quantity Signal Weighting • Only Count Explicit Actions Cold Start Problem • Content-based Relevance • Learning to Rank Signals Boosting + Collaborative Filtering
  29. 29. Knowledge Graphs Domain modeling Multi-modal Learning Merged Content Modalities Thought Vectors Conceptual scoring
  30. 30. Learned Knowledge Graphs Trey Grainger works for Lucidworks. He is speaking at the Activate 2019 conference. #Activate19 (Activate) is being held in Washington, DC September 9-12, 2019. Trey got his masters degree from Georgia Tech. Trey’s Voicemail
  31. 31. Learned Knowledge Graphs Trey Grainger works for Lucidworks. He is speaking at the Activate 2019 conference. #Activate19 (Activate) is being held in Washington, DC September 9-12, 2019. Trey got his masters degree from Georgia Tech. Trey’s Voicemail
  32. 32. Multimodal Learning
  33. 33. Phrase: Vector: apple juice: [ 1, 5, 0, 0, 0, 4, 4, 3 ] cappuccino: [ 0, 5, 3, 0, 4, 1, 2, 3 ] cheese bread sticks: [ 5, 0, 4, 5, 0, 1, 4, 2 ] cheese pizza: [ 5, 0, 4, 4, 0, 1, 5, 2 ] cinnamon bread sticks: [ 5, 0, 4, 5, 0, 1, 4, 2 ] donut: [ 5, 0, 1, 5, 0, 4, 5, 1 ] green tea: [ 0, 5, 0, 0, 2, 1, 1, 5 ] latte: [ 0, 5, 4, 0, 4, 1, 3, 3 ] soda: [ 0, 5, 0, 0, 3, 5, 5, 0 ] water: [ 0, 5, 0, 0, 0, 0, 0, 5 ] Ranked Results: Green Tea 0.94 water 0.85 cappuccino 0.80 latte 0.78 apple juice 0.60 soda … … 0.19 donut Vector Similarity Scores: Vector Similarity (a, b): a · b = |a| × |b| × cos(θ) Ranked Results: Cheese Pizza 0.99 cheese bread sticks 0.91 cinnamon bread sticks 0.89 donut 0.47 latte 0.46 apple juice … … 0.19 water
  34. 34. Sentence Embeddings: [ 2, 3, 2, 4, 2, 1, 5, 3 ] [ 5, 3, 2, 3, 4, 0, 3, 4 ] . . . Document Embedding: [ 4, 1, 4, 2, 1, 2, 4, 3 ] Word Embeddings: [ 5, 1, 3, 4, 2, 1, 5, 3 ] [ 4, 1, 3, 0, 1, 1, 4, 2 ] . . . Paragraph Embeddings: [ 5, 1, 4, 1, 0, 2, 4, 0 ] [ 1, 1, 4, 2, 1, 0, 0, 0 ] . . . Thought Vectors
  35. 35. Keyword Search Collaborative Recommendations Knowledge Graph User Intent Personalized Search Semantic Search Domain-aware Matching The Goal of AI-powered Search Content Understanding User Understanding Domain Understanding
  36. 36. The Next Generation of AI-powered Search
  37. 37. Trey Grainger trey@lucidworks.com @treygrainger Other presentations: http://www.treygrainger.com http://aiPoweredSearch.com Books:

    Be the first to comment

    Login to see the comments

What does it really mean to deliver an "AI-powered Search" solution? In this talk, we’ll bring clarity to this topic, showing you how to marry the art of the possible with the real-world challenges involved in understanding your content, your users, and your domain. We'll dive into emerging trends in AI-powered Search, as well as many of the stumbling blocks found in even the most advanced AI and Search applications, showing how to proactively plan for and avoid them. We'll walk through the various uses of reflected intelligence and feedback loops for continuous learning from user behavioral signals and content updates, also covering the increasing importance of virtual assistants and personalized search use cases found within the intersection of traditional search and recommendation engines. Our goal will be to provide a baseline of mainstream AI-powered Search capabilities available today, and to paint a picture of what we can all expect just on the horizon.

Views

Total views

753

On Slideshare

0

From embeds

0

Number of embeds

115

Actions

Downloads

40

Shares

0

Comments

0

Likes

0

×