Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Search Product Manager: Software PM vs. Enterprise PM or What does that * PM do?

The slide deck from my talk at the virtual Haystack 2020 conference via meetup.

  • Be the first to comment

Search Product Manager: Software PM vs. Enterprise PM or What does that * PM do?

  1. 1. Search Product Manager: Software PM vs. Enterprise PM What does that dam PM do? John T. Kane JT Kane Consultants Email: jtkane@gmail.com Twitter: @EnterpriseSrch https://www.linkedin.com/in/jtkane/
  2. 2. Agenda I) Introduction – My Search PM Journey II) Search Use Cases as a Horizontal Technology III) The Roles of Search and Product Manager within Business IV) Search & Business as Partners V) Thoughts on Search & PM Futures…
  3. 3. Introduction – My Search PM Journey (1) I didn’t emerge as a fully-fledged Search PM as every PM has their path. 1998 – Microsoft: SQL Server 7.0 alpha adds Full-text Search 2007 – IntelliSearch (Norway-based) Search Startup (Search Evangelist) 2008 – FAST ESP (Norway-based) acquired by MSFT – Sales Engineer 2010 – Editor/Contributor Book: Pro. Microsoft Search, published.
  4. 4. Introduction – My Search PM Journey (2) I didn’t know that a Search Product Management was my path. 2010 – Joined HP Inc. as BI/Architect & Search Product Manager 2013 – Joined Lucidworks as PM for Fusion (2012: life-changing event). 2015 – Started at Staples as Search PM for migration from Endeca. 2016 – Joined HP Enterprise as Site Search PM using GSS. 2018 – Joined Voyager Search as PM and Solution Architect. 2018 – Joined Zoosk (Online Dating site) with NO Search UI. 2019 – Started at Getty Images as Search PM improve Search Mission. 2020 – Freelance Search PM & Volunteer PM for COVID19 app.
  5. 5. Search Use Cases as a Horizontal Technology(1) Search Technology is Horizontal much like Relational Databases. As Websites specialize, so too does Search Use Cases get specialized. Search Use Cases can be broken down into three broad Categories: Public facing: Site Search (SEO’s Internal Search), eCommerce Search Dell Laptop Search: https://www.dell.com/koa/search?q=laptop Getty Images Skiing Search: https://www.gettyimages.com/photos/skiing HPE Site Search: https://www.hpe.com/us/en/search-results.html?page=1&q=gen%2010&autocomplete=0 Non-Public facing: Intranet Search, Enterprise Search https://www.microsoft.com/en-us/itshowcase/redefining-the-intranet-site-experience-with-sharepoint-in-office-365 (2019) https://destwo.net/portfolio/case-study-microsoft-web/ (circa 2014) – Microsoft’s Intranet: MSW Hybrid or Specialized Search: GeoSpatial Search, Collaborative Search VoyagerODN Geospatial: https://www.voyagersearch.com/odn-search Multi-Person or Collaborative Search: SearchTogether, and others.
  6. 6. Dynamic Personalized NextGen Search Dynamic Relevance Dynamic Users Dynamic Queries Dynamic Documents Dynamic Information Needs Users change behavior over time, user history Topic Trends, Filtering, document content change User perceived relevance changes Changing query, reformulation, Mission Information needs evolve over time Next Gen. Search Engine All Access points secured All Device form factors
  7. 7. Help! I’m stuck in the head = lost value in the “Long Tail” Source: Last.fm Dataset 72%
  8. 8. The Roles of Search and Product Management within the Business (1) As the Internet and Online Business grew, so too did the need for Technologists who understood the business use cases. The Product Management Role varies by Business & Industry. NO Standards! My "Five Things" or Search Product Feature Framework 1. Highly-Available (HA / AlwaysOn) = NO one point of failure 2. Fast! (Highly-performant) 3. Relevancy = Topical relevant / domain-specific / content-specific 4. Social / Behavioral / Interactive == Improves Relevancy (external signals, clicks, impressions, metrics) 5. Feedback = Always improving / Real-time feedback PM Building new Product or "Five Things“ for a Product Manager 1. Understand your target audience, really well. 2. Understand “why?” 3. Drill down. 4. Make a task as simple as possible 5. Forget about feature parity with competitors & Track all pain points.
  9. 9. The Roles of Search and Product Management within the Business (1) A brief list of Five Traits of Successful Search Product Managers. I’m explicitly avoiding the “good vs. great” PM conversation. 1. Empathy 2. Passion for [Search] products 3. Experimenter’s Mindset / Intellectual Honesty 4. Subject Matter Expert / Domain Knowledge 5. Aspires to build great user experiences
  10. 10. Product Management Roles: Software vs. Enterprise There are overlaps between these Roles and the below is an unordered list of differences (your mileage may vary) Always have Empathy for your Customers. Know them, Interview them, Survey Them! Software Search Product Manager end-product is Software or Service (SaaS) Software can be Search or other domains. Software Sales/Subs drive Business Revenue. Interact with Sales & Sales Engineering. Interact with Consultants / Pro. Services. Work with UX Designers, UI Developers. Work with Backend Search Engineers. Everyone is a Software Search Expert. Passion for multi-Search Use Cases. Empathy for End-Customers, Feel the Pain. Multi-Use Case experienced KPI’s OKR’s. Aspires to be great, but always humble. Able to explain complex topics, simply. Enterprise Product Manager, Search Understand the Business drivers & customers. Understands how Search fits within Business. SME and Domain (Search) Expert Interact with internal Stakeholders. Stakeholders can have conflicting objectives. Understand your Manager’s goals. Over communicate with your Manager. Work with all levels of Engineering Staff. Understand the Business’ End-Customers Research Search Uses, Tech & Experiences. Know Customer Journey & Searcher’s Path. Ensure Search Query Log & Click-stream exist. You are your Business’ Search Expert.
  11. 11. Sample Search Metrics & KPIs (1) Search KPI Goals Increase in visitor-to-site interaction & page depth click-through rate Increase in visitor clicking through to a linked second page Increase in visitor highly-engaged to third or fourth linked page depth Searcher Behavior Metrics 1. Most Popular Searches (must not be viewed in isolation, but in context with % of visitors who use Site Search) 2. Zero Results Searches 3. Top Selected Documents 4. Slow Search Queries 5. Most Popular Facets Value Selected 6. Most Popular Facets Displayed 7. Top X Documents per Top Y Searched Terms of Phrases Content-based Metrics 8. Crawler Statistics (success/failure) – not possible with GSS 9. Search Business Rules (landing pages or re-directs) 10. Term Frequency 11. Search Query-based Duplicate Content Detection 12. QPS – Queries Per Second – helps determine if we have slow searches 13. Collections – not the right name for GSS, but how can we know in advance that our content is crawled by GSS? 14. Error / Fault Reporting 11
  12. 12. Sample Search Metrics & KPIs (2) Information Find Conversion Rate - Total Number of Visits to the Goal Page / Total Visits a) This KPI is a variation on the order conversion rate in which we simply loosen the definition of “order” to reflect visitors seeing critical goal pages b) call this KPI the “image find rate” if the "goal" is for customers who do "research" but do not actually download. (related to searcher intent) c) consider this KPI in tandem with average page views per visit to monitor for an artificially high conversion rate, essentially visitors viewing multiple overlap pages Search to Purchase Conversion Rate - Total Orders Attributed to Searchers / Total Number of Visits to a Search Results Page a) tracks how searching visitors differentially convert into customers to help understanding the real value investing in new search technology. b) this too is special case of the order conversion rate, similar to Information Find Conversion Rate above. Search Results to Site Exits Ratio - Total Site Exits from Search Results Page / Total Number of Visits to a Search Results Page a) Similar to percent zero yield search results, the ratio of search results to site exits ratio will help you understand whether visitors find Site Search useful. b) Roughly the opposite of the search to purchase conversion rate, this ratio can help understand how search is driving visitor failure and dissatisfaction. c) co-present this KPI with average searches per visit, percent zero results searches and percent zero yield searches to provide the proper context. Percent Visitors Using Search - Total Number of Visitors Who See at Least One “Search Results” Page / All Visitors a) this KPI is relatively static, any dramatic changes are likely describing some problem caused by recent changes to the site. Percent Zero Result Searches - Total Number of “Zero Result” Searches / All Search Results a) Both the numerator and denominator in this calculation should be measured in page views, not visits or visitors (may search multiple times during a visit) 12
  13. 13. Sample Search Metrics & KPIs (3) Percent Zero Yield Searches - Total Number of Search Results Pages from which Visitors Did Not Click a Search Result / All Search Results a) If it is easier to track that visitors are clicking on a search result link, you can use that number as well in the numerator (based upon pageview basis) b) 1.00 – (Total Number of Search Results Pages from which Visitors Did Click a Search Result / All Search Results) = Percent Zero Yield Searches (pageview basis) Average Page Views per Visit - Search technology: If your search functionality is poor, visitors may be forced to click to look for information. a) Conversely, if your search functionality is good, visitors may be leveraging search, thusly reducing the number of pages viewed. b) average page views per visit (diagnose problems) Site Search: percent visitors using search (see above), percent “zero result” searches & average searches per visit. Average Searches per Visit - Total Number of Searches (Page Views) / Total Visits a) present this along with percent visitors using search to provide context. The goal is to monitor the visitor’s relationship with deployed search technology. Popular Site Search terms Not visible in Site structure (Website Percent High, Medium and Low Time Spent Visits & diagnose problems) a) site search logs, looking for commonly searched for terms that may highlight visitor interests not clearly represented in the site’s navigational structure. Popular Site Search terms Not visible in Site Clicks - (Percent High, Medium and Low Click Depth Visits & diagnose problems) a) site search logs, looking for commonly searched for terms that may highlight visitor interests not clearly represented in the site’s navigational structure. 13
  14. 14. Search & Data Science Prioritization Process Process Planning Notes: Quad Matrix: Effort vs. Business vs. Risk (LoE) = Prioritized Features.
  15. 15. Search Roadmap – Project Constraints. Triangle: Effort vs. Time vs. Money (budget) = Limited Resources.
  16. 16. What makes a good recommendation? A: RecSys KPIs • Accuracy • Coverage • Long Tail • Serendipity • Novelty • Search and Recommendations are becoming one..
  17. 17. Discovery / Exploratory 17
  18. 18. Dynamic Searchers: Novice vs. Expert 18
  19. 19. Knowledge Graph for Users, Clicks & Image Taxonomy 19
  20. 20. Aspects of a Search Architecture Text-Based Search Ranking (words to match words) – add why. Define language-specific Noun phrase use case / why matters most. Or biggest impact Text Search Models (TF/IDF or BM25 to LTR) Textual Query Preprocessing (via Query Pipeline) Query Expansion (add synonyms & paraphrases) Stemming Algorithm (porter vs. lemmatization) Stopword Removal (some to none) N-Gram Query Segmentation (Fuzzy or part of words) Part-of-Speech Tagging (POS for NLP) Text Sources (add metadata via Indexing Pipeline) Query Example-Based Search Ranking (Reverse image) Concept-Based Search Ranking (GKG & extend, content-side) Query-Concept Mapping (Query-side)
  21. 21. Example Search Roadmap Schedule (20% confidence) Exit Walk: ES-based Reciprocal Recommender & Train Data. Jan. 2020 Apr. 2020 May 2020 Train Engineers in on Solr + ML L2R Develop / Deploy Text Profile, Clicks, Tagging Features Monitor Member Usage Patterns & KPI’s Determine exit metrics for Walk phase Jun. 2020Mar. 2020Feb. 2020 Connect to Front-end UI Architect NLP + ML- based Recommender Test multiple Search with same content Plan Run Phase Features & Benefits. Run: TBD Crawl: Standup new Solr-based Search Walk 2.0: New ES- based Features Jul. 2020 Walk 1.0: ES-based Recommender + Business Features
  22. 22. References: Books, Blogs &Bits.. (1) Enterprise Search: Enhancing Business Performance (2nd edition) - He's expanded this to more use cases https://www.amazon.com/Enterprise-Search-Enhancing-Business-Performance/dp/1491915536/ https://www.cmswire.com/author/martin-white/ - he also writes a column for CMS Wire Designing the Search Experience: The Information Architecture of Discovery https://www.amazon.com/Designing-Search-Experience-Information-Architecture/dp/0123969816 Search Patterns https://www.amazon.com/Search-Patterns-Discovery-Peter-Morville/dp/0596802277 Designing Search: UX Strategies for eCommerce Success https://www.amazon.com/Designing-Search-Strategies-eCommerce- Success/dp/0470942231/ref=pd_lpo_14_t_2/144-5441348-2297159 Search Analytics for your Site https://www.amazon.com/dp/B005EI86HC/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1
  23. 23. References: Books, Blogs &Bits.. (2) Medium.com - is a good source of both business and technical search topics https://medium.com/search?q=%22ecommerce%20search%22 https://medium.com/search?q=solr https://medium.com/search?q=elasticsearch IR or Search Conferences - most papers should be online. https://sigir.org/sigir2020/ https://ecir2020.org/ https://sigir.org/chiir2020/ Real-Time Visual Navigation in Huge Image Sets using Similarity Graphs 2019 Demo site: https://wikiview.net/ - this is Image Exploratory Search UI site that I spoke about. Struggling or Exploring? Disambiguating Long Search Sessions https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/HassanWSDM14.pdf Exploratory search: user behaviour and search engine adaptation by Alan Medlar - Feb 2020 Free access https://dl.acm.org/doi/abs/10.1145/3352683.3352687 Challenges in Supporting Exploratory Search through Voice Assistants - March 2020 Note: this one is related to your question about Conversational Search, re: Alexi, Echo etc. https://arxiv.org/pdf/2003.02986.pdf
  24. 24. References: Books, Blogs &Bits.. (3) Serendipity Search A new online startup serendipitous service to put serendipity back into your life! http://www.serenflipity.com/#intro The serendipity society - Serendipitous Resources & papers https://theserendipitysociety.wordpress.com/resources/ https://twitter.com/ser_soc Planned Luck: How Incubators Can Facilitate Serendipity for Nascent Entrepreneurs Through Fostering Network Embeddedness - May 2020 - By Christian Busch https://journals.sagepub.com/doi/pdf/10.1177/1042258720915798 Observing Serendipity in Digital Information Environments - 2015 https://www.google.co.uk/search?domains=informationr.net&q=serendipity - review of IR papers open Searching Linked Data with a Twist of Serendipity - 2017 http://www-di.inf.puc-rio.br/~casanova//Publications/Papers/2017-Papers/2017-CAiSE-Eichler.pdf
  25. 25. Some Thoughts on Search & PM Futures… Predicting the Future is difficult. Consider this a list of Dots to Connect for future Search Innovation. Always be Learning, Growing New Skills, Be open minded, Find your Niche. Search Technology – Dots to Connect. Search as a Service (SaaS) is foundational. Learning to Rank (LTR) is a critical component Learning to Rank tools will become common. Online Learning to Rank (OLTR) is Self-Learning. Deep Learning for Search will have tires kicked. Carousel UI KPI’s are different than Clicks. Open Source Solr & ElasticSearch will evolve. Graphs (Knowledge, Interest, eCommerce) Niche Search & General Search will co-exist. Dynamic Search will become more relevant. Always have built-in Metrics & Analytics. Aspires to be great, but always humble. Able to explain complex topics, simply. Product Manager – Dots to Connect. Agile Framework will evolve, become practical. Other Frameworks will emerge, Jobs-to-be-Done. Always be Learning, Curious, Open-minded. Stakeholders will always be a Challenge. Business Location-based work will evolve fast. Remote Work requires over-communicating Switch Work hours with Remote-based Time zones. Resume-based Features are Tempting, Resist! Your End-Customers Patterns will change more. Research new Search emerging Technology. Know Customer Journey & Searcher’s Path. Search Query Log + Click-stream is your Gold Mine. You are your Business’ Search Expert.
  26. 26. My Search Journey as a Product Manager Thank You! John T. Kane JT Kane Consultants Email: jtkane@gmail.com Twitter: @EnterpriseSrch https://www.linkedin.com/in/jtkane/
  27. 27. Background slides below.
  28. 28. Search: Consider as the front door • Today’s web search is built on the foundation of classic IR • Treat documents as bags of keywords that are matched against user queries through statistical techniques • Inverted indexes and TF/IDF are at the heart of today’s search engines • Google’s PageRank uses link structures and dynamic ranking to improved relevance of results. • Knowledge Graph & RankBrain build upon these structures.Inverted Index w1 → d1 d4 d50 w2 → d3 d10 d32 …. wN → d11 d20 d2003 28
  29. 29. Contextual Query Understanding • User queries can be reformulated and intent inferred based on different types of context • User: Short term (interest) and Long term (knowledge) models • Task: Current session’s objective explicitly (provided by user) or implicitly (derived from page visitations) captured from the user • Community: Derived from the links between content and social graph • Collaboration: Multiple users collaborating to perform a complex task • Domain/Topic: Query classification leading to better identification of user’s information need • Time-Space: Temporal and Geographic constraints that relevant information should satisfy • Application: Constraints on the sources searched and information retrieved • Web Search uses some of these query contexts to satisfy user’s information needs • E.g. Geographic constraints in Local Search, User interests in personalized Search • Limitations of the frameworks that capture, represent and use what is known to the user • Each user activity provides some information or knowledge about the topic of interest to the user and about the user • Audience intelligence classifies users to a predefined set of categories capturing their interests • Search history captures information on user’s short-term tasks/interest • Interpreting such information from different sources provides the context for understanding user query. 29
  30. 30. Building NextGen Graph Discover User-to-Image Graph Interest Search Users • Find users/experts with interests in certain topics • Search for topics in the Entity Graph and project the results to User Graph Discover Users • Find groups that a user would be interested in • Project user interests from Entity Graph to User Graph to find group membership. • Find users by tagging/annotating KB content • Project tagged content to the User Graph • User group discovery • Cluster users based on the semantic clustering of their interests Search Knowledge • Improving Selection and Relevance • Exploit User interests projected to Entity Graph to select and rank search results • Augmenting Search Results • Recommending related knowledge and resources for a given query Discover Knowledge • Community Interests and Trends • Cluster and track concepts of interest to a contributor community • Search Patterns and Templates • Repurpose patterns in user navigation as templates for future search. 30
  31. 31. Screenshot for “Expanded Query Terms” usage via Vectors

×