Getting good search results is hard; maintaining good relevancy is even harder. Fixing one problem can easily create many others. Without good tools to measure the impact of relevancy changes, there's no way to know if the "fix" that you've developed will cause relevancy problems with other queries. Ideally, much like we have unit tests for code to detect when bugs are introduced, we would like to create ways to measure changes in relevancy. This is exactly what we've done at OpenSource Connections. We've developed a series of tools and practices that allow us to work with content experts to define metrics for search quality. Once defined, we can instantly measure the impact of modifying our relevancy strategy, allowing us to iterate quickly on very difficult relevancy problems. Get an in depth look at the tools we utilize when we not only need to solve a relevancy problem, we need to make sure it stays solved over the product's life.
Scaling API-first ā The story of a global engineering organization
Ā
Test Driven Relevancy -- How to Work with Content Experts to Optimize and Maintain Search Relevancy
1.
2. TEST DRIVEN RELEVANCY
How to Work with Content Experts to Optimize and
Maintain Search Relevancy
Doug Turnbull
Rena Morse
Search Relevancy Expert Director of Semantic Tech
OpenSource Connections Silverchair Information
Systems
3. Its us!
Hi Iām Doug!
@softwaredoug
http://www.linkedin.com/in/softwaredoug
http://bit.ly/softwaredoug
Hi Iām Rena!
renam@silverchair.com
4. How do sales/content curators collaborate with devs?
āWhen doctors search for āmyocardial infarctionā these
documents about āheart attacksā should come upā
āMyocardial in-what-tion? Dangit Rena, Iām a
Solr consultant ā not a doctorā
āI donāt evenā¦ā
ālet me work my Solr magic and get back to
you next weekā¦ā
5. How do content curators collaborate with devs?
ā¢
Doug knows Solr
ā¢
Bob knows his business
Rena knows her content
Supplier Pressure
Sales
Conversions
Niche customers
q={!boost b=log(numCatPictures)}bunnies
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
Myocardial infarction
Renal carcinoma
This is a universal pattern ā it takes different strokes!
6. How do content curators collaborate with devs?
āRena, I fixed that myocardial in-whatever-tion relevancy
issueā
āOk but you broke this other thing I thought
was fixed!ā
<reiterates that heās a search expert not a doctor>
<reiterates sheās a paying client>
āok let me see what I can doā¦. Iāll get back to you in a
weekā
7. Our Problems
ā¢
People Problem: Our collaboration stinks
ā āThrow problems over the fenceā¦ wait a weekā
ā Siloed by expertise (search vs content experts)
ā Potential for politics, anger, frustration (all paths to the dark side)
ā¢
Technical Problem: Search testing is hard
ā Small set of relevancy rules impacts all user searches
ā Therefore: Much easier to have regressions than in software
ā Very easy to have regressions
8. Our Problems
ā¢
In short, Broken Workflow:
ā Iterations between devs and non-technical experts take a long time
ā Devs need immediate access to non-technical expertise to make rapid
progress
ā¢ Gather broken searches
ā¢ Rate existing searches
ā¢ Find searches that have slid backwards in quality
ā Non-technical experts clearly need devs
ā¢ Translate business rules to relevancy rules
ā¢ Bending sophisticated token matching engine to our approximate user
intent
9. Our Problems
ā¢
Our lack of collaboration means our testing stinks
ā Need expert feedback to test
āthis is good, this is bad, this is okā¦.ā
ā¢
Search devs often donāt know good search! They need help.
āI need an army of Renas locked in a room
telling me what is good and badā
10. Solutions?
ā¢
In s/w development -- automated testing is often away to collaborate
ā Devs Can sit together with content experts and ask:
ā¢ What should happen in this case?
ā Then record that in the form of a test
@Given("tab $asciiTab")
public void tab(String asciiTab) {
tab = new Tab(asciiTab, new TabParser());
}
@When("the guitar plays")
public void guitarPlays() {
guitar = new Guitar();
guitar.play(tab);
}
@Then("the following notes will be played $notes")
public void theFollowingNotesWillBePlayed(String notes) {
ensureThat(expectedNotes(notes), arePlayedBy(guitar));
}
11. Test Driven Development with Search
ā¢
Collaborative testing is absolutely essential for search
ā Good search is best defined by experts in the content:
ā¢ Marketing, sales, users, etc
Iām a search expert! Not a Doctor!
How can I possibly measure search
ā¢
Unfortunately thereās nothing to help content experts communicate with search
devs around search (frankly this is rather shocking to me)
Help me help you! I have few ways to
record, measure, and evaluate search
12. Test Driven Development with Search
ā¢
Collaborative testing is absolutely essential for search
ā Every change to search relevancy will cause other queries to change
ā MUST know how much our relevancy has changed
āI fixed your searchā¦ Does it matter that toe fungus query
changed by 30%?ā
āYeah lets see if we can work together to balance
the two relevancy concernsā
āIām glad we have tests to know whatās changed!ā
13. Test Driven Development with Search
ā¢
Apply these ideas to search quality:
ā Given query Y
ā What documents are awesome/good/ok/bad/terrible?
ā¢
ā¢
Record these ratings somewhere
Observe the changes of all queries simultaneously as we modify relevancy params
Now Iāve got the ultimate relevancy workbench. I can
see if my ideas are working or failing right away!
Now I cane see instantly if Dougās changes are making
progress or if weāre moving backwards!
14. Quepid! (give your queries some love!)
ā¢
ā¢
We built a tool (a product!) around these ideas
Now our favorite relevancy workbench
Learn more about Quepid!
http://quepid.io
āHey weāre kicking butt and
taking names on these search
queries!ā
15. Test Driven Development with Quepid
Search Quality is about Collaboration*
*and collaboration is about testing
Demo time