Solr & Lucene at Etsy provides concise summaries of Gregg Donovan's experience using Solr and Lucene at Etsy and TheLadders, including optimizations made to maximize performance out of the box, techniques for low-level hacking, and when each approach is best applied. Key points covered are maximizing Solr functionality, continuous deployment, cheap performance wins, and tools for low-level hacking.

1. Solr & Lucene at Etsy
Gregg Donovan
Technical Lead, Search
gregg@etsy.com

2. 1.5 years Solr & Lucene at Etsy.com
3 years Solr & Lucene at TheLadders.com

4. 8+ million members

5. 9.3 million items

6. 800k+ active sellers

7. 1+ billion pageviews / month

14. Maximize Solr out-of-the-box

15. Hack at a low-level

16. Know when to do each

18. Or

20. Don’t fear trunk

21. builds.apache.org/job/Solr-trunk/changes

24. http://localhost:8393/solr/placesuggest/
select?
q={!lucene}s*
&sfield=latlong&pt=37.595804,-122.364521
&sort=div(geodist(),sqrt(sum(population,50)))
%20asc

25. {!lucene}
{!field}
{!term}
{!boost}
{!func}
{!dismax}
{!edismax}

26. Cheap ranking awesomeness

28. ExternalFileField ftw!

29. schema.xml:
<fieldType name="file" keyField="treasury_id" defVal="0"
stored="false" indexed="true" class="solr.ExternalFileField"
valType="float"/>
<field name="hotness" type="file"/>
/search/data/treasury/external_hotness.1306390802088:
1=2.3
2=1.7
3=1.1
Solr query:
sort={!func}hotness+desc

30. ExternalFileField caveats

31. More relevance: boost query

32. http://localhost:8983/solr/listings/select?
q={!boost b=$rel v=$qq}
&rel=category:furniture^10+OR+((-material:acrylic)
^5)
&qq=desk

33. Impression tracking

34. etsy.com/search?q=desk&explain=1

35. Side-by-Side testing

37. Cheap performance wins

38. Put off sharding till you must

39. cat ${indexDir}/* > /dev/null

40. Return IDs, minimize stored fields

41. RAM: $10-20 / GB

42. SSD: 0.1ms vs 10ms seek

43. Custom?

44. solr-user

45. Tools for low-level hacking

46. Continuous deployment

48. One button.
So easy a dog could do it.

51. MTTR > MTBF

54. github.com/etsy/logster

55. Tracking GC

56. export GC_DEBUG="-verbose:gc -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:
+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy -
XX:AdaptiveSizePolicyOutputInterval=1 -XX:+PrintTenuringDistribution -XX:
+PrintGCDetails -Xloggc:/var/log/search/gc.log"

59. Alerting

60. Testing

62. SaveAsFixture

63. Profiling

64. Java Primitive Library
fastutil
trove4j

65. Know the hooks
SolrRequestHandler
SearchComponent
QParserPlugin
SolrEventListener
SolrCache
ValueSourceParser

66. SolrIndexSearcher gotchas
reference counting
using it as a cache key:
WeakHashMap<SolrIndexSearcher,MyValue> myCache...

67. Example:
personalized collections

69. fq={!term f=id}123 OR {!term f=id}456

70. Need a map of PK to docId

71. Use custom SolrCache plus SolrEventListener
to fill it

72. github.com/giokincade/FastTermFilter

73. i18n currency sorting and filtering

75. currency.xml:
<currencyConfig version="1.0">
! <currencies>
! ! <currency name="United States Dollar" symbol="$" code="USD"/>
! ! <currency name="Australian Dollar" symbol="$" code="AUD"/>
! ! <currency name="Canadian Dollar" symbol="$" code="CAD"/>
! ! <currency name="Czech Koruna" symbol="Kč" code="CZK"/>
...
! </currencies>
! <rates>
! ! <rate from="USD" to="AUD" rate="1.168750"/>
! ! <rate from="USD" to="CAD" rate="1.085000"/>
! ! <rate from="USD" to="CZK" rate="20.107500"/>
! ! <rate from="USD" to="DKK" rate="5.323750"/>
...
</rates>
</currencyConfig>

76. price:[$10.00 to $50.00]
price:[10.00USD to 50.00USD]
price:20.00EUR

77. MoneyFieldType.java:
@Override
public Query getRangeQuery(QParser parser, SchemaField field, String part1, String part2,
final boolean minInclusive, final boolean maxInclusive) {
final MoneyValue p1 = MoneyValue.parse(part1, defaultCurrency);
final MoneyValue p2 = MoneyValue.parse(part2, defaultCurrency);
if (!p1.getCurrencyCode().equals(p2.getCurrencyCode())) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
new ParseException("Cannot parse range query " + part1 + " to " + part2 +
": range queries only supported when upper and lower bound have same
currency."));
}
String currencyCode = p1.getCurrencyCode();
final MoneyValueSource vs = new MoneyValueSource(field, currencyCode, parser);
return new SolrConstantScoreQuery(new ValueSourceRangeFilter(vs,
p1.getAmount() + "", p2.getAmount() + "", minInclusive, maxInclusive));
}

78. Replication gotcha

79. SOLR-2202

80. Related Searches

81. Autosuggest!

82. bjewlery dewelry ejewelry ejwelry ewelery ewerly ewlery fewelry
fewlery fjewelery fjewelry gewerly gewlery hewelery hewelry hewerly
hewlery hjewelry iewelry ijewelry jawelery jawlery jeawlery jeelery
jeelry jeewelery jeewelry jeewlery jeewlry jefwelry jejelry jelelry
jelery jellery jelwelery jelwelry jelwlery jemelry jemerly jemwelry
jeqwelry jerelery jerelry jerely jererly jerlery jerwelery jerwelry
jerwely jerwerly jeselery jeselry jevelry jeverly jewalery jewdelry
jewedlry jeweelrry jeweelry jeweely jeweer jeweery jeweilry jeweiry
jewejery jewejlry jewejrly jewejry jewekey jewekry jewelary jeweldy
jewele jewelee jewelelry jewelera jewelerey jewelerly jewelert
jewelerty jeweleru jeweleruy jeweleryl jewelerys jeweleryy jewelet
jewelety jeweleya jewelfry jewelfy jeweliy jewellryp jewelltry
jewelly jewelory jewelra jewelray jewelre jewelree jewelreyy
jewelrfy jewelrh jewelri jewelrky jewelrly jewelrr jewelrs jewelrsy
jewelrt jewelrty jewelru jewelruy jewelrye jewelryh jewelryl
jewelrym jewelryr jewelrys jewelryt jewelryu jewelryuk
jewelryy jewelrz jewelsry jewelsy jeweltry jewelty jewelw jewelwery
jewelwey jewelwy jewelya jewelyj jewelyr jewelyry jewelyu jewelyy
jewelzry jeweory jewerey jeweriy jewerky jewerlary jewerley jewerli
jewerlly jewerls jewerlt jewerlu jewerlyh jewerlyr jewerlys jewerlyu
jewerry jeweryl jewetry jewewlry jewewly jewewrly jewewry jeweylry
jewiery jewilary jewkery jewlary jewledy jewleery jewlelery jewlely

83. The TermDictionary is not a whitelist