Successfully reported this slideshow.
Upcoming SlideShare
×

# Numeric Range Queries in Lucene and Solr

Presentation covers core lucene/solr stuff which is used in numeric range queries. There are several examples, algorithm discovered by Uwe is briefly explained.

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

### Numeric Range Queries in Lucene and Solr

1. 1. Numeric Range Queries in Lucene and Solr kirilchukvadim@gmail.com
2. 2. Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
3. 3. Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
4. 4. Range Queries: A range query is a type of query that matches all documents where some value is between an upper and lower boundary: Give me: ● Jeans with price from 200 to 300\$ ● Car with length from 5 to 10m ● ...
5. 5. Range Queries: In solr range query is as simple as: q = field:[100 TO 200] We will talk about Numeric Range Queries but you can use range queries for text too: q = field:[A TO Z]
6. 6. Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links (relax)
7. 7. Which field type? Which field type to use for “range” fields (let’s stick with int) in schema? ● solr.IntField ● or maybe solr.SortableIntField ● or maybe solr.TrieIntField
8. 8. Which field type? Let’s assume we have: ● 11 documents, id: 1,2,3,..11 ● each doc has single value “int” price field ● document id is the same as it’s price ● q = *:* "numFound": 11, "docs": [ { "id": 1, “price_field": 1 }, { "id": 2, “price_field": 2 }, ... { "id": 11, “price_field": 11 }]
9. 9. Which field type - solr.IntField q = price_field:[1 TO 10]
10. 10. Which field type - solr.IntField q = price_field:[1 TO 10] "numFound": 2, "start": 0, "docs": [ { "price_field": 1 }, { "price_field": 10 } ] }
11. 11. Which field type - solr.IntField Store and index the text value verbatim and hence don't correctly support range queries, since the lexicographic ordering isn't equal to the numeric ordering [1,10],11,2,3,4,5,6,7,8,9 Interesting, but “sort by” works fine.. Clever comparator knows that values are ints!
12. 12. Which field type - solr.SortableIntField ● q = price_field:[1 TO 10] ○ "numFound": 10 ● “Sortable”, in fact, refer to the notion of making the numbers have correctly sorted order. It’s not about “sort by” actually! ● Processed and compared as strings!!! tricky string encoding: NumberUtils.int2sortableStr(...) ● Deprecated and will be removed in 5.X ● What should i use then?
13. 13. Which field type - solr.TrieIntField ● q = price_field:[1 TO 10] ○ "numFound": 10 ● Recommended as replacement for IntField and SortableIntField in javadoc ● Default for primitive fields in reference schema ● Said to be fast for range queries (actually depends on precision step) ● Tricky and, btw wtf is precision step?
14. 14. Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
15. 15. Under the hood - Index
16. 16. Under the hood - Index NumericTokenStream is where half of magic happens! ● precision step = 1 ● value = 11 00000000 00000000 00000000 00001011 ● Let’s see how it will be indexed!
17. 17. Under the hood - Index Field with precisionStep=1
18. 18. Under the hood - Index shift=0 00001011 11 shift=1 00001010 10 = 5 << 1 shift=2 00001000 8 = 2 << 2 shift=3 00001000 8 = 1 << 3 shift=4 00000000 0 = 0 << 4 shift=5 00000000 0 = 0 << 5 continue…
19. 19. Under the hood - Index How much for an integer? 11111111 11111111 11111111 11111111 Algorithm requires to index all 32/precisionStep terms So, for “11” we have 11, 10, 8, 8, 0, 0, 0, 0, 0….0
20. 20. Under the hood - Index Okay! We indexed 32 tokens for the field. (TermDictionary! Postings!) Where is the trick? Stay tuned!
21. 21. Under the hood - Query
22. 22. Under the hood - Query Sub-classes of FieldType could override #getRangeQuery(...) to provide their own range query implementation. If not, then likely you will have: MultiTermQuery rangeQuery = TermRangeQuery. newStringRange(...) TrieField overrides it. And here comes...
23. 23. Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
24. 24. Numeric Range Query (Decimal) ● Decimal example, precisionStep = ten ● q = price:[423 TO 642]
25. 25. Numeric Range Query (Binary) ● precisionStep = 1 ● q = price:[3 TO 12] 0 1 2 3 4 5 6 7 8 9 10 11 12 13
26. 26. Numeric Range Query (Binary) ● precisionStep = 1 ● q = price:[3 TO 12] SHIFT = 1 0 0 1 1 2 3 2 3 4 5 6 4 7 8 5 9 10 6 11 12 13
27. 27. ... Numeric Range Query (Binary) ● precisionStep = 1 ● q = price:[3 TO 12] 0 0 0 1 0 0 0 0 1 1 2 1 2 3 2 3 4 5 6 3 4 7 8 5 9 10 6 11 12 13
28. 28. Numeric Range Query (Binary) ● precisionStep = 1 ● q = price:[3 TO 12] 0 1 0 0 0 0 1 1 2 1 2 3 2 3 4 5 6 3 4 7 8 5 9 10 6 11 12 13
29. 29. Numeric Range Query (How?) So, the questions is: How to create query for the algorithm?
30. 30. Numeric Range Query (How?) Let’s come back to TrieField#getRangeQuery(...) There are several options: ● field is multiValued, hasDocValues, not indexed ○ super#getRangeQuery ● field is hasDocValues, not indexed ○ new ConstantScoreQuery ( FieldCacheRangeFilter.newIntRange(...) ) ● otherwise ta-da ○ NumericRangeQuery.newIntRange(...)
31. 31. Numeric Range Query (How?) NumericRangeQuery extends MultiTermQuery which is: An abstract Query that matches documents containing a subset of terms provided by a FilteredTermsEnum enumeration. This query cannot be used directly(abstract); you must subclass it and define getTermsEnum(Terms, AttributeSource) to provide a FilteredTermsEnum that iterates through the terms to be matched.
32. 32. Numeric Range Query (How?) Let’s understand how #getTermsEnum works. Returns new NumericRangeTermsEnum(...) The main part is: NumericUtils.splitIntRange(...)
33. 33. Numeric Range Query (How?) Algorithm uses binary masks very much: for (int shift=0; noRanges(); shift += precisionStep): diff = 1L << (shift + precisionStep); mask = ((1L << precisionStep) - 1L) << shift; diff=2 0 0 1 1 2 3 Diff is distance between upper level neighbors Mask is to check if currentLevel node has nodes lower or upper. (1,3 hasLower, 0,2 hasUpper)
35. 35. Numeric Range Query (How?) hasLower = (minBound & mask) != 0L; hasUpper = (maxBound & mask) != mask; nextMinBound = (hasLower ? (minBound + diff) : minBound) & ~mask; nextMaxBound = (hasUpper ? (maxBound - diff) : maxBound) & ~mask;
36. 36. Numeric Range Query (How?) // If we are in the lowest precision or the next precision is not available. addRange(builder, valSize, minBound, maxBound, shift); // exit the split recursion loop (FOR)
37. 37. Numeric Range Query (How?) ● ● ● ● ● shift = 0 diff = 0b00000010 = 2 mask = 0b00000001 = 1 hasLower = (3 & 1 != 0)? = true hasUpper = (12 & 1 != 1)? = true ○ addRange 3..(3 | 1) = 3..3 ○ addRange 12..(12 & ~1) = 12..12 ● nextMin = (3 + 2) & ~1 = 4 ● nextMax = (12 - 2) & ~1 = 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13
38. 38. Numeric Range Query (How?) ● ● ● ● ● ● ● ● min:4; max:10 shift = 1 diff = 0b00000100 = 4 mask = 0b00000010 = 2 hasLower = (4 & 2 != 0) ? = false hasUpper = (10 & 2 != 2) ? = false nextMin = min nextMax = max 0 0 1 1 2 3 2 3 4 5 6 4 7 8 5 9 10 6 11 12 13
39. 39. Numeric Range Query (How?) ● ● ● ● ● ● ● ● min:4; max:10 shift = 2 diff = 0b00001000 = 8 mask = 0b00000100 = 4 hasLower = (4 & 4 != 0) ? = true hasUpper = (10 & 4 != 4) ? = true nextMin = (4 + 8) & ~4 = 8 => min > max END nextMax = (10 - 8) & ~4 = 0 => range 1..2 shift = 2 2 3 0 1 0 0 1 1 2 3 2 3 4 5 6 4 7 8 5 9 10 6 11 12 13
40. 40. Numeric Range Query (How?) TestNumericUtils#testSplitIntRange assertIntRangeSplit(lower, upper, precisionStep, expectBounds, shifts) assertIntRangeSplit(3, 12, 1, true, Arrays.asList( -2147483645,-2147483645, // 3,3 -2147483636,-2147483636, // 12,12 536870913, 536870914), // 1, 2 for shift == 2 Arrays.asList(0, 0, 2) ); // Crappy unsigned int conversions are done in the asserts
41. 41. Numeric Range Query (How?) So, NumericTermsEnum generates and remembers all ranges to match.
42. 42. Numeric Range Query (How?) Basically TermsEnum is an Iterator to seek or step through terms in some order. In our case order is: 0 1 2 3 4 5 6 7 8 9 10 11 12 Then (shift = 1): 0 1 2 3 4 5 6 Then (shift = 2) 0 2 1 ... 3 13
43. 43. Numeric Range Query (How?) Actually we have FilteredTermsEnum: 1. Only red terms are accepted by our enumerator 2. If term is not accepted we advance: FilteredTermsEnum#nextSeekTerm(currentTerm) TermsEnum#seekCeil(termToSeek) Seek term depends on currentTerm and generated ranges.
44. 44. Numeric Range Query (How?) Ok, now we have TermsEnum for MiltiTermQuery and enum is able to seek through only those terms which match appropriate sub ranges. The question is how to convert TermsEnum to Query!?
45. 45. Numeric Range Query (How?) The last trick is query#rewrite() method of MultiTermQuery (rewrite is always called on query before performing search): public final Query rewrite(IndexReader reader) { return rewriteMethod.rewrite(reader, this); } Oh, “rewriteMethod” how interesting… It defines how the query is rewritten.
46. 46. Numeric Range Query (How?) There are plenty of different rewrite methods, but most interesting for us are: ● CONSTANT_SCORE_* ○ BOOLEAN_QUERY_REWRITE ○ FILTER_REWRITE ○ AUTO_REWRITE_DEFAULT
47. 47. Numeric Range Query (How?) BOOLEAN_QUERY_REWRITE 1. Collect terms (TermCollector) by using #getTermsEnum(...) 2. For each term create TermQuery 3. return BooleanQuery with all TermQuery as leafs
48. 48. Numeric Range Query (How?) FILTER_REWRITE 1. 2. 3. 4. 5. Get termsEnum by using #getTermsEnum(...) Create FixedBitSet Get DocsEnum for each term Iterate over docs and bitSet.set(docid); return ConstantScoreQuery over filter (bitSet)
49. 49. Numeric Range Query (How?) AUTO_REWRITE_DEFAULT If the number of documents to be visited in the postings exceeds some percentage of the maxDoc() for the index then FILTER_REWRITE is used, otherwise BOOLEAN_REWRITE is used.
50. 50. Agenda: ● .. ● I promised. Precision Step! ● ...
51. 51. Precision step So, what is precision step and how it affects performance? ● Defines how much terms to index for each value ○ Lower step values mean more precisions and consequently more terms in index ○ indexedTermsPerValue = bitsPerVal / pStep ○ Lower precision terms are non unique, so term dictionary doesn’t grow much, however postings file does
52. 52. Precision step So, what is precision step and how it affects performance? ● ... ○ Smaller precision step means less number of terms to match, which optimizes query speed ○ But more terms to seek in index ○ You can index with a lower precision step value and test search speed using a multiple of the original step value. ○ Ideal step is found by testing only
53. 53. Precision step (Results) According to NumericRangeQuery javadoc: ● Opteron64 machine, Java 1.5, 8 bit precision step ● 500k docs index ● TermRangeQuery in BooleanRewriteMode took about 30-40 seconds ● TermRangeQuery in FilterRewriteMode took about 5 seconds ● NumericRangeQuery took < 100ms
54. 54. Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
55. 55. Useful links ● http://searchhub.org/2009/05/13/exploringlucene-and-solrs-trierange-capabilities/ ● http://www.panfmp.org/ ● http://epic.awi.de/17813/1/Sch2007br.pdf ● http://lucene.apache. org/core/4_3_1/core/org/apache/lucene/search/ NumericRangeQuery.html ● http://en.wikipedia.org/wiki/Range_tree ● me http://plus.google.com/+VadimKirilchuk