This document discusses different approaches for performing joins in Lucene/Solr, including query-time and index-time joins. It describes JoinUtil and BlockJoin for performing query-time and index-time joins respectively. Benchmark results show BlockJoin has faster search performance than JoinUtil, though it has slower reindexing. The document proposes ideas for improving join performance such as incremental index updates and eliminating term enumeration during queries.

2. Approaching Join Index
yet another one join algorithm
Mikhail Khludnev
principal engineer

3. • Grid Dynamics is a Silicon Valley-based leading provider of scalable, next-generation
commerce technology solutions
• Record of outperformance with Tier 1 retail clients
• Fortune 1000 client relationships
PRIVILEGED AND CONFIDENTIAL

4. About Me
● principal engineer at Grid Dynamics
● spoke at few last LuceneRevolutions
● contributed BlockJoin query parser for Solr - {!parent}
● blogged about it at http://blog.griddynamics.com/
● tried to fix threads at DataImportHandler
http://google.com/+MikhailKhludnev

5. You are expected to know
● how Lucene searches/filters
● how it counts facets
● that there are segments
● what is DocValues
● why to join
● RDBMS joins: nested loop join, sort-merge join and hash join.

6. I’m expected to know
● query-time join
● index-time join
● yet another one join

7. Lucene/Solr Is Strong
● searching
○ filtering
● analytics
○ facets
○ pivots
○ stats

8. Lucene/Solr Is Weak in
● robust joins
○ multiple entities
○ relations

9. Joins in General

10. Joins in General

11. Joins in General
PK=FK

12. Joins in General
PK=FK

13. Joins in General
PK=FK

14. children
Joins in General
1:M
parents

15. Executing Join Query
q

16. Executing Join Query
q

17. Executing Join Query
q

18. Executing Join Query
q
fq

19. Executing Join Query
q
fq

20. Join in General
parents ∩ join-relation ∩ children

21. Join in General
parents ∩ join-relation ∩ children
● input
● output
● enumeration order

22. JoinUtil
q
FK[doc#]
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”

23. JoinUtil
q
FK[doc#]
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”
“25”
“17”
...

24. JoinUtil
q
FK
“1” →△
…
“17”→△
“25”→△
...
“25”
“17”
...
FK[doc#]
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”

25. JoinUtil
q
FK FK[doc#]
“1” →△
…
“17”→△
“25”→△
...
“25”
“17”
...
fq
“17”
“17”
“25”
“25”
“56”
“56”
“56”
“25”
“4”
“61”
“25”

26. Block Join
doc#

27. Block Join
doc#

28. Block Join
1 doc#
0
0
1
0
0
1
0
0
1
0

29. Block Join
1 doc#
0
0
1
0
0
1
0
0
1
0
q

30. Block Join
1 doc#
0
0
1
0
0
1
0
0
1
0
q
fq

31. Block Join
1 doc#
0
0
1
0
0
1
0
0
1
0
q
fq

32. Comparison
JoinUtil BlockJoin
searching
is fast
reindexing
is fast

33. Comparison
JoinUtil BlockJoin
searching
is fast
reindexing
is fast

34. Comparison
JoinUtil BlockJoin
searching
is fast < ? <
reindexing
is fast > ? >

35. Comparison
JoinUtil BlockJoin
searching
is fast < ? <
reindexing
is fast > ? >
doc#

36. Comparison
JoinUtil BlockJoin
searching
is fast < ? <
reindexing
is fast > ? >
doc#

37. Comparison
JoinUtil BlockJoin
searching
is fast < ? <
reindexing
is fast > ? >
doc#

38. Join Index
q
doc#[doc#]
3
6
0
3
10
0
6
3

39. Join Index
q
doc#[doc#]
3
6
0
3
10
0
6
3

40. Join Index
q
doc#[doc#]
fq 3
6
0
3
10
0
6
3

41. Join Index
q
fq
doc#[doc#]
4
1
2
6
8
5 10
9

42. Join Index
q
doc#[doc#]
fq
4
1
2
6
8
5 10
9

43. Join Index
q
doc#[doc#]
fq
4
1
2
6
8
5 10
9

44. Benchmarking 2.9 M docs
Latency, ms
the bigger the worse
27
14
10
BlockJoin
(i-time)
Join Index
JoinUtil
(q-time)
https://github.com/m-khl/lucene-solr/tree/dvjoin-benchmark

45. Indexing is still a problem
doc#[doc#]
4
3
6
1
0
3
2
10
0
6
3
8
5 10
9

46. Further Plans
● incremental join-index update
● perhaps just calculate and cache it
● join in both directions
● calculate optimal execution plan of segments enumeration

47. Summary
● Joins in General
● JoinUtil vs Block-join
● {!scorejoin } - SOLR-6234
● updatable DocValues
● opportunities for improving query-time joins:
○ eliminate term enum
○ choose lower cardinality side for enumeration

48. References
● Searching relational content with Lucene's BlockJoinQuery
http://blog.mikemccandless.com/
● Solr Experience: search parent-child relations. Part I
Solr block-join support
http://blog.griddynamics.com/
● https://wiki.apache.org/solr/Join
● http://www.slideshare.net/martijnvg/document-relations
● https://issues.apache.org/jira/browse/SOLR-6234 {!scorejoin }
● Updatable DocValues Under the Hood
http://shaierera.blogspot.com/
● Subject: How to openIfChanged the most recent merge?
at: java-dev@lucene.apache.org

49. Thanks for Joining us!

50. Off scope

51. Joins’ Zoo in Lucene
True Joins
● query-time join
○ JoinUtil
○ {!join }
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join {!
parent}

52. Joins’ Zoo in Lucene
Workarounds
● term positions/SpanQueries
● FieldCollapsing/Grouping
● term decoration
○ spatial
● multivalue fields
True Joins
● query-time join
○ JoinUtil
○ {!join },
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join {!
parent}

53. Joins’ Zoo in Lucene
Workarounds
● term positions/SpanQueries
● FieldCollapsing/Grouping
● term decoration
○ spatial
● multivalue fields
True Joins
● query-time join
○ JoinUtil
○ {!join },
○ {!scorejoin } - SOLR-6234
● index-time join aka block-join {!
parent}

54. Two phase update problem
Subject: How to openIfChanged the most recent merge?
at: java-dev@lucene.apache.org

55. JoinUtil
● query-time
● indexing is fast
● searching is slow, why?
○ expensive term enum
○ single enumeration order
BlockJoin
● index-time
● reindexing whole block is as expensive as mandatory
● searching is darn fast, however
○ can’t reorder child docs