1. 1Berlin | 07/02/2012 | zanox | Presentation title

2. TABLE OF CONTENTS
1. ABOUT ZANOX AND ME
2. REPORTING IN GENERAL
3. ANALYTICS
4. QUESTIONS
2Berlin | 08/05/2013 | zanox | Zanox Reporting Systems

3. THE ZANOX NETWORK
3Berlin | 08/05/2013 | zanox | Zanox Reporting Systems

4. WHO AM I?
4
● Senior Architect at Zanox
● 7+ years experience in building distributed search systems based on Lucene
Lucid Certified Apache Solr/Lucene Developer
● 5+ years of using Hadoop and 2+ years of using HBase for data mining
Cloudera Certified Developer for Apache Hadoop and HBase
● I have applied different machine-learning techniques mainly to optimise
resource usage while performing distributed search during my PhD
● See my book: “Beyond Centralised Search Engines
An Agent-Based Filtering Framework”
● How can you reach me?
● dragan.milosevic@zanox.com
● http://www.linkedin.com/in/draganmilosevic
Berlin | 08/05/2013 | zanox | Zanox Reporting Systems

5. 1. ABOUT ZANOX AND ME
2. REPORTING IN GENERAL
3. ANALYTICS
4. QUESTIONS
5Berlin | 08/05/2013 | zanox | Zanox Reporting Systems

6. REPORTING IN GENERAL
6
Tracking
Custom Map Reduce Distributed Reporting
M
S
S
S
JBoss
S
S
S
S
S
S
S
S
S
M
S
S
S
S
S
S
S
S
S
S
S
S
M
S
S
S
S
S
S
S
S
S
S
S
S
C
C
C
C
C
C
C
D
D
L
G
Y
D
D
L
F
M
T
T
T
T
T
T
T
T
O
O
O
D
D
D
D
IIIII
IIII
III
IIIIIIII
IIIIIIIIII
IIIIII
DD
D
DD
D
D
D D
D
D D
D D
Q
QQQQQ
R
R
R
R
R
R
D D
D D
D
D
D
D
DD
D
D
D
D
D
D D
D D
D D
D D
D D
D D
D D
OKOK
00:0003:0006:0009:0012:0015:0018:0021:0000:0001:0002:0003:0004:0005:0006:00
D D
D D
D D
D D
D D
D D
D D
D D
D D
00:00:00.00000:00:00.00500:00:00.01000:00:00.01500:00:00.02000:00:00.02500:00:00.03000:00:00.03500:00:00.04000:00:00.04500:00:00.05000:00:00.05500:00:00.06000:00:00.06500:00:00.07000:00:00.07500:00:00.08000:00:00.08500:00:00.09000:00:00.09500:00:00.10000:00:00.20000:00:00.30000:00:00.40000:00:00.50000:00:00.60000:00:00.65000:00:00.70000:00:00.75000:00:00.80000:00:00.85000:00:00.90000:00:00.91000:00:00.92000:00:00.93000:00:00.94000:00:00.95000:00:0000:02:0000:04:0000:06:0000:08:0000:10:0000:10:2000:10:4000:11:0000:11:2000:11:4000:12:0000:14:0000:16:0000:18:0000:20:0000:22:0000:24:0000:26:0000:28:00
Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
Name Searcher
Role Knows how to build reports for queries
Able to load assigned indexes from HDFS
Design RPC Server
Query optimiser
Lucene searcher with MMap directories
Report aggregation component
Name Merger
Role Knows how to build sub-queries for searchers
Able to assign indexes to searchers
Aware of available searchers
Design RPC Server
Report aggregation component
Sorting and master-data management
Name Observer
Role Checks the health of mergers and searchers
Automatic deployment of clusters
Repair unhealthy mergers and searchers
Provide information about observed clusters
Design RPC Server
Name Client
Role Keep track of available mergers
Rerouting of failed queries
Take care of load when assigning queries
Design JBoss application
RPC client for communication with mergers
Name Tracking Server
Role Tracks events such as sales, clicks, views …
Writes tracked events to HDFS
…
Design Proprietary application
Name Database server
FTP server
Log server
Role Provides data needed for reporting
…
Design …
Name Google
Yahoo
Miva
Role Provides search engine cost data
…
Design …
Data-Import Data producers are pushing data during a whole day
MR Jobs Reporting jobs are started at midnight
It takes around 6 hours to process data from yesterday
Index loading normally lasts no more that 30 minutes
Reporting There are multiple search clusters for redundancy
Every cluster has one merger that coordinates searchers
The health of every cluster is ensured by observer
Searchers are grouped based on their capabilities
Within 100ms queries are propagated to best searchers
Capable to provide reports in sub-second time
ANALYTICS

7. 1. ABOUT ZANOX AND ME
2. REPORTING IN GENERAL
3. ANALYTICS
4. QUESTIONS
7Berlin | 08/05/2013 | zanox | Zanox Reporting Systems

8. Searching
2012-09-03
2012-09-03
2012-09-03
2012-09-03
2012-09-04
2012-09-04
2012-09-04
2012-09-04
ANALYTICS
8Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
Date PublisherAdvertiser Keyword Commission
JimHUK Auto insurance company 23.56 €
TomDEVK Health insurance 9.24 €
BobDEVK Health insurance 43.81 €
BillHUK Cheap auto insurance 7.22 €
AlHUK Auto insurance company 43.07 €
MikeHUK Auto insurance company 29.98 €
NickDEVK Health insurance 12.03 €
PhilDEVK Cheap auto insurance 9.72 €
Amount
345.06 €
81.35 €
512.19 €
83.76 €
892.12 €
98.17 €
108.19 €
83.21 €
OptimizationSharding
Commission | Amount
23.56 | 345.06 €
9.24 | 81.35 €
43.81 | 512.19 €
7.22 | 83.76 €
43.07 | 892.12 €
29.98 | 98.17 €
12.03 | 108.19 €
9.72 | 83.21 €

9. 2012-09-03
2012-09-03
2012-09-03
2012-09-04
2012-09-04
2012-09-04
ANALYTICS
9Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
Date Advertiser Keyword Publisher – Commission | Amount
HUK Auto insurance company Jim – 23.56 | 345.06 €
DEVK Health insurance
Tom – 9.24 | 81.35 €
Bob – 43.81 | 512.19 €
HUK Cheap auto insurance Bill – 7.22 | 83.76 €
HUK Auto insurance company
Al – 43.07 | 892.12 €
Mike – 29.98 | 98.17 €
DEVK Health insurance Nick – 12.03 | 108.19 €
DEVK Cheap auto insurance Phil – 9.72 | 83.21 €
Searching OptimizationSharding

10. Searching
ANALYTICS
10Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
2012-09-03
2012-09-03
2012-09-03
2012-09-04
2012-09-04
2012-09-04
Date Advertiser Publisher – Commission | Amount
HUK Jim – 23.56 | 345.06 €
DEVK
Tom – 9.24 | 81.35 €
Bob – 43.81 | 512.19 €
HUK Bill – 7.22 | 83.76 €
HUK
Al – 43.07 | 892.12 €
Mike – 29.98 | 98.17 €
DEVK Nick – 12.03 | 108.19 €
DEVK
Keyword
Auto insurance company
Health insurance
Cheap auto insurance
Auto insurance company
Health insurance
Cheap auto insurance Phil – 9.72 | 83.21 €
Q
FILTER
GROUP BY
Jim
PUBLISHER
HUK
ADVERTISER
HUK
ADVERTISER
R
VALUES
103,83 €
COMMISSION
1419,11 €
AMOUNT
SEARCHING
AGGREGATION
LOADING
What does each
searcher when
query comes?
OptimizationSharding
Can we further
improve loading
of documents?
Yes we can!
Find out which
dimensions are
important filters
and use them
while indexing
documents

11. Searching
ANALYTICS
11Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
2012-09-03
2012-09-03
2012-09-03
2012-09-04
2012-09-04
2012-09-04
Date Advertiser Publisher – Commission | Amount
HUK Jim – 23.56 | 345.06 €
DEVK
Tom – 9.24 | 81.35 €
Bob – 43.81 | 512.19 €
HUK Bill – 7.22 | 83.76 €
HUK
Al – 43.07 | 892.12 €
Mike – 29.98 | 98.17 €
DEVK Nick – 12.03 | 108.19 €
DEVK
Keyword
Auto insurance company
Health insurance
Cheap auto insurance
Auto insurance company
Health insurance
Cheap auto insurance Phil – 9.72 | 83.21 €
Q
FILTER
GROUP BY
Jim
PUBLISHER
HUK
ADVERTISER
HUK
ADVERTISER
OptimizationSharding
R
VALUES
103,83 €
COMMISSION
1419,11 €
AMOUNT

12. Searching
ANALYTICS
12Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
● There are 9 dimensions in total, where several
● Have huge cardinality (>1 million different values) and
● At the same time are rarely used as filter, meaning
● They are excellent candidates for being put together with measures.
● There are more than 50 measures that
● Are freely combined in user-defined formulas, but at the same time
● Form only few groups based on usage patterns, meaning
● They contribute to better compression because fields become bigger.
● Ordering in which documents are inserted in index is driven by
● Dimensions that are known to be the most important filters, meaning
● Faster loading as needed documents are closer to each other.
OptimizationSharding

13. OptimizationSearching Sharding
Sharding based on hashing
ANALYTICS
13Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
SEARCHING
AGGREGATION
LOADING
SEARCHING
AGGREGATION
LOADING
SEARCHING
AGGREGATION
LOADING
SEARCHING
AGGREGATION
LOADING
SEARCHING
AGGREGATION
LOADING
Distributed Reporting
M
S
S
S
S
S
S
S
S
S
S
S
S
M
S
S
S
S
S
S
S
S
S
S
S
S O
M
S
S
S
S
S
S
S
S
S
S
S
S O
O
R
VALUES
16,55 €
COMMISSION
523,21 €
AMOUNT
R
VALUES
103,83 €
COMMISSION
1419,11 €
AMOUNT
R
VALUES
78,72 €
COMMISSION
879,34 €
AMOUNT
R
VALUES
134,23 €
COMMISSION
1622,56 €
AMOUNT
R
VALUES
336,16 €
COMMISSION
4464,09 €
AMOUNT
R R R R
Q
FILTER
GROUP BY
Jim
PUBLISHER
HUK
ADVERTISER
HUK
ADVERTISER
QQQQQ
R
VALUES
2,83 €
COMMISSION
19,87 €
AMOUNT
RSharding based on hashing Query is sent to each and every searcher
Searchers work hard to produce reports
Huge amount of data sent over network
Goals for alternative sharding Think very well while distributing data
Network is your friend and precious resource
Produce same report with less searchers

14. Sharding based on selected filters
OptimizationShardingSearching
ANALYTICS
14Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
SEARCHING
AGGREGATION
LOADING
SEARCHING
AGGREGATION
LOADING
M
S S S S S R
VALUES
336,16 €
COMMISSION
4464,09 €
AMOUNT
Q
FILTER
GROUP BY
Jim
PUBLISHER
HUK
ADVERTISER
HUK
ADVERTISER
QQ
VALUES
133,33 €
COMMISSION
2454,22 €
AMOUNT
RRR
VALUES
202,83 €
COMMISSION
2009,87 €
AMOUNT
R

15. OptimizationShardingSearching
ANALYTICS
15Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
● Sharding based on hashing randomly distributes data
● But that is not needed because no relevance will be computed.
● Queries have to be send to each and every searcher, being suboptimal
because much more data have to be transmitted over the network.
● The slowest searcher determine final response time due to completeness.
● How we partition based on filter fields?
● Order dimensions based on their usage frequency as filter in particular group.
● Greater usage frequency means more important for sharding.
● Examples of dimensions that are found to be excellent for sharding are:
time related dimensions and advertiser name dimension.
● Merger uses knowledge about which data each shard has while routing.
● Final result is more stable system and much better QPS.

16. ANALYTICS
16Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
3rdGroup2ndGroup1stGroup
Publisher
Advertiser
Keyword
Publisher
Keyword
Publisher
Advertiser
Distributed Reporting
M
S
S
S
S
S
S
S
S
S
S
S
S
M
S
S
S
S
S
S
S
S
S
S
S
S O
M
S
S
S
S
S
S
S
S
S
S
S
S O
O
OptimizationShardingSearching

17. We have created separate
searcher group for every
possible combination of
dimensions. Therefore we
are well prepared for each
query that can hit us.
But we are attacked where
we haven’t thought that it
is possible. One new field
have been requested and
that has spoiled everything
17Berlin | 07/02/2012 | zanox | Presentation title
3rdGroup2ndGroup1stGroup
Publisher
Advertiser
Keyword
Publisher
Keyword
Publisher
Advertiser
M
4thGroup
Publisher
5thGroup
Advertiser
6thGroup
Advertiser
Keyword
7thGroup
Keyword
Q
FILTER
GROUP BY
Jim
PUBLISHER
Keyword
HUK
ADVERTISER
Insurance
KEYWORD
Website
8thGroup
Advertiser
Website
Keyword
9thGroup
Website
Advertiser
Publisher
10thGroup
Website
11thGroup
Website
Keyword
12thGroup
Website
Advertiser
13thGroup
Website
Publisher
Publisher
There are so
many groups
but no one
has fields
Website
Publisher
Keyword
Creating a separate searcher group for each and every possible
combination of dimensions in each query is not feasible because
number of searcher-groups grows extremely fast (exponentially)
Number of dimensions Number of searcher groups
3 23 – 1 = 7
4 24 – 1 = 15
… …
9 29 – 1 = 511
Solution is to analyze already processed queries to discard low
performing groups and to create new ones for emerging queries
ANALYTICS
OptimizationShardingSearching

18. OptimizationShardingSearching
18Berlin | 07/02/2012 | zanox | Presentation title
3rdGroup2ndGroup1stGroup
Publisher
Advertiser
Keyword
Publisher
Keyword
Publisher
Advertiser
4thGroup
Publisher
5thGroup
Advertiser
6thGroup
Advertiser
Keyword
7thGroup
Keyword
8thGroup
Advertiser
Website
Keyword
9thGroup
Website
Advertiser
Publisher
10thGroup
Website
11thGroup
Website
Keyword
12thGroup
Website
Advertiser
13thGroup
Website
Publisher
queries
451236
response
45.54 ms
queries
9
response
15.54 ms
queries
221128
response
82.87 ms
queries
2876136
response
23.54 ms
queries
236
response
92.33 ms
queries
2
response
147.82 ms
queries
228732
response
61.03 ms
queries
89
response
215.04 ms
queries
15
response
105.02 ms
queries
5644
response
29.76 ms
queries
4
response
563.04 ms
queries
32
response
87.03 ms
queries
543765
response
35.87 ms
14thGroup
Website
Publisher
Keyword
M
Q
FILTER
GROUP BY
Jim
PUBLISHER
Keyword
HUK
ADVERTISER
Insurance
KEYWORD
Website
Publisher
There are so
many groups
but no one
has fields
Website
Publisher
Keyword
ANALYTICS

19. 19Berlin | 07/02/2012 | zanox | Presentation title
3rdGroup9thGroup1stGroup
Publisher
Advertiser
Keyword
Website
Publisher
Publisher
Advertiser
4thGroup
Publisher
11thGroup
Website
Keyword
queries
20236
response
45.54 ms
Advertiser
queries
91234
response
15.54 ms
queries
221128
response
82.87 ms
queries
286136
response
23.54 ms
queries
22336
response
92.33 ms
14thGroup
Website
Publisher
Keyword
queries
32546
response
727.23 ms
M
Q
FILTER
GROUP BY
Jim
PUBLISHER
HUK
ADVERTISER
Insurance
KEYWORD
Website
Publisher
15thGroup
Keyword
Publisher
Fields
Website
Publisher
Keyword
queries
9874
response
129.14 ms
Publisher
Keyword
queries
22678
response
989.28 msFields
=
We optimized system
by adding specialized
group that is excellent
for found slow queries
There is no
group that is
optimized
for queries
having fields
Publisher
Keyword
ANALYTICS
OptimizationShardingSearching

20. ANALYTICS
20Berlin | 08/05/2013 | zanox | Zanox Reporting Systems
● Hadoop makes possible to easily build new indexes:
● Optimize utilization of Hadoop resources because
● Better indexes can be always build by low priority jobs.
● Built OLAP continuously adapts itself to queries and
● There are no more expensive queries in long run.
● How can we decide which group is best of query at hand?
● That’s query routing, which we solved either by
● Asking searchers only to return number of hits and
then to generate report if being selected, or simpler
● By using heuristic based on dimension cardinality.
● But query routing is not a topic of this talk.
OptimizationShardingSearching

21. 1. ABOUT ZANOX AND ME
2. REPORTING IN GENERAL
3. QUERY ROUTING
4. QUESTIONS
21Berlin | 08/05/2013 | zanox | Zanox Reporting Systems

22. 22Berlin | 08/05/2013 | zanox | Zanox Reporting Systems