1. Large Scale Math with
Hadoop MapReduce
Tsz-Wo (Nicholas) Sze, PhD
Hadoop Summit
June 29, 2011
1
2. Who am I?
• Hortonworks Software Engineer
• Apache Hadoop PMC Member
• Mathematician
Interests:
Distributed Computing
Algorithms
Number Theory
2
3. Agenda
• Introduction
• Integer Multiplication
• MapReduce-FFT
• MapReduce-Sum
• MapReduce-SSA
• A New World Record
• The “Machine” Behind the Computation
Tsz-Wo Sze, Hadoop Summit 2011 3
4. Agenda
• Introduction
• Integer Multiplication
• MapReduce-FFT
• MapReduce-Sum
• MapReduce-SSA
• A New World Record
• The “Machine” Behind the Computation
Tsz-Wo Sze, Hadoop Summit 2011 4
5. Typical Hadoop Applications
Major applications of Hadoop include
• Search and crawling
• Text processing
• Machine learning
• ...
Tsz-Wo Sze, Hadoop Summit 2011 5
6. Typical Hadoop Applications
Major applications of Hadoop include
• Search and crawling
• Text processing
• Machine learning
• ...
But not yet commonly used in scientific
or mathematical applications.
Why?
Tsz-Wo Sze, Hadoop Summit 2011 6
7. Why Not Math?
No MapReduce math libraries available, and
More fundamentally,
MapReduce math algorithms are not well studied.
Tsz-Wo Sze, Hadoop Summit 2011 7
8. Existing Library
Really no MapReduce Math Library?
Not exactly.
Tsz-Wo Sze, Hadoop Summit 2011 8
9. Existing Library
Really no MapReduce Math Library?
Not exactly.
Apache Mahout
• A machine learning library.
• Includes packages for matrix operations.
Tsz-Wo Sze, Hadoop Summit 2011 9
10. Existing Library
Really no MapReduce Math Library?
Not exactly.
Apache Mahout
• A machine learning library.
• Includes packages for matrix operations.
Apache Hama (Incubation)
• A matrix computational package.
Tsz-Wo Sze, Hadoop Summit 2011 10
11. Computational Intensive Problems (1)
Integer Factoring
• a.k.a. breaking RSA cryptosystem
Given N , e and c, compute m such that
e
c ≡ m (mod N ),
where N is a product of two primes.
• a 768-bit RSA modulus was factored1 in 2009
1
Kleinjung et al., Factorization of a 768-bit RSA modulus, CRYPTO 2010.
Tsz-Wo Sze, Hadoop Summit 2011 11
13. Computational Intensive Problems (3)
Finding complex zeros of Riemann Zeta function
∞
1
ζ(s) = for s ∈ C, (s) > 1
n=1
ns
and then analytically continued to all s = 1.
Tsz-Wo Sze, Hadoop Summit 2011 13
14. Computational Intensive Problems (3)
Finding complex zeros of Riemann Zeta function
∞
1
ζ(s) = for s ∈ C, (s) > 1
n=1
ns
and then analytically continued to all s = 1.
• Disprove Riemann Hypothesis (RH)
Then, you will get $1,000,000 dollars2.
However, RH is unlikely to be false.
2
See http://www.claymath.org/millennium/Riemann_Hypothesis/.
Tsz-Wo Sze, Hadoop Summit 2011 14
15. Computational Intensive Problems (3)
Finding complex zeros of Riemann Zeta function
∞
1
ζ(s) = for s ∈ C, (s) > 1
n=1
ns
and then analytically continued to all s = 1.
• Disprove Riemann Hypothesis (RH)
Then, you will get $1,000,000 dollars.
However, RH is unlikely to be false.
• More likely:
Obtain more evidents which support RH.
Tsz-Wo Sze, Hadoop Summit 2011 15
16. Computational Intensive Problems (4)
Computing π
Latest world records:
• Five trillion decimal digits (August 2010)
by Alexander Yee & Shigeru Kondo3
3
See http://www.numberworld.org/misc_runs/pi-5t/announce_en.html
Tsz-Wo Sze, Hadoop Summit 2011 16
17. Computational Intensive Problems (4)
Computing π
Latest world records:
• Five trillion decimal digits (August 2010)
by Alexander Yee & Shigeru Kondo
• The two quadrillionth bits (July 2010)
by Tsz-Wo Sze &
the Yahoo! Cloud Computing Team4
4
See http://developer.yahoo.net/blogs/hadoop/2010/09/two_quadrillionth_bit_pi.html
Tsz-Wo Sze, Hadoop Summit 2011 17
18. Missing Functionalities
Fast Fourier Transform (FFT)
– the basic rountine behind many algorithms.
Arbitrary Precision Arithmetic
Integer functions
Floating-point functions
Complex functions
...
Tsz-Wo Sze, Hadoop Summit 2011 18
19. Agenda
• Introduction
• Integer Multiplication
• MapReduce-FFT
• MapReduce-Sum
• MapReduce-SSA
• A New World Record
• The “Machine” Behind the Computation
Tsz-Wo Sze, Hadoop Summit 2011 19
20. Why Integer Multiplication?
There exist fast algorithms.
Many applications
• Division
• Logarithm
• Trigonometric functions
• ...
Tsz-Wo Sze, Hadoop Summit 2011 20
21. Prerequisite of Algorithms
D.J. Bernstein, Fast
multiplication and its
applications, ANTS 2008.
Tsz-Wo Sze, Hadoop Summit 2011 21
22. Integer Multiplication Algorithms
Na¨ O(N 2)
ıve,
Karatsuba, O(N log2 3) = O(N 1.585)
Toom-Cook, O(N log(2D−1)/ log D )
If D = 3, then O(N log 5/ log 3) = O(N 1.465)
FFT-based algorithms O(N log N · · · )
Tsz-Wo Sze, Hadoop Summit 2011 22
23. FFT-based Algorithms
Basic FFT, O(N log N log log N log log log N · · · )
Sch¨nhage-Strassen, O(N log N log log N )
o
Nussbaumer, O(N log N log log N )
log∗ N
F¨rer, O(N (log N )2
u )
log∗ N
De-Kurur-Saha-Saptharishi, O(N (log N )2 )
Tsz-Wo Sze, Hadoop Summit 2011 23
24. Convolution
By the convolution theorem,
a × b = dft−1(dft(a) ∗ dft(b)),
where
× denotes the convolution operator ,
∗ denotes componentwise multiplication,
dft( · ) denotes discrete Fourier transform.
Tsz-Wo Sze, Hadoop Summit 2011 24
25. Sch¨nhage-Strassen Algorithm
o
(SSA)
Represent integers as polynomials. Then, com-
pute convolution with DFTs modulo an integer5.
5
It has the form 2n + 1 and is called the Sch¨nhage-Strassen modulas.
o
Tsz-Wo Sze, Hadoop Summit 2011 25
26. SSA Steps
Step 1: two DFTs,
def ˆ def dft(b);
ˆ
a = dft(a) and b=
Step 2: componentwise multiplication,
def
ˆ ˆ
p = a ∗ b;
ˆ
Step 3: a DFT inverse,
−1
p = dft (ˆ );
p
Step 4: normalization.
Tsz-Wo Sze, Hadoop Summit 2011 26
27. Calculating DFTs
DFT can be calculated by a family of algorithms
called Fast Fourier Transform (FFT).
Tsz-Wo Sze, Hadoop Summit 2011 27
29. Data Model (1)
Need a data model which allows accessing
terabit integers efficiently.
An integer x is represented as a D-dimensional
tuple
x = (xD−1, xD−2, . . . , x0).
Tsz-Wo Sze, Hadoop Summit 2011 29
30. Data Model (2)
Write
D = IJ.
where I and J are powers of two.
Define J-dimensional tuples
(i) def
x = (x(J−1)I+i, x(J−2)I+i, . . . , xi)
for 0 ≤ i < I.
Tsz-Wo Sze, Hadoop Summit 2011 30
31. Data Model (3)
Then,
x(0) x(J−1)I x(J−2)I . . . x0
(1)
x x(J−1)I+1 x(J−2)I+1 . . . x1
. = . . ... .
. . . .
x(I−1) x(J−1)I+(I−1) x(J−2)I+(I−1) . . . xI−1
We call it the (I, J)-format of x.
Tsz-Wo Sze, Hadoop Summit 2011 31
32. Data Model (4)
Each x(i) is a sequence of J records.
Each record is a key-value pair.
Record # <Key, Value>
0 < i, xi >
1 < J + i, xJ+i >
.
. .
.
J −1 < (J − 1)I + i, x(J−1)I+i >
Tsz-Wo Sze, Hadoop Summit 2011 32
33. Data Model (5)
Thus, an integer is stored as I SequenceFiles in
HDFS, each SequenceFile contains J records.
Tsz-Wo Sze, Hadoop Summit 2011 33
34. Parallel-FFT Steps
Step 1: I inner DFTs with J-point,
a(i) = dft(a(i));
Step 2: componentwise shifting,
def
zjI+i = ζ ij a(i)j ;
Step 3: transposition,
[j] def
z = (zjI+(I−1), zjI+(I−2), . . . , zjI );
Step 4: J outer DFTs with I-point,
[j] def
z = dft(z[j]).
Tsz-Wo Sze, Hadoop Summit 2011 34
37. Data Locality
The FFT transposition, which is traditionally dif-
ficult in preserving locality, becomes trivial in
MapReduce.
Tsz-Wo Sze, Hadoop Summit 2011 37
38. MapReduce-FFT (1)
Map function:
(k1, v1) −→ list k2, v2
Algorithm 1 (Forward FFT, Mapper).
(f.m.1) read key i, value a(i);
(f.m.2) calculate a J-point DFT;
(f.m.3) componentwise multiply;
(f.m.4) for 0 ≤ j < J, emit key j, value (i, zjI+i).
Tsz-Wo Sze, Hadoop Summit 2011 38
39. MapReduce-FFT (2)
Reduce function:
(k2, list v2 ) −→ list k3, v3 .
Algorithm 2 (Forward FFT, Reducer).
(f.r.1) receive key j, list [(i, zjI+i)]0≤i<I ;
(f.r.2) calculate an I-point DFT;
(f.r.3) write key j, value z[j].
Tsz-Wo Sze, Hadoop Summit 2011 39
40. Normalization
Normalization can be viewed as a summation of
three integers.
Tsz-Wo Sze, Hadoop Summit 2011 40
41. Summation
Integer summation can be done by (1) componen-
twise summation, (2) carry evaluation and then
(3) parallel carrying.
Tsz-Wo Sze, Hadoop Summit 2011 41
46. MapReduce-SSA
two concurrent forward FFT jobs;
a backward FFT job with componentwise
multiplication and splitting ;
a componentwise summation map-only job;
a carrying job6.
6
It is possible to combine the last two jobs if we modify the shuffle process in MapReduce [.next].
Tsz-Wo Sze, Hadoop Summit 2011 46
48. Cluster Configuration
A shared cluster:
Apache Hadoop 0.20
1350 nodes
6 GB memory per node
2 map tasks & 1 reduce task per node
Imposed a limitation on the aggregated
memory usage of individual jobs.
Tsz-Wo Sze, Hadoop Summit 2011 48
49. Running Time
Actual running time for 236 ≤ N ≤ 240.
11.5
t is the elapsed time in seconds
11
10.5
10
9.5
log(t)
9
8.5
8
7.5
7
32 33 34 35 36 37 38 39 40
log(N)
Tsz-Wo Sze, Hadoop Summit 2011 49
50. Agenda
• Introduction
• Integer Multiplication
• MapReduce-FFT
• MapReduce-Sum
• MapReduce-SSA
• A New World Record
• The “Machine” Behind the Computation
Tsz-Wo Sze, Hadoop Summit 2011 50
51. What is π?
π is a mathematical
constant such that,
for any circle,
circumference C
π= = .
diameter d
Tsz-Wo Sze, Hadoop Summit 2011 51
52. What is π?
π is a mathematical
constant such that,
for any circle,
circumference C
π= = .
diameter d
We have π = 3.244
Tsz-Wo Sze, Hadoop Summit 2011 52
53. What is π?
π is a mathematical
constant such that,
for any circle,
circumference C
π= = .
diameter d
We have π = 3.244
(in hexadecimal )
Tsz-Wo Sze, Hadoop Summit 2011 53
54. Decimal, Hexadecimal & Binary
Representing π in different bases
π = 3.1415926535 8979323846 2643383279 ...
= 3.243F6A88 85A308D3 13198A2E ...
= 11.00100100 00111111 01101010 ...
Bit position is counted after the radix point.
e.g., the eight bits starting at the ninth bit position
are 00111111 in binary or 3F in hexadecimal.
Tsz-Wo Sze, Hadoop Summit 2011 54
55. A New World Record
Yahoo! Cloud Computing (July 2010)
• Machines: Idle slices of 1000-node clusters
Each node has two quad-core 1.8-2.5 GHz CPUs
• Duration: 23 days
• CPU time: 503 years
• Verification: 582 years CPU time
Tsz-Wo Sze, Hadoop Summit 2011 55
56. A New World Record
Bit values (in hexadecimal)
0E6C1294 AED40403 F56D2D76 4026265B
CA98511D 0FCFFAA1 0F4D28B1 BB5392B8
Tsz-Wo Sze, Hadoop Summit 2011 56
57. A New World Record
Bit values (in hexadecimal)
0E6C1294 AED40403 F56D2D76 4026265B
CA98511D 0FCFFAA1 0F4D28B1 BB5392B8
(256 bits)
The first bit position: 1,999,999,999,999,997 (= 2 · 1015 − 3)
The last bit position: 2,000,000,000,000,252 (= 2·1015 +252)
The two quadrillionth (2 · 1015th) bit is 0.
Tsz-Wo Sze, Hadoop Summit 2011 57
58. BBC News (16 Sep 2010)
Pi record smashed as team finds two-quadrillionth digit
http://www.bbc.co.uk/news/technology-11313194
Tsz-Wo Sze, Hadoop Summit 2011 58
59. NewScientist (17 Sep 2010)
New pi record exploits Yahoo’s computers
http://www.newscientist.com/article/dn19465-new-pi-record-exploits-yahoos-com
html
Tsz-Wo Sze, Hadoop Summit 2011 59
60. Other News Coverage
New Pi Record Exploits Yahoo’s Computers
http://cacm.acm.org/news/99207-new-pi-record-exploits-yahoos-computers
The Yahoo! boffin scores pi’s two
quadrillionth bit
http://www.theregister.co.uk/2010/09/16/pi_record_at_yahoo
Pi calculation more than doubles old record
http://www.radionz.co.nz/news/world/57128/pi-calculation-more-than-doubles-ol
Hadoop used to calculate Pi’s two quadrillionth bit
http://www.zdnet.co.uk/blogs/mapping-babel-10017967/hadoop-used-to-calculate-
Tsz-Wo Sze, Hadoop Summit 2011 60
61. Yahoo! researcher breaks Pi record in finding
the two-quadrillionth digit
http://www.engadget.com/2010/09/17/yahoo-researcher-breaks-pi-record-in-findi
Nicholas Sze of Yahoo Finds Two-Quadrillionth
Digit of Pi
http://science.slashdot.org/story/10/09/16/2155227/Nicholas-Sze-of-Yahoo-Find
The 2,000,000,000,000,000th digit of the mathemat-
ical constant pi discovered
http://news.gather.com/viewArticle.action?articleId=281474978525563
Researcher Shatters Pi Record by Finding
Two-Quadrillionth Digit
http://www.maximumpc.com/article/news/researcher_shatters_pi_record_finding_
two-quadrillionth_digit
Tsz-Wo Sze, Hadoop Summit 2011 61
62. A bigger slice of pi
http://radar.oreilly.com/2010/09/strata-week-grabbing-a-slice.html
2 Quadrillionth digit of PI is found: Scientist
celebration in worldwide Pandemonium
http://engforum.pravda.ru/showthread.php?296242-2-Quadrillionth-digit-of-PI-i
And the number is...0
http://www.hexus.net/content/item.php?item=26505
Pi Record Smashed as Team Finds Two-
Quadrillionth Digit
http://hardocp.com/news/2010/09/16/pi_record_smashed_as_team_finds_twoquadril
digit
Tsz-Wo Sze, Hadoop Summit 2011 62
63. Yahoo Engineer Calculates Two Quadrillionth
Bit Of Pi
http://www.webpronews.com/topnews/2010/09/17/yahoo-engineer-calculates-two-qu
A Cloud Computing Milestone: Yahoo!
Reaches the 2 Quadrillionth Bit of Pi
http://www.readwriteweb.com/cloud/2010/09/a-cloud-computing-milestone-ya.
php
Yahoo researcher Nicolas Sze determines
the 2,000,000,000,000,000th digit of the mathematical con-
stant pi
http://www.thaindian.com/newsportal/sci-tech/yahoo-researcher-nicolas-sze-det
100430278.html
...
Tsz-Wo Sze, Hadoop Summit 2011 63
64. Computing π
How to compute the nth bits of π?
Tsz-Wo Sze, Hadoop Summit 2011 64
65. Computing π
How to compute the nth bits of π?
Let’s ignore this question in this talk ...
and focus on:
Tsz-Wo Sze, Hadoop Summit 2011 65
66. Computing π
How to compute the nth bits of π?
Let’s ignore this question in this talk ...
and focus on:
How to execute such huge computation?
Tsz-Wo Sze, Hadoop Summit 2011 66
67. Map- & Reduce-side Computations
Developed a generic framework to execute tasks
on either the map-side or the reduce-side.
Applications define two functions:
• partition(c, m):
partition the computation c into m parts.
• compute(c):
execute the computation c
Tsz-Wo Sze, Hadoop Summit 2011 67
68. Map-side Job
Contains multiple mappers and zero reducers
• A PartitionInputFormat partitions c
into m parts
• Each part is executed by a mapper
Tsz-Wo Sze, Hadoop Summit 2011 68
69. Reduce-side Job
Contains a mapper and multiple reducers
• A SingletonInputFormat launches
a PartitionMapper
• An Indexer launches m reducers.
Tsz-Wo Sze, Hadoop Summit 2011 69
70. Abstract Machine (1)
Machine
– an abstract base class allows abstract Runner(s)
to execute MachineComputable tasks.
Machine subclasses
• Map Side Machine
m100t3: 100 maps with 3 threads each.
• Reduce Side Machine
r50t2: 50 reduces with 2 threads each.
Tsz-Wo Sze, Hadoop Summit 2011 70
71. Abstract Machine (2)
More Machine subclasses
• Mix Machine – chooses Map-/Reduce-side
jobs according to the cluster status.
x-m200t1-r100t2-5: either launch a job with 200 maps
with 1 thread each; or a job with 100 reduces with 2 thread each.
• Alternation Machine – alternates Map-side
and Reduce-side jobs in a regular pattern.
a-m200t1-r100t2-mrr: submit a map job, then a re-
duce job, then another reduce job and repeat this pattern.
• Null Machine – does nothing for testing.
Tsz-Wo Sze, Hadoop Summit 2011 71
72. Utilizing The Idle Slices
Monitor cluster status
• Submit a map-side (or reduce-side) job if there
are sufficient available map (or reduce) slots.
Small jobs
• Hold resource only for a short period of time
Interruptible & resumable
• can be interrupted at any time by simply
killing the running jobs
Tsz-Wo Sze, Hadoop Summit 2011 72
74. The Implementation
Main programs:
DistBbp – a program to submit jobs.
DistSum – distributed summation.
Open source – available at
https://issues.apache.org/jira/browse/MAPREDUCE-1923
Tsz-Wo Sze, Hadoop Summit 2011 74
75. The World Record Computation
35,000 MapReduce jobs, each job either has:
• 200 map tasks with one thread each, or
• 100 reduce tasks with two threads each.
Each thread computes 200,000,000 terms
• ∼45 minutes.
Submit up to 60 concurrent jobs
The entire computation took:
• 23 days of real time and 503 CPU years
Tsz-Wo Sze, Hadoop Summit 2011 75
76. Referneces
• [1] Tsz-Wo Sze. Sch¨nhage-Strassen Algorithm with MapReduce for Mul-
o
tiplying Terabit Integers. Symbolic-Numeric Computation 2011, to ap-
pear. Preprint available at http://people.apache.org/~szetszwo/
ssmr20110430.pdf
• [2] Tsz-Wo Sze. The Two Quadrillionth Bit of Pi is 0! Distributed
Computation of Pi with Apache Hadoop. In IEEE 2nd International
Conference on Cloud Computing Technology and Science (CloudCom),
pages 727-732, 2010. (Earlier versions available at http://arxiv.org/
abs/1008.3171)
Tsz-Wo Sze, Hadoop Summit 2011 76