1. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Masashi Imano
SGI OpenFOAM Cloud
Benchmark
Part 3
1

2. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Test cluster conditions
• Hardware: SGI Altix ICE8200
• CPU: Intel Xeon X5365 3.00GHz 4cores x 2
• Node: 16 nodes
• Memory: 16GB/node (No swap memory)
• Interconnect: InfiniBand DDR
• OS: SUSE LINUX ENTERPRISE SERVER 10.3
• MPI:
✓ SGI MPT (1.2.6)
✓ OpenMPI
(ThirdParty package distributed with OpenFOAM)
2

3. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
• OpenFOAM versions: 2.1.x
• MPI (interconnect: InfiniBand):
✓ OpenMPI (Ver. 1.5.3), mpirun options: --mca btl openib,self,sm
✓ SGI MPT (Ver.1.2.6)
• Number of InfiniBand rails: 1, 2 (export MPI_IB_RAILS=1 or 2)
• Benchmark problem
✓ Flow: Lid-driven 2D cavity(tutorials), 3D cavity
✓ Re: 100 (laminar)
✓ solver: icoFoam (piso)
✓ Linear solver: p(AMG), U(PBiCG)
✓ Mesh:
• 2D (Δt=10-6 s, 100 steps): 1000 2, 2000 2, 4000 2
• 3D (Δt=10-5 s, 50 steps): 100 3, 200 3
x
Ux = 1 m/s
d = 0.1 m
y
Calculation conditions
3

4. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
75
150
225
300
1 2 4 8 16
113.3
155.8
204.4
102.3
54.5 52.3
71.2
135.9
101.9
54.8
16.3
29.1
61.2
89.4
53.5
NumberofCases/Day
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
2D cavity, mesh:10002 (1 Million)
4

5. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
0.2
0.4
0.6
0.8
1
1 2 4 8 16
Parallelefficiency
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
2D cavity, mesh:10002 (1 Million)
5

6. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
18
35
53
70
1 2 4 8 16
63.0
43.4
35.3
17.7
8.7
33.0
29.0
33.6
17.7
8.8 10.812.4
21.1
16.5
8.5
NumberofCases/Day
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
2D cavity, mesh:20002 (4 Millions)
6

7. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
0.2
0.4
0.6
0.8
1
1.2
1 2 4 8 16
Parallelefficiency
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
2D cavity, mesh:20002 (4 Millions)
7

8. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
5
10
15
20
2 4 8 16
20.0
9.8
6.1
3.4
12.9
8.9
6.1
3.5
5.75.15.2
3.3
NumberofCases/Day
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
2D cavity, mesh:40002 (16 Millions)
8

9. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
0.2
0.4
0.6
0.8
1
2 4 8 16
Parallelefficiency(base:2nodes)
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
2D cavity, mesh:40002 (16 Millions)
9

10. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
225
450
675
900
1 2 4 8 16
531.1
835.8
675.2
456.2
203.4
264.8
488.7
421.3420.3
203.2 197.1
403.4
434.4
389.5
194.1
NumberofCases/Day
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
3D cavity, mesh:1003 (1 Million)
10

11. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
0.2
0.4
0.6
0.8
1
1.2
1 2 4 8 16
Parallelefficiency
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
3D cavity, mesh:1003 (1 Million)
11

12. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
38
75
113
150
1 2 4 8 16
142.9
131.9
72.2
29.9
15.5
89.9
107.4
66.5
29.7
15.5
78.8
94.6
62.4
28.9
15.2
NumberofCases/Day
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
3D cavity, mesh:2003 (8 Millions)
12

13. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
0.2
0.4
0.6
0.8
1
1.2
1 2 4 8 16
Parallelefficiency
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
3D cavity, mesh:2003 (8 Millions)
13

14. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
15
30
45
60
4 8 16
58.5
38.1
11.0
42.4
34.8
10.6
38.1
32.7
10.3
NumberofCases/Day
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
3D cavity, mesh:3003 (27 Millions)
14

15. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Benchmark results
0
0.5
1
1.5
2
4 8 16
Parallelefficiency(base:4nodes)
Number of Nodes
OpenMPI SGI MPT(1 Rail) SGI MPT(2 Rails)
3D cavity, mesh:2003 (8 Millions)
15

16. Aug. 11th 2012 OpenFOAM Study Meeting for beginner @ Kansai Masashi Imano (OCAEL Co. Ltd.)
Any Questions?
16