Under the grid computing paradigm, large sets of heterogeneous resources can be aggregated and shared. Grid development and acceptance hinge on proving that grids reliably support real applications, and on creating adequate benchmarks to quantify this support. However, applications of grids (and clouds) are just beginning to emerge, and traditional benchmarks have yet to prove representative in grid environments. To address this chicken-and-egg problem, we propose a middle-way approach: create and run synthetic grid workloads comprised of applications representative for today's grids (and clouds). For this purpose, we have designed and implemented GrenchMark, a framework for synthetic workload generation and submission. The framework greatly facilitates synthetic workload modeling, comes with over 35 synthetic and real applications, and is extensible and flexible. We show how the framework can be used for grid system analysis, functionality testing in grid environments, and for comparing different grid settings, and present the results obtained with GrenchMark in our multi-cluster grid, the DAS.

1. GrenchMark: A Framework for
Analyzing, Testing, and Comparing Grids
A. Iosup, D.H.J. Epema
PDS Group, ST/EWI, TU Delft
April 4, 2012
CCGrid 2006 1

2. Outline
• Introduction and Motivation
• The GrenchMark Framework
• Past and Current Experience with GrenchMark
• A GrenchMark Success Story
• Future Work
• Conclusions
April 4, 2012
2

3. The Generic Problem of
Analyzing, Testing, and Comparing Grids
• Use cases for automatically analyzing, testing, and
comparing Grids
• Comparisons for system design and procurement
• Functionality testing and system tuning
• Performance testing/analysis of grid applications
• …
• For grids, this problem is hard !
• Testing in real environments is difficult
• Grids change rapidly
• Validity of tests
• …
April 4, 2012
3

4. A Generic Solution to
Analyzing, Testing, and Comparing Grids
• “ Generate and run synthetic grid workloads, based
on real and synthetic applications “
• Current alternatives (not covering all problems)
• Benchmarking with real/synthetic applications (representative?)
• User-defined test management (statistically sound?)
• Advantages of using synthetic grid workloads
• Statistically sound composition of benchmarks
• Statistically sound test management
• Generic: cover the use cases’ broad spectrum (to be shown)
April 4, 2012
4

5. Outline
• Introduction and Motivation
• The GrenchMark Framework
• Past and Current Experience with GrenchMark
• A GrenchMark Success Story
• Future Work
• Conclusions
April 4, 2012
5

6. GrenchMark: a Framework for
Analyzing, Testing, and Comparing grids
• What’s in a name?
grid benchmark → working towards a generic tool for the
whole community: help standardizing the testing procedures,
but benchmarks are too early; we use
synthetic grid workloads instead
• What’s it about?
A systematic approach to analyzing, testing, and comparing grid
settings, based on synthetic workloads
• A set of metrics for analyzing grid settings
• A set of representative grid applications
• Both real and synthetic
• Easy-to-use tools to create synthetic grid workloads
• Flexible, extensible framework
April 4, 2012
6

7. GrenchMark: Iterative Research Roadmap
April 4, 2012
7

8. GrenchMark: Iterative Research Roadmap
Simple functional system
A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema,
Synthetic Grid Workloads with Ibis, KOALA, and
GrenchMark, CoreGRID IW, Nov 2005.
April 4, 2012
8

9. GrenchMark: Iterative Research Roadmap
Open-
GrenchMark
Community
Effort
Complex extensible system
This work
April 4, 2012
9

10. GrenchMark Overview:
Generate and Run Synthetic Workloads
April 4, 2012
10

11. GrenchMark Overview: Easy to
Generate and Run Synthetic Workloads
April 4, 2012
11

12. … but More Complicated Than You Think
• Workload structure • Measurement methods
• User-defined and • Long workloads
statistical models • Saturated / non-saturated system
• Dynamic jobs arrival • Start-up, production, and
• Burstiness and self-similarity cool-down scenarios
• Feedback, background load • Scaling workload to system
• Machine usage assumptions • Applications
• Users, VOs • Synthetic
• Metrics • Real
• A(W) Run/Wait/Resp. Time • Workload definition language
• Efficiency, MakeSpan • Base language layer
• Failure rate [!] • Extended language layer
• (Grid) notions • Other
• Co-allocation, interactive jobs, • Can use the same workload for
malleable, moldable, … both simulations and real
environments
April 4, 2012
12

13. GrenchMark Overview:
Unitary and Composite Applications
• Unitary applications
• sequential, MPI, Java RMI, Ibis, …
• Composite applications
• Bag of tasks
• Chain of jobs
• Direct Acyclic Graph-based
(Standard Task Graph Archive)
April 4, 2012
13

14. GrenchMark Overview:
Workload Description Files
• Format:
Number of jobs and application and
Composition Co-allocation types Language extensions
Inter-arrival and
number of components
Combining four workloads into one start time
April 4, 2012
14

15. Using GrenchMark:
Grid System Analysis
• Performance testing: test the performance of an
application (for sequential, MPI, Ibis applications)
• Report runtimes, waiting times, grid middleware overhead
• Automatic results analysis
• What-if analysis: evaluate potential situations
• System change
• Grid inter-operability
• Special situations: spikes in demand
April 4, 2012
15

16. Using GrenchMark:
Functionality Testing in Grid Environments
• System functionality testing: show the ability of
the system to run various types of applications
• Report failure rates
[ arguably, functionality in grids is even more important than
performance !  10% job failure rate in a controlled system
like the DAS ]
• Periodic system testing: evaluate the current state
of the grid
• Replay workloads
April 4, 2012
16

17. Using GrenchMark:
Comparing Grid Settings
• Single-site vs. co-allocated jobs:
compare the success rate of single-site and co-allocated
jobs, in a system without reservation capabilities
• Single-site jobs 20% better vs. small co-allocated jobs (<32 CPUs),
30% better vs. large co-allocated jobs
[setting and workload-dependent !]
• Unitary vs. composite jobs:
compare the success rate of unitary and composite jobs,
with and without failure handling mechanisms
• Both 100% with simple retry mechanism
[setting and workload-dependent !]
April 4, 2012
17

18. A GrenchMark Success Story:
Releasing the Koala Grid Scheduler on the DAS
• Koala [ http://www.st.ewi.tudelft.nl/koala/ ]
• Grid Scheduler with co-allocation capabilities
• DAS: The Dutch Grid, ~200 researchers
• Initially
• Koala, a tested (!) scheduler, pre-release version
• Test specifics
• 3 different job submission modules
• Workloads with different jobs requirements,
inter-arrival rates, co-allocated v. single site jobs…
• Evaluate: job success rate, Koala overhead and bottlenecks
• Results
• 5,000+ jobs successfully run (all workloads); functionality tests
• 2 major bugs first day, 10+ bugs overall (all fixed)
• KOALA is now officially released on the DAS
(full credit to KOALA developers, 10x for testing with GrenchMark)
April 4, 2012
18

19. GrenchMark’s Current Status:
pre-”Open-GrenchMark”
• Already done in Python [http://www.python.org]
• Workload Generator
• Generic Workload Submitter (Koala, Globus GRAM,
option to extend for JSDL, Condor, PBS, LSF, SGE, …)
• Applications
• Unitary, 3 types: sequential, MPI, Ibis (Java)
• +35 real and synthetic applications
• Composite applications: DAG-based
• Extending modeling capabilities
A.Iosup, D.H.J.Epema (TU Delft), C. Franke, A. Papaspyrou,
L. Schley, B. Song, R. Yahyapour (U Dortmund), On modeling
synthetic workloads for Grid performance evaluation, 12th
Workshop on Job Scheduling Strategies for Parallel
Processing (JSSPP), held April 4, 2012
in conjunction with SIGMETRICS
2006, Saint Malo, France, June 2006 (accepted). 19

20. Towards Open-GrenchMark:
Grid traces, Simulators, Benchmarks
• Distributed testing
• Integrate with DiPerF (C. Dumitrescu, I. Raicu, M. Ripeanu)
• Grid traces analysis
• Automatic tools for grid traces analysis
A. Iosup, C. Dumitrescu, D.H.J. Epema (TU Delft), H. Li,
L. Wolters (U Leiden), How are Real Grids Used? The Analysis
of Four Grid Traces and Its Implications, (submitted).
• Use in conjunction with simulators
• Ability to generate workloads which can be used in simulated
environments (e.g., GangSim, GridSim, …)
• Grid benchmarks
• Analyze the requirements for domain-specific grid benchmarks
April 4, 2012
20

21. Conclusion
• GrenchMark
generates diverse grid workloads
easy-to-use, flexible, portable, extensible, …
• Experience
used GrenchMark to test KOALA’s functionality and performance.
used GrenchMark to analyze, test, and compare grid settings.
15,000+ jobs generated and run … and counting.
• (more) advertisement
Have specific grid setting you would like to test?
Test with GrenchMark!
April 4, 2012
21

22. Thank you! Jason Maassen to Hashim Mohamed (Koala),
Many thanks
and Rob van Nieuwpoort (Ibis).
Questions? Remarks? Observations?
All welcome!
GrenchMark
http://grenchmark.st.ewi.tudelft.nl/ [10x Paulo]
Alexandru IOSUP
TU Delft
A.Iosup@tudelft.nl
http://www.pds.ewi.tudelft.nl/~iosup/index.html [google: “iosup”]
April 4, 2012
22

23. April 4, 2012
23

24. Outline
• Introduction [here]
• The GrenchMark framework
• Experience with GrenchMark
• Extending GrenchMark
• Conclusions
April 4, 2012
24

25. Representative Grid applications (1/4)
• Unitary applications
• Just one scheduling unit (otherwise recursive definition)
• Examples:
Sequential, MPI, Java RMI, Ibis, …
• Composite applications
• Composed of several unitary or composite applications
• Examples:
Parameter sweeps, chains of tasks, DAGs, workflows, …
April 4, 2012
25

26. Representative Grid applications (2/4)
Unitary: synthetic sequential
<<single site/single node/single
stdin processor>> stdout
[I] read I1 data elements from stdin
N+1:I1,N*I2 N+2:O1,N*O2,O3
[O] write O1 data elements to stdout System reqs
Processor (float*N*S*C)
for each N steps (i) # superstep Memory (float*(M+1)*S)
[I] read I2 data elements from stdin
for each M memory locations (j) # computation step
[M] get item j of size S
[P] compute C computation units per memory item (fmul),
and store results into temp memory location
[M] put values from temp to location j
[O] write O2 data elements to stdout
[O] write O3 data elements to stdout Also version with
computation and I/O
April 4, 2012
26

27. Representative Grid applications (2/4)
Unitary: synthetic parallel
<<MPI system/single site>>
<<single node/single processor>>
<<single node/single processor>>
[I] read I1 data from in$ProcID <<single node/single processor>>
app-input.i app-output.i
[O] write O1 data to out$ProcID
for each N steps (i) # superstep N+1:
I1,N*I2
Process i N+2:
O1,N*O2,O3
[B] synchronize with barrier
[I] read I2 data from in$ProcID
for each M memory locations (j) Machine i
[M] get item j of size S Processor i (float*N*S*C)
k
[P] compute C fmul/mem Memory (float*N*2S)
[M] put values from temp to j
[C] communicate to all other processes X values and
receive X values from every other processor
[O] write O2 data elements to out$ProcID
[O] write O3 data elements to out$ProcID
April 4, 2012
27

28. Representative Grid applications (3/4)
Unitary: Ibis jobs
• No modeling, just real applications
[ http://www.cs.vu.nl/ibis ]
• physical simulation
• parallel rendering
• computational mathematics
• state space search
• bioinformatics
• data compression
• grid methods
• optimization
April 4, 2012
28

29. Representative Grid applications (3/4)
Composite: DAG-based
• DAG-based applications
• Real DAG
• Chain of tools
• Try to model real or predicted (use) cases
perf1.dat <<final>> param2.in some-list.in
param1.in out_1-2.dat l1p.dat
<<final>>
out2.res
App1 out_1-1.dat Linker1 huge-data.out App2
<<single>> <<link>> <<single>>
perf2.dat
Linker Input Identity
(one task’s output = other’s input, unmodified)
User task Output Final result
April 4, 2012
29