This document compares the performance of Hive and Spark when running the BigBench benchmark. It outlines the structure and use cases of the BigBench benchmark, which aims to cover common Big Data analytical properties. It then describes sequential performance tests of Hive+Tez and Spark on queries from the benchmark using a HDInsight PaaS cluster, finding variations in performance between the systems. Concurrency tests are also run by executing multiple query streams in parallel to analyze throughput.

1. UsingBigBenchtocompareUsingBigBenchtocompare
HiveandSparkHiveandSpark
NicolasPoggi,AlejandroNicolasPoggi,AlejandroMonteroMontero
April2017

2. Outline
1.IntrotoBSCandALOJA
2.BigBench
3.Sequen;altests
1.Datascales
4.Concurrencytests
5.Summary
2

3. Barcelona Supercompu.ng Center (BSC)
•Spanishna*onalsupercompu*ngcenter22yearshistoryin:
•ComputerArchitecture,networkinganddistributedsystems
research
•BasedatBarcelonaTechUniversity(UPC)
•Largeongoinglifesciencecomputa*onalprojects
•Prominentbodyofresearchac*vityaroundHadoopHadoop
•2008-2013:SLAAdap*veScheduler,Accelerators,Locality
Awareness,PerformanceManagement.7publica*ons7publica*ons
•2013-Present:CCost-efficientupcomingBigDataarchitectures
(ALOJAALOJA)8+8+publica*onspublica*ons

4. ALOJA: towards cost-effec2ve Big Data
•Researchprojectforautoma1ngcharacteriza1onand
op1miza1onofBigDataBigDatadeployments
•OpensourceBenchmarking-to-Insightspla?ormandtools
•LargestBigDatapublicrepository(70,000+jobs)
•Communitycollabora1onwithindustryandacademia
hJp://aloja.bsc.eshJp://aloja.bsc.es
BigDataBenchmarkingOnlineRepositoryWeb/ML
Analytics

5. BenchmarkingandBigBench

6. Theneedforanewbenchmarkstandard
•Abenchmarkcapturesthesolu3ontoaproblemandguidedecision
making
•Databaserelatedbenchmarksstandards
•Transac3onal(OLTP):TPCCandE
•DecisionSupport(DSS/OLAP):TPCHandDS
•AndforBigDataanaly3csproper3es?
•3Vs,ML,M/R
•Benchmarkuses:
•Systemtuninganddebugging
•SpreadandbroadBigDataecosystem
•Setcommonrules
•Vendorcomparison
•Transparencyacrosstheindustry
6

7. WhatisBigBench(TPCx-BB1)?
•End-to-endapplica/onlevelbenchmark
•resultofmanyyearsofcollabora/on
•industryandacademia
•CoversmostBigDataAnaly/calproper/es(3Vs)
•Basedonaretailercompany(extensionofTPC-DS)
7[1]:http://www.tpc.org/tpc_documents_current_versions/pdf/tpcx-bb_v1.2.0.pdf
BigBenchhistory

8. BigBenchusecasesandprocessoverview
•3030businessusescasesbusinessusescasescovering:
•MerchandisingMerchandising,
•PricingOp9miza9onPricingOp9miza9on
•ProductReturnProductReturn
•CustomersCustomers...
•Implementa9onresultedin:
•14Declara9veDeclara9vequeries(SQL)
•7QuerieswithNaturalLanguageProcessingNaturalLanguageProcessing
•4Querieswithdatapreprocessingwith
MapReducejobsMapReducejobs.
•5QuerieswithMachineLearningMachineLearningpost
processing.
8

9. BigBenchv1.2–ReferenceImplementa7on
HDFS
HiveMetastore
MapReduceTezSpark
Y
ar
n
HiveSparkSQL
MahoutMLCustomSparkMLlibMachine
Learning
SQL
Engine
TableMetastore
Execution
Engine
Filesyste
m
Benchmarkedsystems:
•Hive+MapReduce+Mahout
•Hive+MapReduce+
Spark_MLlib
•Hive+Tez+Mahout
•Hive+Tez+Spark_MLlib
•SparkSQL+Mahout
•SparkSQL+Spark_MLlib
•Spark2SQL+Mahout
Workinprogress:
•Hive2
•Spark2SQL+Spark_MLlib

10. Thecluster(I)–HDInsightPaaS
10
ModelD4v2
#Headnodes2
#Workingnodes4
#Zookeepernodes3
CPU
Intel(R)Xeon(R)CPUE5-
2673v3
8x2,4GHzcores
RAM28GB
HDFSRemote
SoftwareHortonWorksData
Platform2.5
Non-conditionalmapjoinconversionwithsmalltableslesserthan319MB
Softwareconfiguration
Mapper/Reduccer/Tez
memory
1536MB
Mapper/Reduccer/Tez
HeapSpace
1024MB
Mapper/Reduccer/Tez
Cores
1
HiveMapJoinsYes
Sparkexecutors4
Sparkexecutormemory4608MB
SparkexecutorCores3

11. Sequen&alruns(power)
Queries1-30
Averageofthreeexecu&osof100GBScaleFactor11

12. BigBenchworkload–powertest
12
LoadtoHive
Metastore
Data
GenerationQuery1HDFS
Hive
Query2
….Query30

13. PureQL
13Averageofthreeexecutionsusing100GBScaleFactor

14. Query12CPUbehavior
14
Te
z
Spark
1.6.2
Spark
2.0.2
Averageofthreeexecutionsusing100GBScaleFactor

15. CustomReducers
15Averageofthreeexecutionsusing100GBScaleFactor

16. Query2CPUbehavior
16
Te
z
Spark
1.6.2
Spark
2.0.2
Averageofthreeexecutionsusing100GBScaleFactor

17. NaturalLanguageProcessing
17Averageofthreeexecutionsusing100GBScaleFactor

18. Query27CPUbehavior
18
Te
z
Spark
1.6.2
Spark
2.0.2
Averageofthreeexecutionsusing100GBScaleFactor

19. MachineLearning
19Averageofthreeexecutionsusing100GBScaleFactor

20. Query5CPUbehavior
20
Tez+
Mahout
Tez+
Spark_MLlib
Averageofthreeexecutionsusing100GBScaleFactor

21. 21
Aggregated Results
Averageofthreeexecutionsusing100GBScaleFactor

22. Scaling from 1GB to 1TB
Logscales22

23. Concurrencyruns(throughput)
2,4,8parallelstreams
23

24. BigBenchworkload–Throughputtest
24
Query15Query21
….Query16
Query12Query18
….Query22
Query16Query30
….Query29
LoadData
Data
GenerationHDFS
Hive

25. Thecluster(II)–HDInsightPaaS
25
ModelHDInsightD4v3
#Headnodes2
#Workingnodes7
#Zookeepernodes3
CPU
Intel(R)Xeon(R)CPUE5-
2673v3
8x2,4GHzcores
RAM28GB
55GB(Headnode)
HDFSRemote
SoftwareHortonWorksData
Platform2.5
Softwareconfiguration
Mapper/Reduccer/Tez
memory
1536MB
Mapper/Reduccer/Tez
HeapSpace
1024MB
Mapper/Reduccer/Tez
Cores
1
HiveMapJoinsYes
Sparkexecutors9
Sparkexecutormemory4608MB
SparkexecutorCores3
Non-conditionalmapjoinconversionwithsmalltableslesserthan319MB

26. SparkvsHive+Tezinthroughputtests
26

27. 27

28. 28