SlideShare a Scribd company logo
1 of 33
Download to read offline
Testing Spark and scala
https://github.com/ganeshayadiyala/Scalatest-library-to-unit-test-spark/
● Ganesha Yadiyala
● Big data consultant at
datamantra.io
● Consult in spark and scala
● ganeshayadiyala@gmail.com
Agenda
● What is testing
● Different types of testing process
● Unit tests using scalatest
● Different styles in scalatest
● Using assertions
● Sharing fixtures
● Matchers
● Async Testing
● Testing of spark batch operation
● Unit testing streaming operation
What is testing
Software testing is a process of executing a program or application with the intent
of finding the software bugs.
It can also be stated as the process of validating and verifying that a software
application,
● Meets the business and technical requirements that guided it’s design and
development
● Works as expected
Few of the types of tests
● Unit tests
● Integration tests
● Functional tests
Unit tests
● Unit testing simply verifies that individual units of code (mostly functions) work
as expected
● Assumes everything else works
● Tests one specific condition or flow.
Advantages :
● Codes are more reusable. In order to make unit testing possible, codes need
to be modular. This means that codes are easier to reuse.
● Debugging is easy. When a test fails, only the latest changes need to be
debugged.
Integration tests
● Tests the interoperability of multiple subsystem
● Includes real components, databases etc
● Tests the connectivity of the components
● Hard to test all the cases (combination of tests are more)
● Hard to localize the errors ( may break different reasons)
● Much slower than unit tests
Functional tests
● Functional Testing is the type of testing done against the business
requirements of application
● Use real components and real data
Unit Test in scala
Scalatest
● We use scalatest for unit tests in scala
● For every class in src/main/scala write a test class in src/test/scala
● Consists of suite (collection of test cases)
● You define test classes by composing Suite style and mixin traits.
● You can test both scala and java code
● offers deep integration with tools such as JUnit, TestNG, Ant, Maven, sbt,
ScalaCheck, JMock, EasyMock, Mockito, ScalaMock, Selenium, Eclipse,
NetBeans, and IntelliJ.
Using the scalatest maven plugin
We have to disable maven surefire plugin and enable scalatest plugin
● Specify <skipTests>true</skipTests> in maven surefire plugin
● Add the scalatest-maven plugin and set the goals to test
Different styles in scalatest
● FunSuite
● FlatSpec
● FunSpec
● WordSpec
● FreeSpec
● PropSpec
● FeatureSpec
FunSuite
● In a FunSuite, tests are function values.
● You denote tests with test and provide the name of the test as a string
enclosed in parentheses, followed by the code of the test in curly braces
Ex : com.ganesh.scalatest.specs.FunSuitTest.scala
FlatSpec
● No nesting approach contrasts with the traits FunSpec and WordSpec.
● Uses behavior of clause
Ex : com.ganesh.scalatest.specs.FlatSpecTest.scala
FunSpec
● Tests are combined with text that specifies the behavior of the test.
● Uses describe clause
Ex : com.ganesh.scalatest.specs.FunSpecTest.scala
WordSpec
● your specification text is structured by placing words after strings
● Uses should and in clause
Ex : com.ganesh.scalatest.specs.WordSpecTest.scala
Using Assertions
ScalaTest makes three assertions available by default in any style trait
● assert - for general assertion.
● assertResult - to differentiate expected from actual values.
● assertThrows - to ensure a bit of code throws an expected exception.
Scalatest assertions are defined in trait Assertions. Assertions also provide some
other API’s.
Ex : com.ganesh.scalatest.features.AssertionsTest.scala
Ignoring the test
● Scalatest allows to ignore the test.
● We can ignore the test if we want it to change it implementation and run later
or if the test case is slow.
● We use ignore clause to ignore the test
● We use @Ignore annotation to ignore all the test in a suite.
Ex : com.ganesh.scalatest.features.IgnoreTest.scala
Sharing fixture
A test fixture is composed of the objects and other artifacts, which tests use to do
their work.
When multiple tests needs to work with the same fixture, we can share the fixture
between them.
It will reduce the duplication of code.
By calling get-fixture methods
If you need to create the same mutable fixture objects in multiple tests we can use
get-fixture method
● A get-fixture method returns a new instance of a needed fixture object each
time it is called
● Not appropriate to use if we need to cleanup those objects
Ex : com.ganesh.scalatest.fixtures.GetFixtureTest.scala
By Instantiating fixture-context objects
When different tests need different combinations of fixture objects, define the
fixture objects as instance variables of fixture-context objects.
● In this approach we initialize a fixture object inside trait/class.
● We create a new instance of the fixture trait in the test we need them.
● We can even mix in these fixture traits we created.
Ex : com.ganesh.scalatest.fixtures.FixtureContextTest.scala
By using withFixture
● Allows cleaning up of fixtures at the end of the tests
● If we have no object to pass to the test case, then we can use
withFixture(NoArgTest).
● If we have one or more objects to be passed to test case, then we need to
use withFixture(OneArgTest).
Ex : com.ganesh.scalatest.fixtures.WithFicture*.scala
By using BeforeAndAfter
● Methods which we used till now for sharing fixtures are performed during the
test.
● If exception occurs while creating this fixture then it’ll be reported as test
failure.
● If we use BeforeAndAfter setup happens before the test execution starts, and
cleanup happens once the test is completed
● So if any exception happens in the setup, it’ll abort the entire suit and no more
tests are attempted.
Ex : com.ganesh.scalatest.fixtures.BeforeAndAfterTest.scala
Matchers
ScalaTest provides a domain specific language (DSL) for expressing assertions in
tests using the word should.
Ex : com.ganesh.scalatest.features.MatchersTest.scala
Asynchronous testing
● Given a Future returned by the code you are testing, you need not block until
the Future completes before performing assertions against its value.
● We can instead map those assertions onto the Future and return the resulting
Future[Assertion] to ScalaTest.
● This result is executed asynchronously.
Ex : com.ganesh.scalatest.features.AsyncTest.scala
Testing private methods
● If the method is private in a class we can test it using scalatest.
● We can use PrivateMethodTester trait to achieve this.
● We can use invokePrivate operator to call the private method
Ex : com.ganesh.scalatest.features.PrivateMethodTest.scala
Mocking
Scalatest supports following mock libraries,
● ScalaMock
● EasyMock
● JMock
● Mockito
Ex : com.ganesh.scalatest.mock.MockTest.scala
Testing Spark
Complexities
● Needs spark context for all the tests
● Testing operations such as map, flatmap and reduce.
● Testing streaming application (Dstream operations).
● Making sure that there is only one context for each test case.
Setup
● Instead of creating contexts which are needed for each test suite, we create
the trait which extends BeforeAndAfter, and all our suites will extend this trait.
● In that trait we try to initialize all the contexts in before method
● All the contexts will be destroyed in after method
● Extend this trait in all the test suites
Ex : com.ganesh.scalatest.sparkbatch.EnvironmentInitializerSC.scala
Spark Streaming test
● The full control over clock is needed to manually manage batches, slides and
windows.
● Spark Streaming provides necessary abstraction over system clock,
ManualClock class.
● But its private class, we cannot access it in our testcases
● So we use a wrapper class to use the ManualClock instance in our test case.
Ex : com.ganesh.scalatest.sparkstreaming
Summary
● We can select any of the styles provided by the scalatest, it just differs in how
we write test but will have all the features.
● Make use of assertions and matchers provided by scalatest for better test
cases.
● While testing spark we need to test the logic, so keep your code modular so
that each logic can be tested individually.
● There is a external library called spark testing base which provides many
functions to assert on dataframe level and it has traits which provides you the
contexts needed for the test.
References
● http://www.scalatest.org/
● http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/
● https://www.slideshare.net/remeniuk/testing-in-scala-adform-research

More Related Content

What's hot

Avro Tutorial - Records with Schema for Kafka and Hadoop
Avro Tutorial - Records with Schema for Kafka and HadoopAvro Tutorial - Records with Schema for Kafka and Hadoop
Avro Tutorial - Records with Schema for Kafka and HadoopJean-Paul Azar
 
Kafka Tutorial - introduction to the Kafka streaming platform
Kafka Tutorial - introduction to the Kafka streaming platformKafka Tutorial - introduction to the Kafka streaming platform
Kafka Tutorial - introduction to the Kafka streaming platformJean-Paul Azar
 
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...Edureka!
 
Distributed Tracing with Jaeger
Distributed Tracing with JaegerDistributed Tracing with Jaeger
Distributed Tracing with JaegerInho Kang
 
Building a SIMD Supported Vectorized Native Engine for Spark SQL
Building a SIMD Supported Vectorized Native Engine for Spark SQLBuilding a SIMD Supported Vectorized Native Engine for Spark SQL
Building a SIMD Supported Vectorized Native Engine for Spark SQLDatabricks
 
Performance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark MetricsPerformance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark MetricsDatabricks
 
Dynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache SparkDynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache SparkDatabricks
 
Enabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkEnabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkKazuaki Ishizaki
 
Apache Spark in Depth: Core Concepts, Architecture & Internals
Apache Spark in Depth: Core Concepts, Architecture & InternalsApache Spark in Depth: Core Concepts, Architecture & Internals
Apache Spark in Depth: Core Concepts, Architecture & InternalsAnton Kirillov
 
Improving Apache Spark Downscaling
 Improving Apache Spark Downscaling Improving Apache Spark Downscaling
Improving Apache Spark DownscalingDatabricks
 
Apache Spark on K8S Best Practice and Performance in the Cloud
Apache Spark on K8S Best Practice and Performance in the CloudApache Spark on K8S Best Practice and Performance in the Cloud
Apache Spark on K8S Best Practice and Performance in the CloudDatabricks
 
Productizing Structured Streaming Jobs
Productizing Structured Streaming JobsProductizing Structured Streaming Jobs
Productizing Structured Streaming JobsDatabricks
 
Apache Spark Core—Deep Dive—Proper Optimization
Apache Spark Core—Deep Dive—Proper OptimizationApache Spark Core—Deep Dive—Proper Optimization
Apache Spark Core—Deep Dive—Proper OptimizationDatabricks
 
PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edu...
PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edu...PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edu...
PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edu...Edureka!
 
Why your Spark job is failing
Why your Spark job is failingWhy your Spark job is failing
Why your Spark job is failingSandy Ryza
 
Time Series Analytics Azure ADX
Time Series Analytics Azure ADXTime Series Analytics Azure ADX
Time Series Analytics Azure ADXRiccardo Zamana
 
Making Apache Spark Better with Delta Lake
Making Apache Spark Better with Delta LakeMaking Apache Spark Better with Delta Lake
Making Apache Spark Better with Delta LakeDatabricks
 
Microservices with Kafka Ecosystem
Microservices with Kafka EcosystemMicroservices with Kafka Ecosystem
Microservices with Kafka EcosystemGuido Schmutz
 
Spark with Delta Lake
Spark with Delta LakeSpark with Delta Lake
Spark with Delta LakeKnoldus Inc.
 

What's hot (20)

Avro Tutorial - Records with Schema for Kafka and Hadoop
Avro Tutorial - Records with Schema for Kafka and HadoopAvro Tutorial - Records with Schema for Kafka and Hadoop
Avro Tutorial - Records with Schema for Kafka and Hadoop
 
Kafka Tutorial - introduction to the Kafka streaming platform
Kafka Tutorial - introduction to the Kafka streaming platformKafka Tutorial - introduction to the Kafka streaming platform
Kafka Tutorial - introduction to the Kafka streaming platform
 
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
 
Distributed Tracing with Jaeger
Distributed Tracing with JaegerDistributed Tracing with Jaeger
Distributed Tracing with Jaeger
 
Building a SIMD Supported Vectorized Native Engine for Spark SQL
Building a SIMD Supported Vectorized Native Engine for Spark SQLBuilding a SIMD Supported Vectorized Native Engine for Spark SQL
Building a SIMD Supported Vectorized Native Engine for Spark SQL
 
Performance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark MetricsPerformance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark Metrics
 
Dynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache SparkDynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache Spark
 
Enabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkEnabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache Spark
 
Apache Spark in Depth: Core Concepts, Architecture & Internals
Apache Spark in Depth: Core Concepts, Architecture & InternalsApache Spark in Depth: Core Concepts, Architecture & Internals
Apache Spark in Depth: Core Concepts, Architecture & Internals
 
Improving Apache Spark Downscaling
 Improving Apache Spark Downscaling Improving Apache Spark Downscaling
Improving Apache Spark Downscaling
 
Apache Spark on K8S Best Practice and Performance in the Cloud
Apache Spark on K8S Best Practice and Performance in the CloudApache Spark on K8S Best Practice and Performance in the Cloud
Apache Spark on K8S Best Practice and Performance in the Cloud
 
Productizing Structured Streaming Jobs
Productizing Structured Streaming JobsProductizing Structured Streaming Jobs
Productizing Structured Streaming Jobs
 
Apache Spark Core—Deep Dive—Proper Optimization
Apache Spark Core—Deep Dive—Proper OptimizationApache Spark Core—Deep Dive—Proper Optimization
Apache Spark Core—Deep Dive—Proper Optimization
 
PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edu...
PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edu...PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edu...
PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edu...
 
Spark
SparkSpark
Spark
 
Why your Spark job is failing
Why your Spark job is failingWhy your Spark job is failing
Why your Spark job is failing
 
Time Series Analytics Azure ADX
Time Series Analytics Azure ADXTime Series Analytics Azure ADX
Time Series Analytics Azure ADX
 
Making Apache Spark Better with Delta Lake
Making Apache Spark Better with Delta LakeMaking Apache Spark Better with Delta Lake
Making Apache Spark Better with Delta Lake
 
Microservices with Kafka Ecosystem
Microservices with Kafka EcosystemMicroservices with Kafka Ecosystem
Microservices with Kafka Ecosystem
 
Spark with Delta Lake
Spark with Delta LakeSpark with Delta Lake
Spark with Delta Lake
 

Similar to Testing Spark and Scala

Getting started with karate dsl
Getting started with karate dslGetting started with karate dsl
Getting started with karate dslKnoldus Inc.
 
JAVASCRIPT Test Driven Development & Jasmine
JAVASCRIPT Test Driven Development & JasmineJAVASCRIPT Test Driven Development & Jasmine
JAVASCRIPT Test Driven Development & JasmineAnup Singh
 
JUnit- A Unit Testing Framework
JUnit- A Unit Testing FrameworkJUnit- A Unit Testing Framework
JUnit- A Unit Testing FrameworkOnkar Deshpande
 
S313352 optimizing java device testing with automatic feature discovering
S313352 optimizing java device testing with automatic feature discoveringS313352 optimizing java device testing with automatic feature discovering
S313352 optimizing java device testing with automatic feature discoveringromanovfedor
 
Java Unit Test - JUnit
Java Unit Test - JUnitJava Unit Test - JUnit
Java Unit Test - JUnitAktuğ Urun
 
Unit Testing and Coverage for AngularJS
Unit Testing and Coverage for AngularJSUnit Testing and Coverage for AngularJS
Unit Testing and Coverage for AngularJSKnoldus Inc.
 
[FullStack NYC 2019] Effective Unit Tests for JavaScript
[FullStack NYC 2019] Effective Unit Tests for JavaScript[FullStack NYC 2019] Effective Unit Tests for JavaScript
[FullStack NYC 2019] Effective Unit Tests for JavaScriptHazem Saleh
 
Unit testing in xcode 8 with swift
Unit testing in xcode 8 with swiftUnit testing in xcode 8 with swift
Unit testing in xcode 8 with swiftallanh0526
 
Kirill Rozin - Practical Wars for Automatization
Kirill Rozin - Practical Wars for AutomatizationKirill Rozin - Practical Wars for Automatization
Kirill Rozin - Practical Wars for AutomatizationSergey Arkhipov
 
Unit Testing in Angular
Unit Testing in AngularUnit Testing in Angular
Unit Testing in AngularKnoldus Inc.
 

Similar to Testing Spark and Scala (20)

Scala test
Scala testScala test
Scala test
 
Scala test
Scala testScala test
Scala test
 
Getting started with karate dsl
Getting started with karate dslGetting started with karate dsl
Getting started with karate dsl
 
JAVASCRIPT Test Driven Development & Jasmine
JAVASCRIPT Test Driven Development & JasmineJAVASCRIPT Test Driven Development & Jasmine
JAVASCRIPT Test Driven Development & Jasmine
 
Cypress Testing.pptx
Cypress Testing.pptxCypress Testing.pptx
Cypress Testing.pptx
 
JUnit- A Unit Testing Framework
JUnit- A Unit Testing FrameworkJUnit- A Unit Testing Framework
JUnit- A Unit Testing Framework
 
S313352 optimizing java device testing with automatic feature discovering
S313352 optimizing java device testing with automatic feature discoveringS313352 optimizing java device testing with automatic feature discovering
S313352 optimizing java device testing with automatic feature discovering
 
Java Unit Test - JUnit
Java Unit Test - JUnitJava Unit Test - JUnit
Java Unit Test - JUnit
 
Unit testing
Unit testingUnit testing
Unit testing
 
Unit Testing and Coverage for AngularJS
Unit Testing and Coverage for AngularJSUnit Testing and Coverage for AngularJS
Unit Testing and Coverage for AngularJS
 
[FullStack NYC 2019] Effective Unit Tests for JavaScript
[FullStack NYC 2019] Effective Unit Tests for JavaScript[FullStack NYC 2019] Effective Unit Tests for JavaScript
[FullStack NYC 2019] Effective Unit Tests for JavaScript
 
Unit testing in xcode 8 with swift
Unit testing in xcode 8 with swiftUnit testing in xcode 8 with swift
Unit testing in xcode 8 with swift
 
Intro to junit
Intro to junitIntro to junit
Intro to junit
 
Kirill Rozin - Practical Wars for Automatization
Kirill Rozin - Practical Wars for AutomatizationKirill Rozin - Practical Wars for Automatization
Kirill Rozin - Practical Wars for Automatization
 
Automation for developers
Automation for developersAutomation for developers
Automation for developers
 
Annotations
AnnotationsAnnotations
Annotations
 
Unit testing
Unit testingUnit testing
Unit testing
 
Unit Testing in Angular
Unit Testing in AngularUnit Testing in Angular
Unit Testing in Angular
 
Wso2 test automation framework internal training
Wso2 test automation framework internal trainingWso2 test automation framework internal training
Wso2 test automation framework internal training
 
Testing Angular
Testing AngularTesting Angular
Testing Angular
 

More from datamantra

Multi Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and TelliusMulti Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and Telliusdatamantra
 
State management in Structured Streaming
State management in Structured StreamingState management in Structured Streaming
State management in Structured Streamingdatamantra
 
Spark on Kubernetes
Spark on KubernetesSpark on Kubernetes
Spark on Kubernetesdatamantra
 
Understanding transactional writes in datasource v2
Understanding transactional writes in  datasource v2Understanding transactional writes in  datasource v2
Understanding transactional writes in datasource v2datamantra
 
Introduction to Datasource V2 API
Introduction to Datasource V2 APIIntroduction to Datasource V2 API
Introduction to Datasource V2 APIdatamantra
 
Exploratory Data Analysis in Spark
Exploratory Data Analysis in SparkExploratory Data Analysis in Spark
Exploratory Data Analysis in Sparkdatamantra
 
Core Services behind Spark Job Execution
Core Services behind Spark Job ExecutionCore Services behind Spark Job Execution
Core Services behind Spark Job Executiondatamantra
 
Optimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloadsOptimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloadsdatamantra
 
Structured Streaming with Kafka
Structured Streaming with KafkaStructured Streaming with Kafka
Structured Streaming with Kafkadatamantra
 
Understanding time in structured streaming
Understanding time in structured streamingUnderstanding time in structured streaming
Understanding time in structured streamingdatamantra
 
Spark stack for Model life-cycle management
Spark stack for Model life-cycle managementSpark stack for Model life-cycle management
Spark stack for Model life-cycle managementdatamantra
 
Productionalizing Spark ML
Productionalizing Spark MLProductionalizing Spark ML
Productionalizing Spark MLdatamantra
 
Introduction to Structured streaming
Introduction to Structured streamingIntroduction to Structured streaming
Introduction to Structured streamingdatamantra
 
Building real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark StreamingBuilding real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark Streamingdatamantra
 
Understanding Implicits in Scala
Understanding Implicits in ScalaUnderstanding Implicits in Scala
Understanding Implicits in Scaladatamantra
 
Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2datamantra
 
Migrating to spark 2.0
Migrating to spark 2.0Migrating to spark 2.0
Migrating to spark 2.0datamantra
 
Scalable Spark deployment using Kubernetes
Scalable Spark deployment using KubernetesScalable Spark deployment using Kubernetes
Scalable Spark deployment using Kubernetesdatamantra
 
Introduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actorsIntroduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actorsdatamantra
 
Functional programming in Scala
Functional programming in ScalaFunctional programming in Scala
Functional programming in Scaladatamantra
 

More from datamantra (20)

Multi Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and TelliusMulti Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and Tellius
 
State management in Structured Streaming
State management in Structured StreamingState management in Structured Streaming
State management in Structured Streaming
 
Spark on Kubernetes
Spark on KubernetesSpark on Kubernetes
Spark on Kubernetes
 
Understanding transactional writes in datasource v2
Understanding transactional writes in  datasource v2Understanding transactional writes in  datasource v2
Understanding transactional writes in datasource v2
 
Introduction to Datasource V2 API
Introduction to Datasource V2 APIIntroduction to Datasource V2 API
Introduction to Datasource V2 API
 
Exploratory Data Analysis in Spark
Exploratory Data Analysis in SparkExploratory Data Analysis in Spark
Exploratory Data Analysis in Spark
 
Core Services behind Spark Job Execution
Core Services behind Spark Job ExecutionCore Services behind Spark Job Execution
Core Services behind Spark Job Execution
 
Optimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloadsOptimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloads
 
Structured Streaming with Kafka
Structured Streaming with KafkaStructured Streaming with Kafka
Structured Streaming with Kafka
 
Understanding time in structured streaming
Understanding time in structured streamingUnderstanding time in structured streaming
Understanding time in structured streaming
 
Spark stack for Model life-cycle management
Spark stack for Model life-cycle managementSpark stack for Model life-cycle management
Spark stack for Model life-cycle management
 
Productionalizing Spark ML
Productionalizing Spark MLProductionalizing Spark ML
Productionalizing Spark ML
 
Introduction to Structured streaming
Introduction to Structured streamingIntroduction to Structured streaming
Introduction to Structured streaming
 
Building real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark StreamingBuilding real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark Streaming
 
Understanding Implicits in Scala
Understanding Implicits in ScalaUnderstanding Implicits in Scala
Understanding Implicits in Scala
 
Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2
 
Migrating to spark 2.0
Migrating to spark 2.0Migrating to spark 2.0
Migrating to spark 2.0
 
Scalable Spark deployment using Kubernetes
Scalable Spark deployment using KubernetesScalable Spark deployment using Kubernetes
Scalable Spark deployment using Kubernetes
 
Introduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actorsIntroduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actors
 
Functional programming in Scala
Functional programming in ScalaFunctional programming in Scala
Functional programming in Scala
 

Recently uploaded

AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 

Recently uploaded (17)

AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 

Testing Spark and Scala

  • 1. Testing Spark and scala https://github.com/ganeshayadiyala/Scalatest-library-to-unit-test-spark/
  • 2. ● Ganesha Yadiyala ● Big data consultant at datamantra.io ● Consult in spark and scala ● ganeshayadiyala@gmail.com
  • 3. Agenda ● What is testing ● Different types of testing process ● Unit tests using scalatest ● Different styles in scalatest ● Using assertions ● Sharing fixtures ● Matchers ● Async Testing ● Testing of spark batch operation ● Unit testing streaming operation
  • 4. What is testing Software testing is a process of executing a program or application with the intent of finding the software bugs. It can also be stated as the process of validating and verifying that a software application, ● Meets the business and technical requirements that guided it’s design and development ● Works as expected
  • 5. Few of the types of tests ● Unit tests ● Integration tests ● Functional tests
  • 6. Unit tests ● Unit testing simply verifies that individual units of code (mostly functions) work as expected ● Assumes everything else works ● Tests one specific condition or flow. Advantages : ● Codes are more reusable. In order to make unit testing possible, codes need to be modular. This means that codes are easier to reuse. ● Debugging is easy. When a test fails, only the latest changes need to be debugged.
  • 7. Integration tests ● Tests the interoperability of multiple subsystem ● Includes real components, databases etc ● Tests the connectivity of the components ● Hard to test all the cases (combination of tests are more) ● Hard to localize the errors ( may break different reasons) ● Much slower than unit tests
  • 8. Functional tests ● Functional Testing is the type of testing done against the business requirements of application ● Use real components and real data
  • 9. Unit Test in scala
  • 10. Scalatest ● We use scalatest for unit tests in scala ● For every class in src/main/scala write a test class in src/test/scala ● Consists of suite (collection of test cases) ● You define test classes by composing Suite style and mixin traits. ● You can test both scala and java code ● offers deep integration with tools such as JUnit, TestNG, Ant, Maven, sbt, ScalaCheck, JMock, EasyMock, Mockito, ScalaMock, Selenium, Eclipse, NetBeans, and IntelliJ.
  • 11. Using the scalatest maven plugin We have to disable maven surefire plugin and enable scalatest plugin ● Specify <skipTests>true</skipTests> in maven surefire plugin ● Add the scalatest-maven plugin and set the goals to test
  • 12. Different styles in scalatest ● FunSuite ● FlatSpec ● FunSpec ● WordSpec ● FreeSpec ● PropSpec ● FeatureSpec
  • 13. FunSuite ● In a FunSuite, tests are function values. ● You denote tests with test and provide the name of the test as a string enclosed in parentheses, followed by the code of the test in curly braces Ex : com.ganesh.scalatest.specs.FunSuitTest.scala
  • 14. FlatSpec ● No nesting approach contrasts with the traits FunSpec and WordSpec. ● Uses behavior of clause Ex : com.ganesh.scalatest.specs.FlatSpecTest.scala
  • 15. FunSpec ● Tests are combined with text that specifies the behavior of the test. ● Uses describe clause Ex : com.ganesh.scalatest.specs.FunSpecTest.scala
  • 16. WordSpec ● your specification text is structured by placing words after strings ● Uses should and in clause Ex : com.ganesh.scalatest.specs.WordSpecTest.scala
  • 17. Using Assertions ScalaTest makes three assertions available by default in any style trait ● assert - for general assertion. ● assertResult - to differentiate expected from actual values. ● assertThrows - to ensure a bit of code throws an expected exception. Scalatest assertions are defined in trait Assertions. Assertions also provide some other API’s. Ex : com.ganesh.scalatest.features.AssertionsTest.scala
  • 18. Ignoring the test ● Scalatest allows to ignore the test. ● We can ignore the test if we want it to change it implementation and run later or if the test case is slow. ● We use ignore clause to ignore the test ● We use @Ignore annotation to ignore all the test in a suite. Ex : com.ganesh.scalatest.features.IgnoreTest.scala
  • 19. Sharing fixture A test fixture is composed of the objects and other artifacts, which tests use to do their work. When multiple tests needs to work with the same fixture, we can share the fixture between them. It will reduce the duplication of code.
  • 20. By calling get-fixture methods If you need to create the same mutable fixture objects in multiple tests we can use get-fixture method ● A get-fixture method returns a new instance of a needed fixture object each time it is called ● Not appropriate to use if we need to cleanup those objects Ex : com.ganesh.scalatest.fixtures.GetFixtureTest.scala
  • 21. By Instantiating fixture-context objects When different tests need different combinations of fixture objects, define the fixture objects as instance variables of fixture-context objects. ● In this approach we initialize a fixture object inside trait/class. ● We create a new instance of the fixture trait in the test we need them. ● We can even mix in these fixture traits we created. Ex : com.ganesh.scalatest.fixtures.FixtureContextTest.scala
  • 22. By using withFixture ● Allows cleaning up of fixtures at the end of the tests ● If we have no object to pass to the test case, then we can use withFixture(NoArgTest). ● If we have one or more objects to be passed to test case, then we need to use withFixture(OneArgTest). Ex : com.ganesh.scalatest.fixtures.WithFicture*.scala
  • 23. By using BeforeAndAfter ● Methods which we used till now for sharing fixtures are performed during the test. ● If exception occurs while creating this fixture then it’ll be reported as test failure. ● If we use BeforeAndAfter setup happens before the test execution starts, and cleanup happens once the test is completed ● So if any exception happens in the setup, it’ll abort the entire suit and no more tests are attempted. Ex : com.ganesh.scalatest.fixtures.BeforeAndAfterTest.scala
  • 24. Matchers ScalaTest provides a domain specific language (DSL) for expressing assertions in tests using the word should. Ex : com.ganesh.scalatest.features.MatchersTest.scala
  • 25. Asynchronous testing ● Given a Future returned by the code you are testing, you need not block until the Future completes before performing assertions against its value. ● We can instead map those assertions onto the Future and return the resulting Future[Assertion] to ScalaTest. ● This result is executed asynchronously. Ex : com.ganesh.scalatest.features.AsyncTest.scala
  • 26. Testing private methods ● If the method is private in a class we can test it using scalatest. ● We can use PrivateMethodTester trait to achieve this. ● We can use invokePrivate operator to call the private method Ex : com.ganesh.scalatest.features.PrivateMethodTest.scala
  • 27. Mocking Scalatest supports following mock libraries, ● ScalaMock ● EasyMock ● JMock ● Mockito Ex : com.ganesh.scalatest.mock.MockTest.scala
  • 29. Complexities ● Needs spark context for all the tests ● Testing operations such as map, flatmap and reduce. ● Testing streaming application (Dstream operations). ● Making sure that there is only one context for each test case.
  • 30. Setup ● Instead of creating contexts which are needed for each test suite, we create the trait which extends BeforeAndAfter, and all our suites will extend this trait. ● In that trait we try to initialize all the contexts in before method ● All the contexts will be destroyed in after method ● Extend this trait in all the test suites Ex : com.ganesh.scalatest.sparkbatch.EnvironmentInitializerSC.scala
  • 31. Spark Streaming test ● The full control over clock is needed to manually manage batches, slides and windows. ● Spark Streaming provides necessary abstraction over system clock, ManualClock class. ● But its private class, we cannot access it in our testcases ● So we use a wrapper class to use the ManualClock instance in our test case. Ex : com.ganesh.scalatest.sparkstreaming
  • 32. Summary ● We can select any of the styles provided by the scalatest, it just differs in how we write test but will have all the features. ● Make use of assertions and matchers provided by scalatest for better test cases. ● While testing spark we need to test the logic, so keep your code modular so that each logic can be tested individually. ● There is a external library called spark testing base which provides many functions to assert on dataframe level and it has traits which provides you the contexts needed for the test.