Testing Spark applications to improve quality

  • twitter
  • linkedin
  • facebook
  • email

Developers at Adevinta Spain wanted to run unit, integration and end-to-end tests on Spark applications. Learn how they implemented this.

A useful read for big data developers, get into the detail of how the team at Adevinta Spain implements unit, integration and end-to-end tests on Spark applications as they build a Data Platform. The full article reveals a typical project layout, how to use Spark sessions, what the various test functions enable and how this improves the quality of the code and the CI pipelines.

Highlights include:

  • Layout of a Spark project
  • How to isolate different Spark tests
  • Details of the test implementation code
  • The idea behind SharedSparkSessionHelper
  • Links to real-life Adevinta Spain examples

Read the full article on Adevinta Spain tech blog

More resources

Discover our media resources, brand assets, guidelines, photos and more

This site is registered on wpml.org as a development site.