Taming data workflows with a move to Athena

  • twitter
  • linkedin
  • facebook
  • email

Find out how leboncoin shifted from Apache Kafka to Athena to better leverage high volumes of data.

After many years of using Apache Kafka, the tech teams at leboncoin needed to rethink workflows in order to be able to analyse, transform and consolidate events in “offline” data storage. Learn how they went from a tedious, manual process to a fully automated workflow, using AWS S3/Athena/Spark, minimising the amount of human input required.

Highlights include:

  • A deep dive into our team’s journey from the Kafka “hot” to S3/Athena “cold” storage workflow
  • How they programmed a solution to fully automate an error-prone archiving process
  • The technical and organisational challenges of ensuring consistency
  • Examples of connector configurations, base configuration and error handling
  • How monitoring and alerting keeps our datalake “fresh” for our data scientists, analysts and data services

Read the full article here.

Taming data workflows with a move to Athena

View our jobs

View all our jobs

More resources

Discover our media resources, brand assets, guidelines, photos and more

This site is registered on wpml.org as a development site.