Rgl3hwfqlpy7umx9r9vd
SkillsCast

One-by-one Is No Fun: Lessons learned writing Kafka ETL jobs

29th April 2017 in London at CodeNode

There are 19 other SkillsCasts available from Data Science Festival 2017

Please log in to watch this conference skillscast.

Https s3.amazonaws.com prod.tracker2 resource 41088130 skillsmatter conference skillscast o9nohu

Dean Morin has been writing ETL jobs using Kafka for a couple of years. In this talk, you will explore Dean's experiences doing just about everything wrong, before figuring out what does work. You will also discover:

  1. What Kafka is
  2. What the major frameworks are, and how they steer you towards one-by-one message processing
  3. Why you shouldn’t do that, including performance measurements for different methods of loading data into a Postgres data warehouse
  4. How to avoid one-by-one processing

YOU MAY ALSO LIKE:

One-by-one Is No Fun: Lessons learned writing Kafka ETL jobs

Dean Morin

Dean is a Data Engineer originally from Vancouver, now living in San Francisco. His general fussiness and paranoia make him suited to the hairball which is the data world. He’s especially interested in writing metadata-driven ETL systems.

SkillsCast

Please log in to watch this conference skillscast.

Https s3.amazonaws.com prod.tracker2 resource 41088130 skillsmatter conference skillscast o9nohu

Dean Morin has been writing ETL jobs using Kafka for a couple of years. In this talk, you will explore Dean's experiences doing just about everything wrong, before figuring out what does work. You will also discover:

  1. What Kafka is
  2. What the major frameworks are, and how they steer you towards one-by-one message processing
  3. Why you shouldn’t do that, including performance measurements for different methods of loading data into a Postgres data warehouse
  4. How to avoid one-by-one processing

YOU MAY ALSO LIKE:

About the Speaker

One-by-one Is No Fun: Lessons learned writing Kafka ETL jobs

Dean Morin

Dean is a Data Engineer originally from Vancouver, now living in San Francisco. His general fussiness and paranoia make him suited to the hairball which is the data world. He’s especially interested in writing metadata-driven ETL systems.

Photos