Please log in to watch this conference skillscast.
Dean Morin has been writing ETL jobs using Kafka for a couple of years. In this talk, you will explore Dean's experiences doing just about everything wrong, before figuring out what does work. You will also discover:
- What Kafka is
- What the major frameworks are, and how they steer you towards one-by-one message processing
- Why you shouldn’t do that, including performance measurements for different methods of loading data into a Postgres data warehouse
- How to avoid one-by-one processing
YOU MAY ALSO LIKE:
One-by-one Is No Fun: Lessons learned writing Kafka ETL jobs
Dean is a Data Engineer originally from Vancouver, now living in San Francisco. His general fussiness and paranoia make him suited to the hairball which is the data world. He’s especially interested in writing metadata-driven ETL systems.