Adding Support For Distributed Transactions To The Redpanda Streaming Engine

Data Engineering Podcast

0:00
00:45:58
10
10

Adding Support For Distributed Transactions To The Redpanda Streaming Engine

Data Engineering Podcast

Transactions are a necessary feature for ensuring that a set of actions are all performed as a single unit of work. In streaming systems this is necessary to ensure that a set of messages or transformations are all executed together across different queues. In this episode Denis Rystsov explains how he added support for transactions to the Redpanda streaming engine. He discusses the use cases for transactions, the different strategies, semantics, and guarantees that they might need to support, and how his implementation ended up improving the performance of bulk write operations. This is an interesting deep dive into the internals of a high performance streaming engine and the details that are involved in building distributed systems.
Episodes
Date
Duration
Recommended episodes :

Experimentation and A/B Testing For Modern Data Teams With Eppo

Data Engineering Podcast

Data Driven Hiring For Data Professionals With Alooba

Data Engineering Podcast

Creating A Unified Experience For The Modern Data Stack At Mozart Data

Data Engineering Podcast

The podcast Data Engineering Podcast has been added to your home screen.

Transactions are a necessary feature for ensuring that a set of actions are all performed as a single unit of work. In streaming systems this is necessary to ensure that a set of messages or transformations are all executed together across different queues. In this episode Denis Rystsov explains how he added support for transactions to the Redpanda streaming engine. He discusses the use cases for transactions, the different strategies, semantics, and guarantees that they might need to support, and how his implementation ended up improving the performance of bulk write operations. This is an interesting deep dive into the internals of a high performance streaming engine and the details that are involved in building distributed systems.
Subscribe Install Share
Data Engineering Podcast

Thank you for your subscription

For a better experience, also consider installing the application.

Install