Discover a practical solution for the Dual Write Problem in distributed systems using the Outbox Pattern. This guide presents a C# demo app with MongoDB, Kafka, and Debezium, illustrating reliable data propagation between services. …
Category: Distributed Systems
This article explores producing and consuming Kafka messages in Protobuf format, adhering to Cloud Events spec, and covers polymorphic data type serialization. …
In this article, we explore Kafka with Protobuf & C#, including a demo for message production & consumption. See how to manage your schema using Confluent Schema Registry and Control Center. …
A blog post about using C# to produce and consume Kafka messages in CloudEvents format. We’ll use JSON for serialization for now, but I’ll also explore Avro and Protobuf in a subsequent article. …
Exploring Kafka Steams Partitioning, Scaling, and Fault Tolerance – we’ll build a simple streaming app and inspect the contents of the repartition topic, RocksDB state store, and the Kafka internal changelog topic. Also, we’ll see how the workload gets distributed between the running app instances. …
An article about integration tests for your Kafka workflow. Using C# and the “Testcontainers” framework, we first run Kafka in Docker, call an API that will push a message to Kafka, consume the produced Kafka message ensuring it’s correct, and finally drop the created containers. …
A blog post where we use Wireshark to trace the Kafka networking protocol. The client is a simple C# app interacting with a multi-broker Kafka cluster. …
In this article, I’ll introduce Kafka listeners, the client-broker “handshake” and pub/sub interactions. …
Monitoring Public Transportation with the Kafka Ecosystem – a demo project part of the Udacity “Data Streaming” Nanodegree. It contains data streaming workflows using Kafka Producers and Consumers, JSON and Avro serialization, Kafka Connect (JDBC), REST Proxy, Robinhood Faust, and KSQL. …
Migrating Your Reporting Queries From a “General Purpose” DB to a Data Warehouse – Performance Overview. The article focuses on MongoDB and ClickHouse, but most takeaways apply to other “traditional” DBs and “data warehouse” solutions. …