This article explores producing and consuming Kafka messages in Protobuf format, adhering to Cloud Events spec, and covers polymorphic data type serialization. …
Blog Posts
In this article, we explore Kafka with Protobuf & C#, including a demo for message production & consumption. See how to manage your schema using Confluent Schema Registry and Control Center. …
A blog post about using C# to produce and consume Kafka messages in CloudEvents format. We’ll use JSON for serialization for now, but I’ll also explore Avro and Protobuf in a subsequent article. …
Exploring Kafka Steams Partitioning, Scaling, and Fault Tolerance – we’ll build a simple streaming app and inspect the contents of the repartition topic, RocksDB state store, and the Kafka internal changelog topic. Also, we’ll see how the workload gets distributed between the running app instances. …
An article about integration tests for your Kafka workflow. Using C# and the “Testcontainers” framework, we first run Kafka in Docker, call an API that will push a message to Kafka, consume the produced Kafka message ensuring it’s correct, and finally drop the created containers. …
A blog post where we use Wireshark to trace the Kafka networking protocol. The client is a simple C# app interacting with a multi-broker Kafka cluster. …
In this article, I’ll introduce Kafka listeners, the client-broker “handshake” and pub/sub interactions. …
In this article, we’ll continue where we left off and see how to properly handle the networking issues we had last time. Concretely, we need to find a way to host our Fargate Task in a private subnet (so it’s not reachable publicly) but also allow for outbound internet traffic through a static public IP address. We will whitelist this IP in Atlas, making the Mongo cluster accessible from the running container. …
In this couple of articles, I’ll demonstrate how you can deploy a sample .NET console app to AWS and run it periodically as a Fargate task. That’s a common use case for any kind of background service you need to run on some predefined time intervals. …
In this article, I’ll present a demo project for classifying the sentiment of posts from the Stocktwits social media. The community on Stocktwits is full of investors, traders, and entrepreneurs. Each message posted is called a Twit. This is similar to Twitter’s version of a post, called a Tweet. Using Pytorch, we’ll build a model around these twits that generate a sentiment score. …