This guide walks you through building a sample Data Lake in AWS by pushing a News API data stream into S3 using a Firehose C# client, analyzing the data with Athena, and setting up an ETL process via Glue to load the data into MySQL RDS. …
Author: Vasil Kosturski
This blog post offers a detailed tutorial on configuring strongSwan for an AWS Site-to-Site VPN. It includes step-by-step instructions, security practices, and troubleshooting tips for robust connectivity between local networks and AWS. …
In this demo, we set up primary and replica MySQL RDS instances in two regions. We then insert some data into the primary and read from the replica, accessing the databases through EC2 hosts. We also configure a CloudWatch alarm for high replication lag and demonstrate promoting the replica. …
Discover a practical solution for the Dual Write Problem in distributed systems using the Outbox Pattern. This guide presents a C# demo app with MongoDB, Kafka, and Debezium, illustrating reliable data propagation between services. …
This article explores producing and consuming Kafka messages in Protobuf format, adhering to Cloud Events spec, and covers polymorphic data type serialization. …
In this article, we explore Kafka with Protobuf & C#, including a demo for message production & consumption. See how to manage your schema using Confluent Schema Registry and Control Center. …
A blog post about using C# to produce and consume Kafka messages in CloudEvents format. We’ll use JSON for serialization for now, but I’ll also explore Avro and Protobuf in a subsequent article. …
Exploring Kafka Steams Partitioning, Scaling, and Fault Tolerance – we’ll build a simple streaming app and inspect the contents of the repartition topic, RocksDB state store, and the Kafka internal changelog topic. Also, we’ll see how the workload gets distributed between the running app instances. …
An article about integration tests for your Kafka workflow. Using C# and the “Testcontainers” framework, we first run Kafka in Docker, call an API that will push a message to Kafka, consume the produced Kafka message ensuring it’s correct, and finally drop the created containers. …
A blog post where we use Wireshark to trace the Kafka networking protocol. The client is a simple C# app interacting with a multi-broker Kafka cluster. …