Table of Contents
Intro
In this article, we will delve into producing and consuming Kafka messages that are serialized in Protobuf format while adhering to the Cloud Events spec. We’ll build upon and enhance our previous articles on JSON formatted Cloud Events and using Protobuf and Schema Registry for Kafka messaging.
Additionally, we will tackle a common use case—serialization and deserialization of polymorphic data types. In our demo, we’ll work with an Animal base class and a few derived classes – Cat, Dog, and Bird. Depending on the runtime type of the Animal, the sender publishes the concrete type, and the consumer deserializes it appropriately.
You can find the complete source code on GitHub.
Setting Up Kafka
To set up your environment for this demo, you can use the docker-compose file from the previous article to launch the required Docker containers. This file comprises the following services:
- Kafka
- Zookeeper
- Schema Registry
- Control Center
You’ll need to create the “animals” topic. You can use any preferred tool, or you can create it via Confluent Control Center, as we did previously, for our “users” topic.
Protobuf Schema and Polymorphic Types
Although Protocol Buffers don’t natively support polymorphism, you can still achieve it by leveraging “OneOf” types by defining a union of multiple types that can be stored in a single field.
For example, in our demo program, the Animal message has a OneOf field animal_type, which can contain three different types of animals: Cat, Dog, or Bird. The consumer would check a special field indicating the concrete type and read the corresponding properties. You’ll see this in action later in the code.
Here is the full animal.proto content:
Generating the C# Classes
The Animal class is generated based on the animal.proto file and the protobuf compiler. Here is a sample command you can use if you want to regenerate the C# class yourself:
protoc --csharp_out=. animal.proto
The Demo App
The sample application starts a Kafka producer and consumer to publish/receive Animal messages, serialized using the schema from the previous section.
We’ve already presented how to produce and consume Cloud Events messages using JSON, so please check the previous article for details. Here, let’s just focus on the differences that allow us to use Protocol Buffers to serialize the Cloud Events data field.
One of the core points is that we’ll be using the Binary content mode for the payload. Recall that the other supported Content Mode is Structured, which we employed with JSON messages.
Producer
The first thing to note is that we are using the ProtobufEventFormatter class provided by the Cloud Events C# SDK.
The formatter is typically responsible for serializing/deserializing the messages. In the current case, though, we’ll just be sending a byte array as a Cloud Event payload, so the Protobuf formatter doesn’t really do much.
To make this more concrete, check the StartProducer method:
Let’s go over a few key points.
We generate an Animal instance using the ProduceAnimal function:
We create random data using AutoFixture. The switch statement demonstrates how a specific animal type is produced using the OneOf type field.
The data field of the Cloud Event is populated with the binary content of the generated animal. The ToBinaryArray method is part of the C# class generated from the protobuf schema.
Also note how we set the Binary content mode when invoking the ToKafkaMessage function on the Cloud Event.
Consumer
The consumer part looks like this:
To convert the Cloud Event back into an Animal object, we simply use the ParseFrom method from the protobuf-generated Animal class. Next, we examine the AnimalTypeCase property to determine the specific animal type (Cat, Bird, or Dog) we are working with.
Running the App
Running the app should produce a result like this:
Summary
This article showcased how to produce and consume Kafka messages using Protobuf serialization in compliance with the Cloud Events specification.
Thanks for reading, and see you next time!
