Intro

In the previous article, we explored when Eventual Consistency is not enough and the challenges we might face in a distributed system due to data replication.

You’ve seen concrete violations of the most popular Causal Consistency Guarantees – Read Your Write, Monotonic Reads, Monotonic Writes and Writes Follow Reads.

This post will be more practical as I’ll provide some actual code examples that break Causal Consistency Guarantees in MongoDB.

We’ll also make an initial attempt to fix that by using “majority” reads and writes.

This will prove insufficient, which will lead the discussion towards Logical Clocks and  Causally Consistent Sessions in Mongo, which I’ll review in the following articles.

For the coding part here, I will use the Mongo C# driver, but that’s just a personal preference. You can achieve the same in your favorite environment.

The version of Mongo I’m using is 5.0

“Read Your Write” Violation – Recap

I would strongly suggest visiting the previous article to see much more in-depth use cases, but at this point, it’s enough to understand the workflow from the diagram below:

In essence, the Client performs a Write operation to the Primary server. Then he tries to read the same Write, but the Read request goes to the Secondary. The Write is still not replicated to the Secondary, so the Client cannot read his write.

It’s time to replicate this in Mongo.

Fundamentals of Database Engineering

Learn ACID, Indexing, Partitioning, Sharding, Concurrency control, Replication, DB Engines, Best Practices and More!

Code Example

Again, the code below is in C#, but it’s simple enough to reproduce in whatever language you prefer.

Spend a moment to review the following snippet:

static void Main(string[] args)
{
    var client = new MongoClient("mongodb://localhost:27018,localhost:27019,localhost:27020/?replicaSet=rs0");
    var collection = client.GetDatabase("test-db")
        .GetCollection<BsonDocument>("test-collection")
        .WithReadPreference(ReadPreference.Secondary);

    var sw = new Stopwatch();
    sw.Start();
    while (sw.ElapsedMilliseconds < 5000)
    {
        var newDocument = new BsonDocument();

        collection.InsertOne(newDocument);

        var foundDocument = collection.Find(Builders<BsonDocument>.Filter.Eq(x => x["_id"], newDocument["_id"]))
            .FirstOrDefault();

        if (foundDocument == null)
            throw new Exception("Document not found");
    }
    
    Console.WriteLine("Success!");
}

This program, in most cases, will output an exception:

Unhandled exception. System.Exception: Document not found

Here’s a brief description:

1. We connect to a Mongo replica set with three servers running on ports 27018, 27019, and 27020.
2. We specify that the reads should go to a Secondary server.
3. Then in a loop, we keep inserting a document and trying to read it right after that. This exactly replicates the workflow from Fig. 1
4. At some point, the document (most probably) won’t be found because the replication to the Secondary hasn’t caught up yet.

Note that I’ve set a limit of five seconds for the while loop as it has been more than enough to reproduce the issue “on my machine.”

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems [Kleppmann, Martin] on Amazon.com. *FREE* shipping on qualifying offers. Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

First Try – Majority Writes and Reads

A “Majority Write” means that a Write operation will be reported successful to the Client only if it has been propagated to the majority (*) of the servers in the replica set.

(*) For simplicity, let’s say that a “majority” of servers means one plus half the number of the servers, rounded down. For example, a majority of 4 would be 3, a majority of 7 would be 4, and so on. In reality, some secondaries would not be used to calculate the majority, but that’s irrelevant for this discussion.

A “Majority Read” dictates that the data returned from a query has been acknowledged by a majority of the replica set members.

The majority reads and writes can be used in Mongo by specifying “majority” Write Concern or “majority” Read Concern for your query or globally.

Now, it might be tempting to think that using a Majority Write followed by a Majority Read would solve our “Read Your Write” problem.

Not really.

Let’s give it a try:

static void Main(string[] args)
{
    var client = new MongoClient("mongodb://localhost:27018,localhost:27019,localhost:27020/?replicaSet=rs0");
    var collection = client.GetDatabase("test-db")
        .GetCollection<BsonDocument>("test-collection")
        .WithWriteConcern(WriteConcern.WMajority)
        .WithReadConcern(ReadConcern.Majority)
        .WithReadPreference(ReadPreference.Secondary);

    var sw = new Stopwatch();
    sw.Start();
    while (sw.ElapsedMilliseconds < 5000)
    {
        var newDocument = new BsonDocument();

        collection.InsertOne(newDocument);

        var foundDocument = collection.Find(Builders<BsonDocument>.Filter.Eq(x => x["_id"], newDocument["_id"]))
            .FirstOrDefault();

        if (foundDocument == null)
            throw new Exception("Document not found");
    }
    
    Console.WriteLine("Success!");
}

When you run this, you’ll again encounter the same “Document not found” exception.

What’s Still Wrong?

The Majority Read and Write Concerns only ensure that the data written or read is durable and will not be rolled back during a failover.

It doesn’t guarantee that a Majority Read on a Secondary will return the latest Majority Write you applied to the Primary.

Here is why.

Every Secondary server keeps an in-memory snapshot of the most recent Majority Write it’s aware of.

Let’s make that clear with the diagram below:

Here’s a summary of the steps in this workflow:

1. The Client triggers an insert of a new document with a “majority” Write Concern.
2. The Primary server accepts the Write and propagates it to the Secondary.
3. The Secondary applies the Write and sends an acknowledgment to the Primary.
4. At this point, the Secondary contains the inserted document, but it’s not part of its most recent Majority Write Snapshot. The Primary sends a notification to the Secondary (via regular replication mechanism) to update its snapshot.
5. The Client receives a message that the document is inserted successfully with _id “abc.”
6. He then sends a Majority Read request to the Secondary to read the inserted document.
7. The Secondary still hasn’t advanced its’ Majority Write Snapshot, so document “abc” is not found.

Summary and What’s Next

In this article, you saw a concrete code example for violating the Read Your Write concern in Mongo.

We’ve discussed the “majority” Read and Write concerns and naively tried to fix our consistency problem.

Then, we’ve touched on the topic of “majority snapshots” and how the secondary replicas get updated with the latest “majority” data.

In the next articles, we’ll fix the Read Your Write violation using Causally Consistent Sessions.

This will walk us through the concept of Logical (Lamport) Clocks, Cluster Time, and Operation Time in Mongo.

Stay tuned, and thanks for reading!

Resources

1. Designing Data-Intensive Applications
2. Introduction to Database Engineering, Udemy