When Kafka Can Lose an Acknowledged Record

Kafka stores a sequence of records across several servers so applications can publish events and other applications can read them later. A producer writes a record. The server currently accepting writes for that part of the sequence is the leader, and other servers keep follower copies.

At 10:00:02 a producer writes order 5502 and receives success. At 10:00:08 two servers fail. A stale third server becomes leader, reuses the same position for order 6011, and the acknowledged order disappears from every copy. This chapter follows that one partition byte by byte and then shows three other ways an acknowledged tail can be lost.

Kafka's documentation states the guarantee precisely: "a committed message will not be lost, as long as there is at least one in sync replica alive, at all times." Every loss in this piece comes from one of two places: "committed" meaning less than the producer assumed, or "at least one in sync replica alive" not holding.

For data that cannot be recreated, a common baseline is replication factor 3, min.insync.replicas=2, producers with acks=all, unclean.leader.election.enable=false, and eligible leader replicas on for current KRaft clusters. Spread the replicas across independent failure domains. The trade-off is deliberate: the partition stops accepting writes when it cannot keep two copies in sync.

Before the failures: the pieces in one picture#

A topic is a named stream. Kafka divides it into partitions, and each partition is an ordered log whose entries have increasing offsets. A broker is one Kafka server. For each partition, one broker hosts the leader replica and other brokers host follower replicas.

The producer sends each record to the leader. Followers copy the leader's log. The controller chooses a new leader when a broker fails. Modern Kafka stores controller metadata and runs controller elections through KRaft, Kafka's built-in consensus system, instead of ZooKeeper.

What "committed" means#

Each partition has one leader and some followers. Producers write to the leader; followers fetch from it. The leader tracks which replicas are keeping up. A follower that has not fetched recently enough is removed from the in-sync replica set (ISR).

The log end offset (LEO) is the number the replica will assign next. If the last stored record is offset 100, the LEO is 101. The high watermark (HW) is the first offset that is not committed; ordinary consumers can read records below it. This “next offset” convention avoids an off-by-one trap: HW = 101 means offsets through 100 are committed.

The important word is "currently". The ISR is not the set of all replicas; it shrinks when replicas fall behind and grows when they catch up. Here is a partition with a replication factor of 3 at one moment:

Here ISR = {0, 1} and HW = 101. Offsets 81 through 100 are committed because both current ISR members have them. Broker 2 does not. Every loss path below reuses partition orders-7, brokers 0–2, and this range of offsets.

The three settings#

With min.insync.replicas left at its default of 1, acks=all can mean “acknowledged by one machine.” The producer waits for every current ISR member; the minimum setting decides whether an ISR of size one is allowed to acknowledge at all.

Managed services choose different defaults, so check yours. Confluent Cloud sets min.insync.replicas to 2 and doesn't allow unclean election to be enabled. AWS MSK sets min.insync.replicas to 2 on three-zone clusters but 1 on two-zone clusters, and leaves unclean leader election enabled on clusters without tiered storage.

The table below is a starting policy, not a substitute for testing the availability and recovery needs of one workload.

Loss 1: acks=1 and leadership moves before replication#

With acks=1, the leader acknowledges as soon as the write is in its own log. The producer documentation describes the consequence directly: "should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost." Failure is one trigger; a planned leader move can expose the same gap. The clean follower takes leadership without the acknowledged record, and the old leader truncates its divergent tail when it rejoins.

Disabling unclean election doesn't help here, because nothing unclean happened: the new leader was in the ISR. An open Kafka issue reports the same loss during planned leader changes on Kafka 4.1.2 with acks=1, and none with acks=all under the same test (KAFKA-20554). acks=1 has been the wrong default for anything you can't afford to lose since Kafka 3.0 changed it, but producers configured before then, or copied from old examples, still use it.

Loss 2: the ISR shrinks to one#

acks=all waits for every replica in the ISR, and the ISR can contain only the leader. Kyle Kingsbury's Jepsen analysis in 2013 demonstrated this on a three-node cluster using the equivalent of acks=all: the ISR shrank to the leader, which then acknowledged writes stored on that broker alone. When that leader was partitioned away, another node was promoted and the acknowledged writes stayed with the old leader.

In Vanlightly's second test, acks=all lost more than acks=1 did because a one-broker ISR still satisfied “all.” The fix is min.insync.replicas=2 with a replication factor of 3: the leader refuses writes (with a retriable error) when fewer than two replicas are in sync, so every acknowledged write is on at least two machines. The price is availability: with two of three replicas out of sync, that partition stops accepting writes until one catches up.

Loss 3: unclean leader election#

Now the case the setting is named for. Start from the partition above, with acks=all and min.insync.replicas=2, so offsets 81 to 100 are on brokers 0 and 1.

The consumer API's documentation for the resulting exception puts it bluntly: "In the event of an unclean leader election, the log will be truncated, previously committed data will be lost, and new data will be written over these offsets." That last clause is what makes this worse than a simple loss. Any system that stored "processed up to offset 100" now has a position that refers to records it has never seen.

With unclean election disabled, broker 2 cannot take leadership while brokers 0 and 1 are unavailable. The partition stays offline and producers receive errors. Kafka's design documentation frames this as a choice with no free option. Jay Kreps, then at LinkedIn, put it more memorably in his response to the Jepsen post: "Is it better to be alive and wrong or right and dead?" Kafka changed the default to "right and dead" in 0.11, in 2017.

Datadog experienced temporary data loss after an unclean election on a cluster running a version older than 0.11; a second cluster held the recoverable copy. Honeycomb's report on a December 2025 incident describes the other side: cleanup after a disaster-recovery exercise destroyed brokers and left partitions with no leader. The resulting dirty elections could roll offsets back. Roughly a third of partitions were affected and one lost all of its data, on a cluster spread across three availability zones.

If you ever do need availability back more than you need the lost tail, make it a deliberate, one-time action rather than a setting: kafka-leader-election.sh --election-type unclean runs one unclean election for the partitions you name (KIP-460 added it because the setting is easy to forget to turn off again). On KRaft clusters, the setting itself, when enabled dynamically, only takes effect when a periodic check runs, every five minutes by default.

Loss 4: the last replica standing loses its page cache#

Here is the case that surprises people who have done everything above correctly. Kafka writes to the operating system's page cache and, by default, never forces data to disk: log.flush.interval.messages defaults to the largest possible number. The documentation explains the choice: "we do not want to require the use of fsync on every write for our consistency guarantees as this can reduce performance by two to three orders of magnitude." Durability comes from replication instead: a crashed broker recovers the data it lost from the other replicas.

That works as long as some replica still has the data. The design proposal for Kafka's fix, KIP-966, describes what happens when it doesn't:

In KIP-966's words, the "last replica standing" state "when combined with a data loss unclean shutdown event can turn a local data loss scenario into a global data loss scenario, i.e., committed data can be removed from all replicas." Kafka's issue tracker has reports of exactly this, including a last in-sync replica that came back with an empty disk and took the partition's data with it (KAFKA-15495).

The fix is eligible leader replicas (ELR), part one of KIP-966. It changes two rules: the high watermark only advances while the ISR has at least min.insync.replicas members, and replicas that were in sync when the ISR shrank stay eligible for leadership. After an unclean shutdown, the controller can elect an eligible replica that still has the data instead of the one that lost it. ELR became available in Kafka 4.0 and is enabled by default for new clusters from 4.1. When ELR is enabled, min.insync.replicas must exist at cluster level, cannot be removed, and cannot be altered at broker level; changing the cluster or topic value clears tracked ELR state. With min.insync.replicas=1, ELR has no extra replica to preserve.

The other defence is physical. A process crash doesn't lose the page cache; losing power or crashing the operating system does. Jack Vanlightly's analysis of why Kafka doesn't need fsync makes the dependency explicit: "if all brokers lose power simultaneously then there is a risk of data loss." Spreading replicas across independent failure domains reduces the chance that one power or host event removes every cached copy; it does not protect against region-wide or operator-correlated loss. For the few topics where even that isn't enough, the flush settings can force writes to disk, at a large cost in throughput, and an independent replay source or archive may still be required.

Kafka's fixes over time#

This history was checked against Apache Kafka's KIPs and current 4.3 documentation in September 2026. Kafka 4.3.1 is the current stable patch. Some entries changed a default; others changed the replication protocol, so a client setting alone cannot supply the fix.

Seeing it happen#

The names below are complete JMX object names from Apache Kafka's monitoring documentation, checked in September 2026. The count metrics should normally be zero. A rate can briefly become non-zero during a planned broker event, so alert thresholds should match the recovery time your cluster has demonstrated.

Consumers show the same events from the other side. LinkedIn's engineers wrote in 2016 that "one typical cause of offset resets is unclean leader election", and that "it is a good practice to monitor the cluster's unclean leader election rate." A sudden offset reset on a consumer group is worth treating as a possible data-loss event until proven otherwise.

A checklist#

Sources#

• Apache Kafka 4.3, Design: replication and unclean leader election, broker configs, producer configs, monitoring, eligible leader replicas, and LogTruncationException
• KIPs: KIP-101, KIP-106, KIP-279, KIP-320, KIP-679, KIP-966
• Kyle Kingsbury, Jepsen: Kafka (2013); Jay Kreps, A few notes on Kafka and Jepsen (2013)
• Jack Vanlightly, How to lose messages on a Kafka cluster, part 2 (2018), KIP-966: fixing the last replica standing issue (2023), and Why Apache Kafka doesn't need fsync to be safe (2023)
• Datadog, Lessons learned from running Kafka at Datadog (2019); Honeycomb, Incident report: exercises, cleanups, and evacuations (2026); LinkedIn, Kafkaesque days at LinkedIn, part 1 (2016)
• AWS, MSK default configuration; Confluent, Manage topics in Confluent Cloud