Reusing serverId Across Process Restarts Can Cause Incorrect Mastership Decisions

[shivamka1]

Summary

In the current leader-election example, serverId may be generated from a deterministic source such as:

String serverId = Integer.toHexString(random.nextInt());

where random is initialised using this.hashCode().

zookeeper-book-example/src/main/java/org/apache/zookeeper/book/Master.java

Line 107 in 135b79b

private Random random = new Random(this.hashCode());

This can cause the same serverId to be reused across application restarts.
If the process restarts quickly (before the previous ZooKeeper session has expired), the new instance may incorrectly believe it is already the master.

This leads to an inconsistent application state where:
• ZooKeeper still considers the old session alive,
• the ephemeral /master node still belongs to that session,
• but the new process thinks it owns the mastership because the stored serverId matches its own. ‼️

Steps to reproduce

ZooKeeper binds ephemeral nodes (like /master) to sessions, not to processes.

When Process A becomes the master:

1. It opens session S1.
2. It creates /master (ephemeral) with data X.
3. It crashes, but ZooKeeper still keeps session S1 alive until the session timeout expires.

If Process B restarts and also gets serverId = "X":
• It calls checkMaster().
• It sees /master with data X.
• It mistakenly concludes “I am the master.”

But in reality:
• /masterstill belongs to the dead process’s session S1.
• B does not actually hold the lock.
• There is no real leader during this window.

This is a subtle but serious correctness problem.

Impact: New instance may skip leader election entirely because it believes it already won.

Expected Behaviour

A restarted process must always generate a new unique contender identity so that:
• If /master exists and contains a different ID, the new process correctly concludes that the old session is still considered alive.
• If /master is eventually deleted after session expiration, the new process can safely run for master again.

Proposed Fix

Replace any deterministic or repeatable serverId with a per-process unique value, for example:

String serverId = UUID.randomUUID().toString();

This ensures:
• Each process run has a unique identity.
• checkMaster() can correctly distinguish between:
• “I created /master”
• “The previous instance created /master, and ZooKeeper has not expired its session yet.”

[shivamka1]

One could argue that when a new process starts, it always begins by attempting to create the /master znode, so it should never mistakenly assume it is already the master. That part is correct: the initial election step does not cause the problem.

The real issue appears only in the ambiguous failure case, when create("/master", …) throws a ConnectionLossException. In this situation the client must call checkMaster() to determine what actually happened, because it does not know whether the create operation reached the ZooKeeper server or not.

If serverId is reused across process restarts, the new process may observe that:
• /master still exists (because the old session has not yet expired), and
• the znode’s data matches its own serverId.

Based on this comparison, the process incorrectly concludes that it is the master, even though the ephemeral node still belongs to the old session.

This is why serverId must be unique per process instance (for example using UUID.randomUUID()), so that checkMaster() can reliably distinguish between the current process and any previous instance whose ephemeral nodes are still present.