Overview

• On 17th Dec 2025, there were reports of multiple Mainnet Bor nodes at RPC providers stuck at a few particular blocks in the range from 80440819 to 80443486 but the block production remained healthy with the rest of the network proceeding as usual.
• After an initial investigation, a version with a workaround was rolled out to allow these nodes to bypass the block range, enabling all nodes to get back in sync within a few hours of the incident.
• The root cause was found later, and corresponding fixes were made.

Root Cause Analysis

• The root cause of the incident was found to be due to double sealing of blocks during this time period, which resulted in longer block times.
• This, in turn, caused delayed finalization, which resulted in multiple span rotations with the same start-end block range, but different block authors (i.e. span ids 12771 and 12772 had 0x41018795fa95783117242244303fd7e26e964ee8and0x25b9fc2ed95bbaa9c030e57c860545a17694f90d as producers, respectively).
• The block producers switched mid-span when a span rotation was instructed by Heimdall due to late finalisation.
• Now the nodes that were even slightly lagging started rejecting the incoming blocks, which were produced according to span 12771, because the producer didn't match up with what was being expected from the updated span 12772.
• An important factor in this scenario was that only two block producers were online at the time, as the third was being migrated to a new instance with improved hardware.

Incident Timeline (UTC)

• 17th December 2025, 18:30:00 UTC: An alert was received indicating that multiple Mainnet Bor nodes at RPC providers are stuck in a short block range.
• 17th December 2025-18:55:00 UTC: An attempt was made to whitelist certain block hashes at particular blocks by using --eth.requiredblocks but since there were multiple blocks at which nodes were stuck, the decision was to ignore block author check wrt span for that block range.
• 17th December 2025-20:50:00 UTC: The PR for the hotfix was opened, and v2.5.6-beta3 was released.
• 17th December 2025-20:55:00 UTC: An initial investigation was done to figure out what caused the incident.
• 17th December 2025-22:07:00 UTC: v2.5.6-beta3 rolled out, and most nodes were back in sync within a few hours.

Follow-Ups

• An immediate fix to unstuck bor nodes was to ignore the block author as long as it was part of the Veblop validator set. This is the PR: Pull Requestfix: ignore invalid signer error for specific block rangeMerged
• The double sealing of blocks was causing higher block times (which in turn was delaying finalization). Ever since it was fixed, there have been no further inadvertent span rotations due to late finalisation. This is the PR that fixed: Pull Requestfix(miner): track pending work using block number to avoid duplicate block building workMerged
• There is a routine in Bor designed to prevent nodes from staying on the wrong fork after getting the latest span. The check verifies the non-finalized blocks against the latest span, and rewinds if a mismatch is found. However, this logic was never triggered due to the incomplete setup of the whitelist service. Fixed in this PR: Pull RequestFix missing chain validator service setupMerged
• Two more block producers were made online - one being migrated and another one that was added after the Heimdall hardfork on Dec 17th