Prysm has revealed that a bug introduced in a testnet a month before Ethereum’s Fusaka upgrade was the cause of an Ethereum node validation issue that affected its client earlier this month.
Ethereum developer Terence Tsao posted a post-mortem on Sunday detailing the Fusaka mainnet Prysm incident that impacted the network on Dec. 4.
Prysm nodes experienced “resource exhaustion” when processing attestations from out-of-sync nodes, it stated. This caused Prysm to replay past epoch blocks and recompute expensive state transitions, resulting in a significant impact on performance due to the excessive workload.
The post-mortem revealed that the bug had been present on testnets for a month before the incident, but wasn’t triggered.
“The bug was introduced in Prysm PR 15965 and deployed to testnets a month before the incident without the trigger happening.”
Testnets are designed to identify bugs, but they aren’t a foolproof method.
In May 2023 — a month after the Shanghai hard fork — Ethereum developers were sent into a frenzy when the network temporarily lost transaction finality for around 25 minutes, then again for over an hour the following day, before the blockchain recovered on its own.
Prysm has been patched
Instead of using the current head state, Prysm regenerated prior states from scratch, creating a massive computational burden.
For more than 42 epochs, the network saw an 18.5% missed slot rate with participation dropping to 75% while validators lost approximately 382 Ether (ETH) in attestation rewards, it said.
Related: Vitalik Buterin says Ethereum can handle temporary loss of finality
Node operators were instructed to deploy a temporary solution while developers worked on an update patch for Prysm clients.
Client diversity saved the day
The incident could have been much worse if it had targeted Ethereum’s dominant consensus client, Lighthouse, developers said.
Offchain Labs’ Prysm is the second-largest Ethereum client with a 17.6% share, according to ClientDiversity.
“Client diversity prevented a noticeable impact on Ethereum users. A client with more than 1/3rd of the network would have caused a temporary loss in finality and more missed blocks.”
However, the incident highlighted that Lighthouse is dangerously close to the two-thirds threshold where a single client bug could finalize an invalid chain.
Lighthouse currently has a client share of 52.6%, down from around 56% at the time of the incident.
Magazine: Big questions: Would Bitcoin survive a 10-year power outage?