Bug fixing in traditional software development is often likened to solving a difficult puzzle, each presenting its own challenges. This task has always been complex and time-consuming. However, resolving bugs in a blockchain system is even more demanding due to its transparent & permissionless nature and the high stakes involved with users’ funds. The intricacies of this task are significant and call for developers to think creatively and critically to come up with a solution. Let’s examine these complexities in more detail.
In blockchain, transparency refers to the public availability of the smart contract’s code. This unique characteristic enables anyone to review the code, scrutinize its logic, and identify potential bugs or vulnerabilities. The ability to access and examine the code fosters a collaborative environment where developers and security experts can come together to identify and address any issues they discover.
Blockchain systems are designed to be permissionless, meaning anyone can access and interact with the smart contract without requiring approval. This aspect can be both advantageous and challenging for bug fixing. On the one hand, it allows a wide range of white hat security experts to contribute to solving an identified bug while, in the meantime, opening up the possibility of black hat hackers exploiting it.
Bugs in smart contracts can put users’ funds at risk. If a vulnerability is exploited, users can suffer financial losses. This factor adds urgency and pressure to fix the bug promptly and effectively to minimize user impact and safeguard the protocol’s reputation.
In this third part of our article series, we infiltrate the Ethereum Virtual Machine with our team of seasoned blockchain security sentinels. Till now, we have delved deeply into the execution of code on the EVM’s world computer. We explored the intense environment of a war room where responsible experts tackle all critical conditions, with up to hundreds of millions of dollars at stake.
Cracking The Code: Bug Fixing On Blockchain
Identify and Select a Patch
Once a bug is identified, the development team will work on a fix. The war room engineers at this point in time will;
- analyze the code (localization/ detection) of the vulnerability,
- understand the nature of the bug (categorize it in a pre-defined severity), and
- create a patch or update (formulate a fix) to address the issue.
Let’s delve into the various approaches taken to resolve these elusive issues while deploying a bug fix in a mission-critical financial instrument coded in the smart contract on the blockchain.
- To identify the bug, refer to the previous article where we discuss the multiple scenarios of how a bug can be identified and fire up a war room where a team of experts treats it as an operation theatre, performing diagnosis and surgery for the exploitable (or already exploited
) bug. - Once the issue is confirmed, a list containing multiple possible fixes may be devised as a reflex action.
- When evaluating fixes from the list above, several factors impact the feasibility of selection, with the nature of the bug being the most critical. However, the bug’s severity also plays a significant role in determining the appropriate fix.
At this point, the security partners of the protocol may be consulted to evaluate and finalize the fix as a patch.
- Once a bug fix is finalized, we move on to the next phase, executing it as a transaction within the dark forest of the Blockchain.
Implementing the Bug Fix as a Transaction on the Blockchain
The bug fix for the identified vulnerability will be executed as a transaction on the blockchain. This transaction includes relevant information such as the smart contract’s address, the function signature to be called with the necessary parameters required for executing the fix, and the values supplied in the function arguments.
When the bug fix transaction is broadcasted on the blockchain, it enters the mempool, awaiting confirmation and validation from the network. At this stage, the transaction becomes visible to all participants on the network. However, the transparency and permissionlessness of blockchain acts as a double-edged sword.
All the while the transaction sits there, ready to be executed, and is visible to the public, the possibility of an attack increases by the passing second. Envision a zero-day at this point and multiply the intensity of damage to the protocol’s reputation by incorporating factors such as the users’ funds, the permissionless ness to interact with the smart contract infrastructure, and the transparency, providing ease-of-access to an adversary to understand and grab the problem to turn it around for their own benefit.
Say hello to Front-running
Front-running is a technique where an attacker observes a transaction before it is processed and executes a similar transaction with a higher gas price to ensure it gets prioritized. If the bug fix is not carefully crafted, an attacker may still be able to exploit the identified bug despite the fix attempt by a privileged goodwill user.
A whitehat hacker named @trust__90 found a vulnerability in a protocol that brought risk to almost 100 ETH of users’ funds. They tried to execute a white-hat rescue, only to get frontrun on the way and losing about 90% of the funds at risk by some MEV Bots.
Front-running at any cost?
When it comes to web3 security, front-running remains a persistent attack vector that continues to pose a significant threat despite being identified as a vulnerability. This leads us to many of the scenarios where not even bug-fix transitions are secured from it.
From a common person’s perspective, front running might involve obstructing or delaying transactions that directly involve money or may have some financial gain in the future. However, instances have occurred where attempts to fix vulnerabilities, like in the SirenMarketV1 protocol, were front-run, leading to the discovery of a new attack vector named Block-Stuffing Attack. The attacker submitted a large number of high-gas transactions, which the blockchain’s mempool environment rationally prioritizes for miner profit. This action essentially blocked the fix from being deployed for almost 30 blocks. Thus, the adversary’s will extended the 0-day vulnerability in this case.
Block stuffing is a technique where an attacker submits tons of transactions with a higher gas fee than an honest user. Miners will fill their blocks with just those transactions until the block gas limit is reached. Hence, an honest user gets completely ignored if they keep following a standard way of executing transactions on the blockchain* aka the public mempool.
(*Hint: Flashbots)
Front-running, is it something new?
Frontrunning refers to the unethical practice in financial markets where a trader or broker uses advanced knowledge about upcoming large buy or sell orders. This occurs when the frontrunner executes their own trades ahead of the client’s order, utilizing the imminent price increase or decrease to gain a profit. By front-running, the frontrunner essentially jumps in front of the client’s order and capitalizes on the anticipated price movement.
Context: In blockchain networks, transactions are typically held in a mempool, a pool of pending transactions, before being added to blocks through a consensus mechanism like proof-of-work or proof-of-stake. Miners or validators are responsible for selecting transactions from the mempool to include in the next block. Front running in blockchain occurs when a participant, often a watcher, miner, or validator, obtains information about a pending transaction. In frontrunning transactions are rapidly executing a competing transaction to capitalize on it. This is commonly done to exploit price fluctuations in DeFi platforms where the order of transactions on the blockchain determines transaction execution.
Already H4cked -> Negotiations
If an attacker successfully front-runs the bug fixing transaction and exploits the bug, it is crucial to negotiate with the attacker to mitigate the damage and protect users’ funds. In such cases, working with experts in the field and involving law enforcement might be necessary to resolve the situation.
Trust Security lays out a standard for the lines of defense in such scenarios where negotiations are inevitable. The main pointers from their piece for the context focus on the fact that “a sophisticated negotiation plan needs to be put in place to prepare for all eventualities.” And that a good strategy must take into consideration the following:
- “A strong monetary incentive to return the funds or to report the perpetrator.
- Psychological manipulation of the attacker through carefully constructed clues and bluffs.
- Setting up OpSec traps for the hacker to trip over.”
[icon name=”check-circle” prefix=”fas”] Circuit Breakers: Pause, Emergency, Exits, or actions that fall in the category are components that reduce the potential of overall loss and play n essential role in the line of defense. As stated in the linked article above, we find a relevant fact: “Limited-loss schemes are incredibly powerful because ideally, they are independent of the rest of the functionality.”
User Communications
If the attacker is not caught and the incident passes the initial response period, it is vital to handle the situation transparently. Users should be provided with detailed information about the attack, the bug, and the measures taken to resolve the issue. Additionally, steps should be communicated to prevent similar incidents in the future. Consideration should be given to compensating affected users to restore their trust in the smart contract and the development team.
In a war room, DevSecOps is a crucial part of the team. Their main responsibility is to manage patch transactions while being transparent with the affected community. Transparency is an essential blockchain component; trust cannot be effectively established in the wider blockchain landscape without it. Therefore, it’s necessary to replace trustlessness with transparency to ensure confidence in the blockchain.
Disclosure and Post-Mortem Report
After successfully patching the code, it is important to disclose the issue professionally and safely to the user base. A comprehensive post-mortem report should be shared detailing the bug, its impact, the steps taken to address it, and the measures implemented to prevent similar incidents. The team may also consider hiring third-party auditors to conduct regular security audits to minimize the likelihood of future bugs. These steps aim to ensure transparency, accountability, and the protection of users’ funds when addressing and bug fixing in a blockchain system.
TL;DR
Bug fixing in blockchain systems involves unique considerations due to transparency, permissionlessness, and users’ funds at risk. Once a bug is identified, the development team analyzes the code and creates a fix implemented as a transaction on the blockchain. However, the transparent nature of blockchain introduces the risk of front-running, where an attacker exploits the bug fix. Negotiation and collaboration with experts may be necessary in case of exploitation. Transparent handling, user communication, and compensation may be required to restore trust. After patching, a detailed disclosure and post-mortem reports are essential. Regular security audits can help prevent future bugs.
This is the third part in the Infiltrating the EVM series.
