Because decentralized finance (DeFi) protocols and smart contracts now manage more than $100 billion in digital assets, security concerns have increased. Traditional human audits of smart contract code are time-consuming, costly, and can miss vulnerabilities. As a result, teams are exploring automated defenses that can operate faster and at scale across many contracts.

To address these issues, OpenAI, in partnership with Paradigm (a crypto investment firm) and OtterSec (a blockchain security company), has introduced a testing framework known as EVMbench: Evaluating AI Agents on Smart Contract Security. This benchmark has been designed to evaluate how effectively AI agents can identify, fix, and even exploit security weaknesses in smart contract code in realistic environments.

The EVMbench framework uses a dataset of 120 vulnerabilities derived from real-world smart contract audits and research events. These vulnerabilities come from multiple prior audits and represent common security flaws encountered in DeFi codebases. The benchmark tests AI agents by placing them in controlled blockchain environments where they must analyze, detect, and react to these vulnerabilities.

AI Agent Roles in Security Testing

Under the benchmark:

• AI agents first scan smart contract code to detect potential weaknesses.

• If a vulnerability is found, they attempt to patch or remediate the issue without interfering with legitimate functionality.

• If the fix is incomplete or incorrect, agents then simulate attacks against the same vulnerabilities to confirm whether weaknesses remain.

Results from these tests indicate that AI systems are currently more effective at detecting vulnerabilities than at safely fixing them. In particular, they identify weak points in contract logic at higher rates than they correctly patch the code without breaking other functions.

Separate reporting on the benchmark’s detailed performance confirms these trends. In real test setups where agents were given full autonomy on local blockchains:

• Some AI agents successfully exploited a high proportion of vulnerabilities on their own.

• Fix rates were significantly lower than detection rates, especially when agents did not know where vulnerabilities were located in large codebases.

These findings show that while current AI tools can efficiently identify risk points, the ability to safely and automatically remediate vulnerabilities remains in development.

Potential Uses and Risks

Frameworks like EVMbench are intended to provide a standardized way to measure and compare AI performance on security tasks relevant to blockchain and DeFi environments. This helps developers assess whether different systems can meaningfully contribute to reducing the frequency and severity of exploits affecting smart contracts.

However, tests have also highlighted another dimension: the same AI capabilities that help defense could be used offensively. Because these tools can identify deep vulnerabilities quickly, there is potential for attackers to repurpose them for hostile exploitation if safeguards and access controls are not enforced.

The benchmark release coincides with broader interest in using AI agents as part of live monitoring and security operations for blockchain assets. Developers are examining how such agents might automatically watch deployed contracts and flag emerging threats faster than manual methods.

Technology and Industry Context

EVMbench’s tests draw on vulnerabilities curated from dozens of prior audits and real-world smart contract code analyses overseen by security firms and independent researchers. These datasets are used to simulate both defensive and exploitative actions by AI systems.

With smart contract platforms handling large volumes of financial activity, automated tools that can process complex code and respond quickly are seen as a logical next step for scalability. The benchmark initiative reflects industry efforts to quantify and track progress in these AI capabilities over time.

Source: The News Crypto