Ape has you covered: All about coverage Part 2

Before you start delving into the practical aspects, it is essential to understand the concept of code coverage. If you are unfamiliar with coverage, we recommend referring to our first article on the subject.

The latest update, version 0.6.13 of Ape and version 0.6.9 of ape-vyper, introduces the coverage support exclusively for Vyper contracts. However, since Ape is a plugin based framework, this system facilitates adding coverage support for other languages, such as Solidity and Cairo.

So without further ado, lets dive in:

Demonstrating Coverage with a Token Contract:

In this article, we are going to employ a Token contract as an illustrative example to showcase the functionality of coverage. You can follow this link here for the repository.

How to use Ape Coverage

Utilising coverage functionality within the Ape framework is straightforward. You can initiate coverage by including the `--coverage` option when running `ape test`. This will generate a comprehensive coverage report. It is worth noting that certain types of coverage will require a provider plugin that supports transaction tracing such as `ape-hardhat` or `ape-foundry`.

https://docs.apeworx.io/ape/latest/userguides/testing.html#contract-coverage

This is our token contract for this example:

Currently we have not set any tests for this contract so our coverage will show  0%.

Ape shows the name, statements (Stmts), misses (Miss), coverage (Cover), and functions (Funcs). This is modelled after coverage-py and pytest-cov style reports.

• Name: Name of the source identifier

• Statements (Stmts): Amount of statements present in each contract

• Misses (Miss): Amount of statements not covered in tests

• Coverage (Cover): Percentage of the contract that is covered by tests

• Functions (Funcs): Percentage of functions/methods that are tested in each contract

A statement is either a group of values that have the same line numbers in a source file or it is a value with an extra tag to identify builtin logic that is injected by the compiler (such as the default or fallback method).

The coverage report is normally generated in the terminal but you can also generate the coverage reports as an external XML or HTML files by configuring them in the `ape-config.yaml` like:

Thorough Code Coverage

You can also see a more verbose coverage report by setting the field `terminal` or `html` to include `verbose: True`. Verbose outputs are useful when you are trying to find the missing areas to cover. This will break down each contract to show each function and how well the functions are covered individually.

This will result in the following output for our contract `TestToken.vy`:

The `_*builtin*_` functions listed under your contract coverage functions are the functionalities that the compiler uses internally that are not in the source code, such as non-payable checks, safe-math checks, and the not-implemented contract default method check. Tests for these conditions are necessary to get to 100% coverage

Excluding Methods and Contracts

Like gas reporting, you can also exclude contracts and methods from tracking coverage using your `ape-config.yaml` file.

You may want to use exclusions for methods or contracts that you don’t plan on pushing to production but are still using for debug or testing purposes.

We will be excluding the `DEBUG_mint` function from testing.

Adding Coverage

Now let’s try and add some coverage for our `TestToken.vy` contract.

Some methods will show zero statements. One example with a zero statement may be from an auto-generated getter method for a public variable: certain versions of Vyper do not contain source mappings for these methods. However, Ape will still check to see if this method has been called in your tests. To get 100% coverage, you must call these methods in your tests.

In this case, these will be methods like `name()`, `symbol()`, `decimals()`, `totalSupply()`, `balanceOf()`, and `allowance()`. A simple example of tests that you can make for these are:

By just adding this test, the coverage report for the TestToken contract will now show:

So now all we need to do is to add tests for the other functions: `approve()`, `transfer()` and `transferFrom()`

By incorporating these small tests, our test coverage will now reflect that we have 100% coverage for these functions.

However, it is crucial to recognise that achieving full function coverage does not guarantee thoroughness in testing your contract.  “Are these tests adequately comprehensive for my contract?” While the introduced test may offer substantial coverage, it is important to acknowledge that they may not encompass every possible scenario in which your contract may execute. Even with diligent testing and high code coverage, there is always room for enhancement. The journey towards code perfection is a continuous one.

Currently we only support statement coverage right now, if you were to add branch coverage, it would show less than 100% for total coverage.

One more thing to note is that some methods may have a full method signature while others do not. Methods that have the full signature mean that their short name is shared with other methods. This happens in Vyper from auto-generated kwarg-based methods, or in Solidity when overloaded functions are added. Thus, the full selector is used to distinguish the methods in the coverage (and gas) reports.

Using Coverage Responsibly

Remember in our previous article that we said “Code coverage metrics, as a percentage score, are really telling you very little about whether your code is doing the right thing.” Writing an abundance of tests is not the sole objective.  Instead, it is crucial to focus on crafting tests that align closely with your specific objectives, ensuring optimal suitability and effectiveness. The coverage tool serves as a means to gauge the extent to which your code has been thoroughly exercised with these examples, or guide you towards areas that need more attention.

We trust that this article has provided valuable guidance on harnessing Ape's new coverage feature to prepare your contracts for production deployment. It is important to remember that while coverage is an invaluable tool, it should be supplemented with other methodologies to ensure the overall safety and security of your contract.

Consider complementing code coverage with techniques such as fuzzing, red teaming, and exploring unconventional attack vectors. By adopting a holistic approach to testing and security analysis, you can fortify your contract against potential vulnerabilities and mitigate risks effectively.

Embrace the power of comprehensive testing strategies and exploit the full potential of code coverage as a part of your robust development and deployment pipeline.

Thank you for reading and enjoy our coverage tool!

- ApeWorX

For the latest on all things Ape follow us on: Apeworx.io | Discord | Twitter | Bluesky