Subscribe to pornstache.eth
Subscribe to pornstache.eth
Share Dialog
Share Dialog
<100 subscribers
<100 subscribers
This is the 2nd part of the long article on blockchain that I divided up. This article discusses auditing, how the process works, what auditors look for, and what you should be learning if you want to get into auditing smart contracts.
From @SpearBitDAO - @jakelynch
To start, we set up a private discord channel with you and we'll have someone from our audit team speak with you to figure out the scope and provide a quote. We'll also figure out a time and who will be staffed on the audit. Since we're just starting out, we're fairly selective about what we're auditing atm. Once we engage, we'll take a deposit and at that point you'll give the auditors access to the codebase. Then they'll do their job and notify you of any found vulnerabilities while advising you on how to fix any vulnerabilities.
If a vulnerability requires an extensive fix, we reserve the right to ask for a separate engagement. Our rule of thumb for now is about 80% of the week spent on audit and 20% spent on helping fix vulnerabilities. Once finished, we provide a report.
You should use msg.sender for authorization (if another contract calls your contract msg.sender will be the address of the contract and not the address of the user who called the contract). It's also worth mentioning that by using tx.origin you're limiting interoperability between contracts because the contract that uses tx.origin cannot be used by another contract as a contract can't be the tx.origin. https://ethereum-contract-security-techniques-and-tips.readthedocs.io/en/latest/recommendations/#avoid-using-txorigin
From my experience, it's either:
access control mishaps
order of operations error, leading to flash loan or re-entrancy vulnerabilities
issues where two different parties can be the same, like self-transfers or self-liquidations, sometimes break accounting because they operate on cached values in memory instead of always reading the latest one from storage
batch functions that don't work correctly when passing in a batch array with duplicate values
trusting user input
or a combination of flash loans + oracle manipulation/re-entrancy
Flash loan vulnerabilities can become pretty advanced and require a security mindset which I think most devs don't actually have. From my experience most devs don't create threat models for flash loan/oracle manipulation vectors + thats when they get auditors to look into it.
You can see on https://rekt.news/leaderboard/ most have been unaudited and get attacked by one of the listed points.
Q) What general advice would you give to aspiring blockchain bug bounty hunters?
A) A great way to get started learning about smart contract exploits is to practice by participating in wargames and CTFs. You can find some good ones in https://github.com/crytic/awesome-ethereum-security 's repository.
Also, do not get discouraged if you haven’t found a bug yet, even though you have spent a lot of time searching. Often you will be working with code that has been audited by world-class security experts and put through extensive testing. That does not mean that there are no bugs, though — an audit is no guarantee for security and the list of audited projects which have been exploited is long enough.
In fact, both ArmorFi and Fei Protocol were audited. This is the reason Immunefi exists in the first place. Yet, it is easy to get lost in the fallacy of feeling like you are wasting your time when you spend hours upon hours while apparently not achieving any result. In these moments, it is good to remember yourself that even in periods of low or no findings, the knowledge and insight you are gaining by studying and trying to exploit smart contracts will in the long run equip you with a skillset only very few people have, which in itself will be more than rewarding.
Data from 16 case studies of 2020 DeFi exploits show that 1) 72.3% of hacks come from the Financial Model: e.g. interplay between bonding curve and constant product amm, Balancer hack 29/06/2020, 2) 27.4% from Insecure Implementations: e.g. re-entrancy & 3) 0.3% from Arbitrage.
Why is this so? Insecure implementations have been well-studied by past works, so many existing auditing tools can catch them (i.e. slither). But none of the existing tools can check for financial model unsoundness!
Look for:
Before: Control Flow (e.g., if, return, require)
Before: Memory variables created AND used after
After: Storage rights
Auditing Approach
While auditing, always keep an open communication channel with the developers to make sure you can ask questions if you are not sure about what they intend something is meant to do.
Write test cases/fuzzing for the most critical functions.
If the dev patches/fixes are small and you are able to verify the changes you can continue, otherwise if the changes are large and change a good portion of the codebase, the current audit is essentially void and a new auditor should be used since reviewing a new codebase in such a small period of time wont be effective.
If the math is too complicated, beyond comprehension, tell the devs and recommend they find someone else suitable.
Informational adds: gas optimization, linting, unnecessary code.
Read about the project to get an idea of what the smart contracts are meant to do. Glance over all the resources about the project that were made available to you.
Glance over the smart contracts to get an idea of the smart contracts architecture. Tools like Surya can come in handy.
Create a threat model and make a list of theoretical attack vectors (thinking, "how would an attack try to exploit this contract?") including all common pitfalls (i.e., before Solidity 0.8.0, overflows and underflows weren't reverted and SafeMath.sol was required) and past exploit techniques (re-entrancy, flashloans, pool balance manipulation, etc).
Look at places that can do value exchange, also checking if there are missing role requirements, allowing you to review the most critical functionality within a few hours. Especially functions like transfer, transferFrom, send, call, delegatecall, and selfdestruct. Walk backward from them to ensure they are secured properly (useful when viewing large codebases that have thousands of lines to review). Brain-storm new attacks for each smart contract - usually a day for each contract - creating proof-of-concept attacks (i.e., change a param to a certain something, going a different direction with the function flow, etc)
Do a line-by-line review of the contracts. Looking for logic bugs (where the contract is not behaving in a way it is supposed to, i.e., the whitepaper and smart contract don't align) and making sure the contract is safe from the exploits listed earlier.
After telling the devs about the review, do another round of review, however not as comprehensive.
What are the 3 first things you look at in Smart Contracts?
Architecture
Access control
Fund transfers
Q) Do you have any advice for people who would like to go into this area of using formal verification on blockchain protocols and smart contracts? E.g. what are the pre-requisites? And if one is interested, what are the companies that we can apply for?
A) I think learning how to use the FV tools to their best, and a bit of how they work, will give you both practical experience in the topic + give you pointers to learn more about the theoretical aspects if you're interested. As far as companies go I can't really say specific names, but I guess companies that use FV for their audits.
As a very quick thought, accesses that public/external functions give, unsafe external calls and assembly.
Smart contract audits should not be used as stamps of approvals. Words like "Passed audit" are a big red flag. Audits are basically detailed code reviews. They do not guarantee security. It's very hard to quantify/standardize such a process. This is why there are so many shitty audit firms around. I don't expect reputable firms to take "personal loss" in case of accidents. That's a completely different product offering (Insurance). However, I do expect Insurance providers and Code auditors to work closely in the future.
Q) What CTFs/war games/materials do you recommend doing for someone aspiring to take your #1 spot on Code4rena?
A) I started out doing Damn Vulnerable Defi https://cmichel.io/damn-vulnerable-de-fi-solutions/, Ethernaut https://cmichel.io/ethernaut-solutions/ and Capture The Ether https://cmichel.io/capture-the-ether-solutions/. The links here include my solutions if you get stuck but try to do it on your own first. Another great way that got me in contact with other auditors and all major auditing firms was Paradigm CTF https://cmichel.io/paradigm-ctf-2021-solutions/
Q) What tools do you use when auditing?
A) I don't use any tools that directly perform vulnerability analysis. I only use the great "Solidity Visual Developer" VSCode extension that highlights storage variables and function parameters which makes it easier to have some context when reading a new codebase.
Q) How do you calculate the audit's cost?
A) Scoping audits is always a tough task and imo you only get better at it with experience. But I'll give you a rule of thumb: Let's say you can audit 200 lines of code per hour (which is a standard assumption among some auditors afaik) - adjust that parameter down if the code is complex, math-heavy or the documentation is bad. You then take the lines of code and divide it by 200 LOC/h, then you get the hours required to audit the code for a single person. If you're an independent auditor you should also add 5h-10h for compiling the report and all the biz-dev work, plus answering questions. Then you multiply it by your hourly rate. More on hourly rates below.
How much time do you spend on a contract on average till you "stop" looking at it, i.e. when new C4 projects launch?
How much do you use tools and how much is really looking at the code and manual work?
How do you make sure that your possible exploit really is possible when most C4 code is not deployed yet?
And: Do you have N/A too or do you report only 100% sure vulns?
That's a good question. I could always spend more time on the code and it would increase the likelihood of me finding bugs. But at some point, there's the point of "diminishing returns", where it's not reasonable to spend any more time on the code. Alexander Schlindwein was asked the same question regarding bug bounties but I think it applies to audits as well: The approach which works best for me is to set myself the goal of fully understanding the system to the point where I could reimplement it from scratch without being allowed a look at the original codebase. Not from remembering the code, but from having understood what the application is supposed to do. If you have examined a project that far and have not found a bug, the chances of finding one by continuing is low. https://medium.com/immunefi/interview-with-legendary-bug-bounty-hunter-alexander-schlindwein-cced9c73c02a Realistically, I often stop before reaching that point due to time constraints and opportunity costs when I think my limited time is better spent elsewhere.
100% manual work for me
I definitely have N/As as well as I cannot code up a reproduction in a test framework for every issue I find. It would take too much time and I only do it for complex critical issues. Personally, I also think it's better to err on the side of "this looks like an issue to me, so I'll submit it" than "not sure if it's an issue, I keep it to myself". I've had devs tell me that even though my submission was wrong, they feel more confident in their code now because they reviewed the issue again. I'd like to stress though that I really appreciate it when a protocol has a great test setup that allows me to easily add a new test case without having to write huge amounts of setup code to get the protocol into the desired state first.
Q) Can you describe/breakdown your auditing process?
Besides the technical skills like knowing many types of exploits, knowing the EVM well, or having seen issues of similar protocols, some personality traits that I think are useful: conscientiousness - I feel like some auditors don't even try to find all bugs and just want to be done with their job as quickly as possible. This happens especially if the incentives are not aligned and you get paid a fixed salary as is often the case with traditional auditing firms. So you want to hire people that are conscientious, who take their job seriously and take pride in their work.
You don't need cryptography knowledge to be an auditor but I see more and more math-heavy DeFi protocols, so being good at math is definitely a plus.
My income streams are 1) Code4rena 2) audits of protocols that reach out to me 3) investments 4) onlyfans
My auditing process is pretty straightforward. First I read the documentation. Then I read the code from top to bottom, I order the contracts in a way that makes sense: for example, I read the base class contract first before I read the derived class contract. I don't use any tools, but I heavily take notes and scribble all over the code. 😃 I'm using the "Solidity Visual Developer" extension which comes with the @audit, @audit-info, @audit-ok, @audit-issue markers which I all use to categorize my notes. After I read the entire codebase once I revisit my notes and resolve any loose ends or things I didn't understand earlier. Afterwards, I create my audit report out of these notes.
Q) Any must-reads on auditing you recommend?
A) Subscribe to the "BlockThreat Newsletter" by https://twitter.com/_iphelix. It's a weekly newsletter consisting of all security incidents, post mortems, and other security-related topics that happened in the past week. @Rajeev | Secureum list is great and compact https://secureum.substack.com/p/smart-contract-security-101-secureum
Q) Salary for auditors based on experience/skin level?
A) I'm not an expert on this but I'd say hourly rates for auditors are roughly:
Junior: 100$/h
Experienced: 100$-250$/h
Top Auditors: 250$-1000$/h
If you're asking for what I think the three most common vulnerabilities are, I would say
missing input validation (not checking if a token is legit, etc)
improper access control (function should be private but is public, should be onlyOwner but isn't, etc)
bad math
However, just know that the key to success isn't to memorize a checklist of vulnerabilities and go through them one by one, it's being able to react to what you see and pattern match based on your experiences. A checklist wouldn't have found the Poly Network exploit, but a careful reading and analysis successfully prevented a similar bug in Optimism from going live (ethereum-optimism/contracts#360)
My policy is that everything stays private by default unless I have some reason to make it public. I don't have a good reason to de-anonymize myself right now so I don't.
Q) What activities were the most productive for you to gain a deep understanding of lower-level ethereum concepts?
A) Always be curious and don't be afraid to dig deep into concepts. Have you ever wondered how a mapping is implemented, and why do we assume collisions are impossible? What's the difference between calling a regular function and a view function, and how does the EVM enforce that? There's plenty of questions that are waiting to be asked about Solidity and Ethereum that most people just take for granted, and the answers to those questions will require you to explore the EVM. There's no better way to learn about low-level concepts than this imo
Q) How much code do you write on a weekly basis vs read?
A) I split about 50/50 right now because I'm working on some tooling, but sometimes I might go weeks without writing any code at all
Q) How did you get to where you are now?
A) I started with smart contract security by reading blogs about common vulnerabilities just to get myself up to speed, and then slowly worked my way down the stack as I felt more and more comfortable about what I knew. Jumping right into the EVM seems like a great way to burn yourself out if you're not prepared for it
Q) What is your tooling like?
A) My tooling is very simple.
Browsing source code:
GitHub web, for smaller projects (basically most projects, I did my review of Uniswap v3 on GitHub)
VSCode, for bigger projects (0x v4)
Exploit development:
Remix
A custom remix plugin to let me write quick scripts (impersonate accounts, print debug messages, etc). You can do the same with Hardhat/hevm, I'm just too used to my workflow now
Finding random things on-chain
etherscan.io
ethtx.info
ethervm.io
seth
Q) What qualities make a good auditor/security engineer?
A) My default answer is being able to look at something and question how it really works, but actually now that I think about it, I want to run a few experiments to try and test that hypothesis. Maybe more to come 😄
Q) How would you describe your auditing process?
A) If it's a small enough codebase, I skim and audit at the same time, getting a sense for how the code works while looking for common issues. If the codebase is too big, then I need to really focus on understanding the logic before I can move on to looking for bugs. The auditing process itself is really just approaching the code at a bunch of different angles and trying to tease out what's being assumed and how I might break it. Different angles help me narrow down the scope so I know which functions I care about and which ones I don't
Q) Any recommended must-read resources on security/auditing?
A) Nothing in particular, I wouldn't attribute my knowledge to any one (or n < 3) blog post or the like. My number one recommendation is always to just go out there and do the thing, whether it's on live contracts (Immunefi) or CTFs (there's this really neat one called Paradigm CTF 😉)
Q) Is there a better model than requiring auditors (where the cards are so heavily stacked against them)? rugdoc seems to cover lots of contracts at a fast time
A) imo auditors should be extremely vocal about the fact that audits aren't guarantees of security but simply third-party code reviews, and they should make sure that their clients don't treat it as such. Right now things are so bad that a handful of auditors are singlehandedly killing the reputation of the space as a whole because they keep producing reports which they themselves present as guarantees of security, only for the project to be hacked because they missed a critical bug or two.
Q) In terms of security; What are the 3 first things you look at in Smart Contracts?
A) That's an easy one. You go top to bottom. So first 3 things are pragma, imports and the contract's name 😛 The reality is that my specific steps will change depending on what I'm looking at. So I first need to understand what is it that I'm looking at, at least from a conceptual level. So documentation, tests, whitepapers, etc. are handy. And I usually start there. The goal is to understand the architecture and the main roles. I'm a visual thinker, so at this point I draw a lot. I should also say that documentation can be deceiving, so I try to have a healthy distrust for it.
And then there's a handful of things that tend to be quite sensitive, and usual sources of errors. Which is what I think your question is pointing at. So transfers of funds, access controls, complex inheritance chains, upgradeability, external calls to user-controlled targets, assembly, integrations with tokens, integrations with price oracles, heavy math, interfaces with other systems, complex business logic - these are all usual suspects, and I'd say they all deserve attention.
Be ready to read lots of code. Thousands of lines. If you're coming from black box web app testing, you might not be so used to reading code for hours. And at least when it comes to smart contracts, that's a must.
Q) What tools do you use to audit/review contracts?
A) Nothing fancy. At the beginning, whatever let's me draw diagrams. So Slither, Surya and those kind of tools allow me to see inheritance chains, call graphs, exposed functions, etc. That's helpful to see how complex the beast is, and which are the main entry points. I don't use the Solidity Auditor VS Code extension because I feel it clutters the UI too much, at least for my taste. So usually it's just me reading code, taking notes (either in code with comments or a txt file), drawing stuff, and just thinking. I also think people underestimate the power of pen and paper - I've found bugs just following the logic in pen and paper. And then I'm keen on using the existing test suite of the projects to try out attacks or debug the behavior of complex functions. For note taking, some fellow auditors use smarter note taking apps such as Obsidian, Roam. Also others use Kanban boards to track their thoughts and exploration paths. I've tried those workflows as well, they can be useful.
Q) What advice do you have for someone coming in for an entry-level position, how do they position themselves to get noticed by security companies out there? What do they do to improve their skills.
A) Good one. Ok so there are two separate questions, not necessarily one tied to the other. Long answer coming! To improve your skills, I can at least tell you what I did (and continue doing):
Play every smart contract CTF available out there (in increasing complexity, probably Capture The Ether, Ethernaut, Damn Vulnerable DeFi, Paradigm CTF), and hit your head against the walls until you solve the challenges. At the end it's not about the actual solution what matters, but the learning process you went through to reach that solution.
Read most (if not all) vulnerability incidents, post mortems and similar material that is published, and take your time to understand them. But distrust the publisher, and actually try to reproduce the exact vulnerability and attack yourself. If you can code proof of concept exploits, that's also cool. I've been doing this quite often lately for vulnerabilities that catch my attention, and honestly it's been a great way to continue learning about vulnerabilities, attack vectors, tooling, etc.
Skim through published audit reports, and read issues that are interesting for you, if any. You'll learn what kind of things are being found, how they're reported, and how they're solved. Which is a nice exercise - at least has been useful for me.
Participate in workshops, conferences, etc, either live or watching recordings. Some talks you'll like, some you won't and you'll just close after 5 min. But give yourself the chance to be exposed to something new that's being shared by a security researcher, and probably you'll learn something you did not know, or you'll refresh on previous stuff.
I haven't done this actually, but I guess that trying to participate in bug bounty programs or in contests can be a cool way to start putting your skills to use, even if the things you end up finding are low/informational. You'll get exposure to real Solidity code and its intricacies, which is always nice.
Either as an independent security researcher, or as an employee of a security firm, but force yourself to start making a living out of this. You'll get real exposure to code, projects, vulnerabilities, incidents, and all sort of knowledge that you'll only get if you're every single day working on this. On how to get noticed, I'd say that usually having a personal blog in which you write about the stuff you learn, research you're doing, CTF challenges you solve, writeups of bounties you've earned, and similar, it's a great way to show your work and skills. Also if you're more of a coder, a public GitHub with your own projects or solid contributions to open-source libraries and tools (related to security) can also be useful. Participating as speaker in security-related conferences is another one. Or just sharing in Twitter cool stuff. I don't know, probably there are lots of other ways, and sometimes even depends on what each security firm is looking for.
Look for attack vectors and check if they can be exploited.
Most protocols are forks of primitive protocols (i.e., Synthetix, SushiSwap, Pancakeswap, Compound, Uniswap-v2), so understanding those really well will help with majority of audits.
Check out the original Github repo link for much more info; I glossed over a few areas. I will post another article just on auditing/security next. It was so long it needed it’s own article.
This is the 2nd part of the long article on blockchain that I divided up. This article discusses auditing, how the process works, what auditors look for, and what you should be learning if you want to get into auditing smart contracts.
From @SpearBitDAO - @jakelynch
To start, we set up a private discord channel with you and we'll have someone from our audit team speak with you to figure out the scope and provide a quote. We'll also figure out a time and who will be staffed on the audit. Since we're just starting out, we're fairly selective about what we're auditing atm. Once we engage, we'll take a deposit and at that point you'll give the auditors access to the codebase. Then they'll do their job and notify you of any found vulnerabilities while advising you on how to fix any vulnerabilities.
If a vulnerability requires an extensive fix, we reserve the right to ask for a separate engagement. Our rule of thumb for now is about 80% of the week spent on audit and 20% spent on helping fix vulnerabilities. Once finished, we provide a report.
You should use msg.sender for authorization (if another contract calls your contract msg.sender will be the address of the contract and not the address of the user who called the contract). It's also worth mentioning that by using tx.origin you're limiting interoperability between contracts because the contract that uses tx.origin cannot be used by another contract as a contract can't be the tx.origin. https://ethereum-contract-security-techniques-and-tips.readthedocs.io/en/latest/recommendations/#avoid-using-txorigin
From my experience, it's either:
access control mishaps
order of operations error, leading to flash loan or re-entrancy vulnerabilities
issues where two different parties can be the same, like self-transfers or self-liquidations, sometimes break accounting because they operate on cached values in memory instead of always reading the latest one from storage
batch functions that don't work correctly when passing in a batch array with duplicate values
trusting user input
or a combination of flash loans + oracle manipulation/re-entrancy
Flash loan vulnerabilities can become pretty advanced and require a security mindset which I think most devs don't actually have. From my experience most devs don't create threat models for flash loan/oracle manipulation vectors + thats when they get auditors to look into it.
You can see on https://rekt.news/leaderboard/ most have been unaudited and get attacked by one of the listed points.
Q) What general advice would you give to aspiring blockchain bug bounty hunters?
A) A great way to get started learning about smart contract exploits is to practice by participating in wargames and CTFs. You can find some good ones in https://github.com/crytic/awesome-ethereum-security 's repository.
Also, do not get discouraged if you haven’t found a bug yet, even though you have spent a lot of time searching. Often you will be working with code that has been audited by world-class security experts and put through extensive testing. That does not mean that there are no bugs, though — an audit is no guarantee for security and the list of audited projects which have been exploited is long enough.
In fact, both ArmorFi and Fei Protocol were audited. This is the reason Immunefi exists in the first place. Yet, it is easy to get lost in the fallacy of feeling like you are wasting your time when you spend hours upon hours while apparently not achieving any result. In these moments, it is good to remember yourself that even in periods of low or no findings, the knowledge and insight you are gaining by studying and trying to exploit smart contracts will in the long run equip you with a skillset only very few people have, which in itself will be more than rewarding.
Data from 16 case studies of 2020 DeFi exploits show that 1) 72.3% of hacks come from the Financial Model: e.g. interplay between bonding curve and constant product amm, Balancer hack 29/06/2020, 2) 27.4% from Insecure Implementations: e.g. re-entrancy & 3) 0.3% from Arbitrage.
Why is this so? Insecure implementations have been well-studied by past works, so many existing auditing tools can catch them (i.e. slither). But none of the existing tools can check for financial model unsoundness!
Look for:
Before: Control Flow (e.g., if, return, require)
Before: Memory variables created AND used after
After: Storage rights
Auditing Approach
While auditing, always keep an open communication channel with the developers to make sure you can ask questions if you are not sure about what they intend something is meant to do.
Write test cases/fuzzing for the most critical functions.
If the dev patches/fixes are small and you are able to verify the changes you can continue, otherwise if the changes are large and change a good portion of the codebase, the current audit is essentially void and a new auditor should be used since reviewing a new codebase in such a small period of time wont be effective.
If the math is too complicated, beyond comprehension, tell the devs and recommend they find someone else suitable.
Informational adds: gas optimization, linting, unnecessary code.
Read about the project to get an idea of what the smart contracts are meant to do. Glance over all the resources about the project that were made available to you.
Glance over the smart contracts to get an idea of the smart contracts architecture. Tools like Surya can come in handy.
Create a threat model and make a list of theoretical attack vectors (thinking, "how would an attack try to exploit this contract?") including all common pitfalls (i.e., before Solidity 0.8.0, overflows and underflows weren't reverted and SafeMath.sol was required) and past exploit techniques (re-entrancy, flashloans, pool balance manipulation, etc).
Look at places that can do value exchange, also checking if there are missing role requirements, allowing you to review the most critical functionality within a few hours. Especially functions like transfer, transferFrom, send, call, delegatecall, and selfdestruct. Walk backward from them to ensure they are secured properly (useful when viewing large codebases that have thousands of lines to review). Brain-storm new attacks for each smart contract - usually a day for each contract - creating proof-of-concept attacks (i.e., change a param to a certain something, going a different direction with the function flow, etc)
Do a line-by-line review of the contracts. Looking for logic bugs (where the contract is not behaving in a way it is supposed to, i.e., the whitepaper and smart contract don't align) and making sure the contract is safe from the exploits listed earlier.
After telling the devs about the review, do another round of review, however not as comprehensive.
What are the 3 first things you look at in Smart Contracts?
Architecture
Access control
Fund transfers
Q) Do you have any advice for people who would like to go into this area of using formal verification on blockchain protocols and smart contracts? E.g. what are the pre-requisites? And if one is interested, what are the companies that we can apply for?
A) I think learning how to use the FV tools to their best, and a bit of how they work, will give you both practical experience in the topic + give you pointers to learn more about the theoretical aspects if you're interested. As far as companies go I can't really say specific names, but I guess companies that use FV for their audits.
As a very quick thought, accesses that public/external functions give, unsafe external calls and assembly.
Smart contract audits should not be used as stamps of approvals. Words like "Passed audit" are a big red flag. Audits are basically detailed code reviews. They do not guarantee security. It's very hard to quantify/standardize such a process. This is why there are so many shitty audit firms around. I don't expect reputable firms to take "personal loss" in case of accidents. That's a completely different product offering (Insurance). However, I do expect Insurance providers and Code auditors to work closely in the future.
Q) What CTFs/war games/materials do you recommend doing for someone aspiring to take your #1 spot on Code4rena?
A) I started out doing Damn Vulnerable Defi https://cmichel.io/damn-vulnerable-de-fi-solutions/, Ethernaut https://cmichel.io/ethernaut-solutions/ and Capture The Ether https://cmichel.io/capture-the-ether-solutions/. The links here include my solutions if you get stuck but try to do it on your own first. Another great way that got me in contact with other auditors and all major auditing firms was Paradigm CTF https://cmichel.io/paradigm-ctf-2021-solutions/
Q) What tools do you use when auditing?
A) I don't use any tools that directly perform vulnerability analysis. I only use the great "Solidity Visual Developer" VSCode extension that highlights storage variables and function parameters which makes it easier to have some context when reading a new codebase.
Q) How do you calculate the audit's cost?
A) Scoping audits is always a tough task and imo you only get better at it with experience. But I'll give you a rule of thumb: Let's say you can audit 200 lines of code per hour (which is a standard assumption among some auditors afaik) - adjust that parameter down if the code is complex, math-heavy or the documentation is bad. You then take the lines of code and divide it by 200 LOC/h, then you get the hours required to audit the code for a single person. If you're an independent auditor you should also add 5h-10h for compiling the report and all the biz-dev work, plus answering questions. Then you multiply it by your hourly rate. More on hourly rates below.
How much time do you spend on a contract on average till you "stop" looking at it, i.e. when new C4 projects launch?
How much do you use tools and how much is really looking at the code and manual work?
How do you make sure that your possible exploit really is possible when most C4 code is not deployed yet?
And: Do you have N/A too or do you report only 100% sure vulns?
That's a good question. I could always spend more time on the code and it would increase the likelihood of me finding bugs. But at some point, there's the point of "diminishing returns", where it's not reasonable to spend any more time on the code. Alexander Schlindwein was asked the same question regarding bug bounties but I think it applies to audits as well: The approach which works best for me is to set myself the goal of fully understanding the system to the point where I could reimplement it from scratch without being allowed a look at the original codebase. Not from remembering the code, but from having understood what the application is supposed to do. If you have examined a project that far and have not found a bug, the chances of finding one by continuing is low. https://medium.com/immunefi/interview-with-legendary-bug-bounty-hunter-alexander-schlindwein-cced9c73c02a Realistically, I often stop before reaching that point due to time constraints and opportunity costs when I think my limited time is better spent elsewhere.
100% manual work for me
I definitely have N/As as well as I cannot code up a reproduction in a test framework for every issue I find. It would take too much time and I only do it for complex critical issues. Personally, I also think it's better to err on the side of "this looks like an issue to me, so I'll submit it" than "not sure if it's an issue, I keep it to myself". I've had devs tell me that even though my submission was wrong, they feel more confident in their code now because they reviewed the issue again. I'd like to stress though that I really appreciate it when a protocol has a great test setup that allows me to easily add a new test case without having to write huge amounts of setup code to get the protocol into the desired state first.
Q) Can you describe/breakdown your auditing process?
Besides the technical skills like knowing many types of exploits, knowing the EVM well, or having seen issues of similar protocols, some personality traits that I think are useful: conscientiousness - I feel like some auditors don't even try to find all bugs and just want to be done with their job as quickly as possible. This happens especially if the incentives are not aligned and you get paid a fixed salary as is often the case with traditional auditing firms. So you want to hire people that are conscientious, who take their job seriously and take pride in their work.
You don't need cryptography knowledge to be an auditor but I see more and more math-heavy DeFi protocols, so being good at math is definitely a plus.
My income streams are 1) Code4rena 2) audits of protocols that reach out to me 3) investments 4) onlyfans
My auditing process is pretty straightforward. First I read the documentation. Then I read the code from top to bottom, I order the contracts in a way that makes sense: for example, I read the base class contract first before I read the derived class contract. I don't use any tools, but I heavily take notes and scribble all over the code. 😃 I'm using the "Solidity Visual Developer" extension which comes with the @audit, @audit-info, @audit-ok, @audit-issue markers which I all use to categorize my notes. After I read the entire codebase once I revisit my notes and resolve any loose ends or things I didn't understand earlier. Afterwards, I create my audit report out of these notes.
Q) Any must-reads on auditing you recommend?
A) Subscribe to the "BlockThreat Newsletter" by https://twitter.com/_iphelix. It's a weekly newsletter consisting of all security incidents, post mortems, and other security-related topics that happened in the past week. @Rajeev | Secureum list is great and compact https://secureum.substack.com/p/smart-contract-security-101-secureum
Q) Salary for auditors based on experience/skin level?
A) I'm not an expert on this but I'd say hourly rates for auditors are roughly:
Junior: 100$/h
Experienced: 100$-250$/h
Top Auditors: 250$-1000$/h
If you're asking for what I think the three most common vulnerabilities are, I would say
missing input validation (not checking if a token is legit, etc)
improper access control (function should be private but is public, should be onlyOwner but isn't, etc)
bad math
However, just know that the key to success isn't to memorize a checklist of vulnerabilities and go through them one by one, it's being able to react to what you see and pattern match based on your experiences. A checklist wouldn't have found the Poly Network exploit, but a careful reading and analysis successfully prevented a similar bug in Optimism from going live (ethereum-optimism/contracts#360)
My policy is that everything stays private by default unless I have some reason to make it public. I don't have a good reason to de-anonymize myself right now so I don't.
Q) What activities were the most productive for you to gain a deep understanding of lower-level ethereum concepts?
A) Always be curious and don't be afraid to dig deep into concepts. Have you ever wondered how a mapping is implemented, and why do we assume collisions are impossible? What's the difference between calling a regular function and a view function, and how does the EVM enforce that? There's plenty of questions that are waiting to be asked about Solidity and Ethereum that most people just take for granted, and the answers to those questions will require you to explore the EVM. There's no better way to learn about low-level concepts than this imo
Q) How much code do you write on a weekly basis vs read?
A) I split about 50/50 right now because I'm working on some tooling, but sometimes I might go weeks without writing any code at all
Q) How did you get to where you are now?
A) I started with smart contract security by reading blogs about common vulnerabilities just to get myself up to speed, and then slowly worked my way down the stack as I felt more and more comfortable about what I knew. Jumping right into the EVM seems like a great way to burn yourself out if you're not prepared for it
Q) What is your tooling like?
A) My tooling is very simple.
Browsing source code:
GitHub web, for smaller projects (basically most projects, I did my review of Uniswap v3 on GitHub)
VSCode, for bigger projects (0x v4)
Exploit development:
Remix
A custom remix plugin to let me write quick scripts (impersonate accounts, print debug messages, etc). You can do the same with Hardhat/hevm, I'm just too used to my workflow now
Finding random things on-chain
etherscan.io
ethtx.info
ethervm.io
seth
Q) What qualities make a good auditor/security engineer?
A) My default answer is being able to look at something and question how it really works, but actually now that I think about it, I want to run a few experiments to try and test that hypothesis. Maybe more to come 😄
Q) How would you describe your auditing process?
A) If it's a small enough codebase, I skim and audit at the same time, getting a sense for how the code works while looking for common issues. If the codebase is too big, then I need to really focus on understanding the logic before I can move on to looking for bugs. The auditing process itself is really just approaching the code at a bunch of different angles and trying to tease out what's being assumed and how I might break it. Different angles help me narrow down the scope so I know which functions I care about and which ones I don't
Q) Any recommended must-read resources on security/auditing?
A) Nothing in particular, I wouldn't attribute my knowledge to any one (or n < 3) blog post or the like. My number one recommendation is always to just go out there and do the thing, whether it's on live contracts (Immunefi) or CTFs (there's this really neat one called Paradigm CTF 😉)
Q) Is there a better model than requiring auditors (where the cards are so heavily stacked against them)? rugdoc seems to cover lots of contracts at a fast time
A) imo auditors should be extremely vocal about the fact that audits aren't guarantees of security but simply third-party code reviews, and they should make sure that their clients don't treat it as such. Right now things are so bad that a handful of auditors are singlehandedly killing the reputation of the space as a whole because they keep producing reports which they themselves present as guarantees of security, only for the project to be hacked because they missed a critical bug or two.
Q) In terms of security; What are the 3 first things you look at in Smart Contracts?
A) That's an easy one. You go top to bottom. So first 3 things are pragma, imports and the contract's name 😛 The reality is that my specific steps will change depending on what I'm looking at. So I first need to understand what is it that I'm looking at, at least from a conceptual level. So documentation, tests, whitepapers, etc. are handy. And I usually start there. The goal is to understand the architecture and the main roles. I'm a visual thinker, so at this point I draw a lot. I should also say that documentation can be deceiving, so I try to have a healthy distrust for it.
And then there's a handful of things that tend to be quite sensitive, and usual sources of errors. Which is what I think your question is pointing at. So transfers of funds, access controls, complex inheritance chains, upgradeability, external calls to user-controlled targets, assembly, integrations with tokens, integrations with price oracles, heavy math, interfaces with other systems, complex business logic - these are all usual suspects, and I'd say they all deserve attention.
Be ready to read lots of code. Thousands of lines. If you're coming from black box web app testing, you might not be so used to reading code for hours. And at least when it comes to smart contracts, that's a must.
Q) What tools do you use to audit/review contracts?
A) Nothing fancy. At the beginning, whatever let's me draw diagrams. So Slither, Surya and those kind of tools allow me to see inheritance chains, call graphs, exposed functions, etc. That's helpful to see how complex the beast is, and which are the main entry points. I don't use the Solidity Auditor VS Code extension because I feel it clutters the UI too much, at least for my taste. So usually it's just me reading code, taking notes (either in code with comments or a txt file), drawing stuff, and just thinking. I also think people underestimate the power of pen and paper - I've found bugs just following the logic in pen and paper. And then I'm keen on using the existing test suite of the projects to try out attacks or debug the behavior of complex functions. For note taking, some fellow auditors use smarter note taking apps such as Obsidian, Roam. Also others use Kanban boards to track their thoughts and exploration paths. I've tried those workflows as well, they can be useful.
Q) What advice do you have for someone coming in for an entry-level position, how do they position themselves to get noticed by security companies out there? What do they do to improve their skills.
A) Good one. Ok so there are two separate questions, not necessarily one tied to the other. Long answer coming! To improve your skills, I can at least tell you what I did (and continue doing):
Play every smart contract CTF available out there (in increasing complexity, probably Capture The Ether, Ethernaut, Damn Vulnerable DeFi, Paradigm CTF), and hit your head against the walls until you solve the challenges. At the end it's not about the actual solution what matters, but the learning process you went through to reach that solution.
Read most (if not all) vulnerability incidents, post mortems and similar material that is published, and take your time to understand them. But distrust the publisher, and actually try to reproduce the exact vulnerability and attack yourself. If you can code proof of concept exploits, that's also cool. I've been doing this quite often lately for vulnerabilities that catch my attention, and honestly it's been a great way to continue learning about vulnerabilities, attack vectors, tooling, etc.
Skim through published audit reports, and read issues that are interesting for you, if any. You'll learn what kind of things are being found, how they're reported, and how they're solved. Which is a nice exercise - at least has been useful for me.
Participate in workshops, conferences, etc, either live or watching recordings. Some talks you'll like, some you won't and you'll just close after 5 min. But give yourself the chance to be exposed to something new that's being shared by a security researcher, and probably you'll learn something you did not know, or you'll refresh on previous stuff.
I haven't done this actually, but I guess that trying to participate in bug bounty programs or in contests can be a cool way to start putting your skills to use, even if the things you end up finding are low/informational. You'll get exposure to real Solidity code and its intricacies, which is always nice.
Either as an independent security researcher, or as an employee of a security firm, but force yourself to start making a living out of this. You'll get real exposure to code, projects, vulnerabilities, incidents, and all sort of knowledge that you'll only get if you're every single day working on this. On how to get noticed, I'd say that usually having a personal blog in which you write about the stuff you learn, research you're doing, CTF challenges you solve, writeups of bounties you've earned, and similar, it's a great way to show your work and skills. Also if you're more of a coder, a public GitHub with your own projects or solid contributions to open-source libraries and tools (related to security) can also be useful. Participating as speaker in security-related conferences is another one. Or just sharing in Twitter cool stuff. I don't know, probably there are lots of other ways, and sometimes even depends on what each security firm is looking for.
Look for attack vectors and check if they can be exploited.
Most protocols are forks of primitive protocols (i.e., Synthetix, SushiSwap, Pancakeswap, Compound, Uniswap-v2), so understanding those really well will help with majority of audits.
Check out the original Github repo link for much more info; I glossed over a few areas. I will post another article just on auditing/security next. It was so long it needed it’s own article.
Do another review from the perspective of every actor in your threat model (i.e., liquidity provider, user swapping), calling the functions in various ways (the most common use cases and only after with uncommon use cases/combinations).
Run tools like Slither and review their output.
Glance over the test cases and code coverage.
Do another review from the perspective of every actor in your threat model (i.e., liquidity provider, user swapping), calling the functions in various ways (the most common use cases and only after with uncommon use cases/combinations).
Run tools like Slither and review their output.
Glance over the test cases and code coverage.
No activity yet