# Understanding Ethereum Network Upgrades: Lecture 5

> Testing and Implementation - Mastering Ethereum Network Upgrades

**Published by:** [Wiki of Web3](https://paragraph.com/@education-dao/)
**Published on:** 2024-05-28
**Categories:** ethereum, lectures/seminars, courses
**URL:** https://paragraph.com/@education-dao/dencun-5

## Content

This lecture covers the processes of testing the various components of a network upgrade, and how these components are implemented once successfully tested.Index Lecture Office Hours Guest SpeakerSupplemental Resources Lecture Lecture 5 Slideshow▼Lecture Transcript The following transcript was auto-generated, and may content syntactical errors. 00:00:09.000 --> 00:00:17.000Today is January seventeenth, Wednesday, January seventeenth and, 2024 and we're in the week of testing and implementation.00:00:17.000 --> 00:00:28.000So we're going to be talking about both generically, what does testing and implementation look like for, an upgrade and then we will get into some.00:00:28.000 --> 00:00:42.000Specifics of what testing and implementation. Looks like for the Denkun upgrade. So. Take ticket off today, you know.00:00:42.000 --> 00:00:46.000How do we test what is changing? And I think the short answer. Is we do it, we start incrementally in piecewise.00:00:46.000 --> 00:01:01.000So there's always some testing going on. That's incremental and piecewise. It's testing out EIPs that's testing out.00:01:01.000 --> 00:01:11.000Potential attack vectors on the network. These are kind of always happening. And they're happening.00:01:11.000 --> 00:01:20.000During network upgrade like in the run up to network upgrades, but also in the planning phase it's constantly kind of happening.00:01:20.000 --> 00:01:31.000But then as we get closer to actually making a network upgrade. Those. We test everything altogether at the same time.00:01:31.000 --> 00:01:40.000So, we start out with this incremental piecewise process and then we move to like doing everything and seeing how it works together.00:01:40.000 --> 00:01:48.000So We're gonna touch on. Really. These 4.00:01:48.000 --> 00:01:55.000Elements of what gets tested and then we're going to talk about like testing before the upgrade and then testing after an upgrade occurs.00:01:55.000 --> 00:02:03.000And there actually was. A test network that happened today. So we're also going to talk about what some high level results of that.00:02:03.000 --> 00:02:17.000So just to sort of cover the like our. We're going to talk about the implementation of EIPs by client teams or within in an Ethereum client.00:02:17.000 --> 00:02:21.000EIPs by client teams or within an Ethereum client, whether it's execution layer client or the CL client.00:02:21.000 --> 00:02:37.000Test cases, who writes them, where do they exist? What are the testing tools that exist we're gonna talk about shadow forks and dev nets which are some of the coolest like Names of stuff that we gave in Ethereum.00:02:37.000 --> 00:02:44.000We've done we've done ourselves sometimes a disservice with the naming but not with shadow forks shadow for the naming but not with shadow forks.00:02:44.000 --> 00:02:55.000Shadow Forks is that's a sweet name. And then we're gonna talk about test net forks in general and what those do and the overall like upgrade itself, which is the implementation.00:02:55.000 --> 00:03:11.000So we'll jump right into it. So the first thing we need to do if we look back at our week that we talked about the like writing EIPs, when we write an EIP, we have to actually implement it in an Ethereum client.00:03:11.000 --> 00:03:21.000So whether that's the EIP for an execution layer client or a consensus layer client. We actually have to implement that EIP in order to test it, right?00:03:21.000 --> 00:03:40.000You have to have some sort of implementation. So client teams do this. If you join all core Devs calls, There was always a part of the call where we go, they go around to the client teams, particularly on the execution layer and say, have you implemented these changes?00:03:40.000 --> 00:03:49.000There's often times folks from the EF who are asking questions around, hey, when is this team going to finish their implementation?00:03:49.000 --> 00:03:57.000And there's a lot of back and forth between the various teams on getting that implementation done so that tests can be written.00:03:57.000 --> 00:04:11.000But. Because all of these clients are in open source software repositories, I think actually all of them are pretty much hosted on GitHub, maybe if one or 2 on Gitlab.00:04:11.000 --> 00:04:15.000You could just fork that repo if you are an author of EIP and if you have the knowledge of that code base, you could implement it yourself.00:04:15.000 --> 00:04:30.000So this is a reason why we talked about reference implementations from EIP authors being so helpful. They actually make it faster for client teams for to be able to figure out how they're going to implement that, that EIP in the client.00:04:30.000 --> 00:04:47.000So, like I said, anyone can do this. You can just create a fork of a software repository and add those add those changes into it and oftentimes that's what you see is done.00:04:47.000 --> 00:04:48.000It's a combination of the client teams, but if someone is properly motivated, they can go ahead and do it.00:04:48.000 --> 00:05:04.000With their client of choice. Once that EIP is implemented, it's actually then possible for any user to download that and actually just run.00:05:04.000 --> 00:05:08.000From either a Genesis block or from a point, a certain point in the state, a local, a local version of the chain.00:05:08.000 --> 00:05:23.000And that could be a fork, but it could also just be there running the chain and seeing how it how it works to see the impact that the changes have, right?00:05:23.000 --> 00:05:33.000And when I say see the impact, I'm talking both about like literally like you know, potentially doing some actions on that chain.00:05:33.000 --> 00:05:36.000That's sort of just generally probing to see what might be different or it could be actually running like a full test suite.00:05:36.000 --> 00:05:53.000That is, is specifically written. So, not making a distinction there. The reason I bring that up is because Once you have something that you're able to run locally, a version of the chain.00:05:53.000 --> 00:06:05.000That can be useful for iterating on writing test cases because test cases are written typically towards what are the expected behaviors and what are the what are the behaviors that are out of bounds.00:06:05.000 --> 00:06:06.000So having something that you can refer to and actually go ahead and start writing those tests is critically important.00:06:06.000 --> 00:06:17.000Anything you want to add here, Matt.00:06:17.000 --> 00:06:30.000One thing that is kind of noted is that the client implementations often yield spec changes because we go to build something in we're like the spec actually doesn't work in practice or we realize that the spec is.00:06:30.000 --> 00:06:41.000Maybe over overly permissive or overly restrictive in certain cases, it may trigger portions of a client code that is not aware of.00:06:41.000 --> 00:06:54.000So for example, if I have a any IP that impacts the way that maybe the state tree works that could have cascading impacts up and down the clients that we are kind of unaware of that trigger new considered security considerations.00:06:54.000 --> 00:07:07.000So in reality, the first set of implementations are always fuzzy. Because once we The client implementations exist in isolation until we link them to other clients in these Dev Nets and test nets.00:07:07.000 --> 00:07:16.000So we yield a lot of results because We have a spec about a chunk of functionality and then we have all the other company pieces of the client.00:07:16.000 --> 00:07:30.000And then we have how your client peers with other clients. So We, you know, we do a lot of iteration based on this because these, initial implementations may pass all of the reference tests and all of these other tests.00:07:30.000 --> 00:07:44.000Because the specification is sound but it has upstream impacts in the client code that we don't we're not aware of so these first rounds of iterations are very much so in flux and the spec often changes beyond what was in scope for the EIP.00:07:44.000 --> 00:07:53.000But we need the kind of interoperability testing of clients to be able to understand the test cases better and to understand the client code.00:07:53.000 --> 00:08:03.000In terms of other clients, the Ethereum protocol itself and things we might not be aware of. So there's a lot of back and forth at this stage, I would say.00:08:03.000 --> 00:08:09.000And so while we said that, you know, client implementation, we these slides are sequential as Matt is referencing.00:08:09.000 --> 00:08:11.000This is more of a back and forth process. You can almost think of this happening as like concurrent to each other, right?00:08:11.000 --> 00:08:19.000So, what's great is that the EF and all core devs do keep a software repository.00:08:19.000 --> 00:08:33.000HOST on GitHub, under the Ethereum repo of Ethereum test cases. So I'll just click into that if you want to actually see what that looks like.00:08:33.000 --> 00:08:41.000And here we go. It's their name tests. You can see that there is a lot of tests in here, some are not relevant anymore, proof of work tests, right?00:08:41.000 --> 00:08:49.000And there is documentation to go along with that. The slides all link that I'll put them in, and afterwards, but we'll have all these these readings you can click out onto.00:08:49.000 --> 00:09:00.000There's actually, so like I said, there's general tests and then there's those that are specific to upgrade, right?00:09:00.000 --> 00:09:13.000So if you think about it, there's a set of tests that. The, that should be passable by any client pair or just or an execution layer client.00:09:13.000 --> 00:09:19.000Or a consensus like your client because that's like just you know, adhering to the overall spec, right?00:09:19.000 --> 00:09:26.000And then there are specific tests that, might be things that need to be tested in the upgrade.00:09:26.000 --> 00:09:33.000There is a full tutorial. So if you're interested in participating in this, there is the Ethereum test.00:09:33.000 --> 00:09:51.000Doc repo that's linked in this software repository. It's like a sub repo and it actually walks you through Testing against your client, it uses GET as the sort of reference implementation client, but you can.00:09:51.000 --> 00:10:00.000Try this out with anything else and I think this is a really great place to start if you're interested in being part of testing.00:10:00.000 --> 00:10:15.000So These tests. Assume in this case that you're using a tool called retest. And it's a tool that allows for the running of the test and the generation of the stats on tests.00:10:15.000 --> 00:10:29.000Now I think Matt. And I in our day jobs. You know, other test suites, right?00:10:29.000 --> 00:10:43.000Because we, work. So, Matt is definitely having to work with this test suite, but then there's like other test suites that the base you team and the take-out team take advantage of.00:10:43.000 --> 00:10:56.000Me being from like the inferior side and working on decentralized infrastructure network, we also use different test suites in order to test performance of certain, let's, if we just leave it to Ethereum, there's different test suites that we have.00:10:56.000 --> 00:11:07.000So what I pointed out because Just because we're only highlighting retest, as sort of this.00:11:07.000 --> 00:11:16.000Anchoring tool and it very much is an anchoring tool. It doesn't mean that different, stakeholders in the space.00:11:16.000 --> 00:11:29.000Are just limited to using this tool. Oftentimes, there are other testing tools used. The nice thing about retest, is that there's new releases that are pushed out for each fork.00:11:29.000 --> 00:11:46.000So We have a release here that allows us to be able to run a suite of tests and it gives us the stats that we need and brings in sort of all the tests that we need for this.00:11:46.000 --> 00:12:01.000What you need to know is that there are teams of folks who are like writing these tests, thinking about them, and they are working if you go into the ER and Discord, there's a channel.00:12:01.000 --> 00:12:09.000We highlight it later in this deck. It's called hashtag testing. If I Remember correctly, while it's currently, tagged.00:12:09.000 --> 00:12:16.000Underneath it's underneath like the heading Den Coon. It's a channel that moves around with different upgrades.00:12:16.000 --> 00:12:18.000Like if you go back in the history, it's actually the whole sense that ER. Discord started of testing.00:12:18.000 --> 00:12:26.000So it has that the testing history in in conversation that happened for Shanghai and Capella and going back so on and so forth.00:12:26.000 --> 00:12:41.000That channel just like gets moved with each upgrade. So that is kind of where things happen. Retest E, there is a web interface.00:12:41.000 --> 00:12:51.000It is a, a stark web interface. Let's put it that way. But sometimes, you know, sometimes you like brutalist architecture.00:12:51.000 --> 00:13:01.000Sometimes you really like brutalist web design. And so there is a web interface that you can see that can help run the test.00:13:01.000 --> 00:13:07.000So very useful that we have this. When you think about it, there's a lot of work that goes into building this shared infrastructure.00:13:07.000 --> 00:13:20.000And now I think that helps contextualize the importance of our C, the Ethereum Foundation. If we think back on that graph of overlapping, it's not just the ideation and creation and vetting of ideas.00:13:20.000 --> 00:13:28.000It's also building the DevOps infrastructure. So there is an Ethereum Foundation DevOps team that really takes ownership of a lot of this work.00:13:28.000 --> 00:13:33.000Anything else you want to add here, Matt?00:13:33.000 --> 00:13:34.000Yeah, where we just recently started to migrate a lot of these tests to something called hive testing suite.00:13:34.000 --> 00:13:44.000The reason being is that Hive takes a new account, multi-client use cases for testing and it also takes no account the CL to EL separation.00:13:44.000 --> 00:13:55.000So I, when I will go back and add hive to this. Slide when we go back to share the materials.00:13:55.000 --> 00:14:16.000But hive is a super important one for us to test. A lot of, a lot of the reason being is that it exercises the EVM and state tests but also exercises like the reality of proof of stakes so it it exercises the fork choice rule it exercises the engine API at exercises sync testing.00:14:16.000 --> 00:14:24.000It's really a moralistic picture that is slowly becoming this de facto standard. We're also going to, I believe this might be.00:14:24.000 --> 00:14:42.000Big news, but I believe we're incorporating into that testing harness. So there'll be kind of a one complete suite going forward because now that we have the merge and proof of stake retest deep is really a way to test execution layer implementations of the EVM and some of the state stuff.00:14:42.000 --> 00:14:54.000So we're kind of creating a new picture, but we're rolling all these things together. In the Ethereum Foundation leads these efforts continuously, but Client teams create new test cases all the time.00:14:54.000 --> 00:15:07.000So like, you know, For example, the bug that impacted based on main net on January sixth, we're potentially, we're adding back that test, that corner case of what happened in the state dB implementation to the high testing suite so that we can.00:15:07.000 --> 00:15:17.000Not have that happen to again or anyone else. So it's kind of like Tom said a collaborative process.00:15:17.000 --> 00:15:24.000Alright, so now let's talk Shadow Forks in Dev Nets. So first. These are my definitions of what a shadow fork in Devnet is.00:15:24.000 --> 00:15:29.000I'm sure if you Google this, there could be different ones, but I've just from seeing them.00:15:29.000 --> 00:15:49.000A shadow fork is basically when you take an existing network that could be the theory of main net that could be a test net like girly, it could be, Sapolia, it could be, and you are literally taking from a point of a set.00:15:49.000 --> 00:15:57.000State, you know, so think about it like a certain block. From that point, you're forking the test and applying some changes.00:15:57.000 --> 00:16:10.000So we use Shadow Forks because it allows you to implement the upgrade or part of the upgrade and observe the conditions starting from a known specific existing state.00:16:10.000 --> 00:16:29.000So there is a, if, you check out in, there's the Ethereum Hack MD, which is essentially a, I would call it a scientific document writing, platform.00:16:29.000 --> 00:16:35.000We have these, you could see here's girly, shadow fork 0 analysis.00:16:35.000 --> 00:16:43.000And it talks about the test setup, provides some information about. Health of the network. These look like screenshots of.00:16:43.000 --> 00:16:54.000Almost certainly Graafanna. Which is an observability tool and it is, you know, sort of running through what happened there.00:16:54.000 --> 00:17:02.000So we have a spec sheet for every shadow fork and it lets basically anyone who's participating know and define the parameters.00:17:02.000 --> 00:17:16.000It's important for setup. It's also important for analysis, right? So here we have the spec sheet, talks about everything that will will happen, talks about when it will start up, when it will be shut down.00:17:16.000 --> 00:17:25.000What is included in it when has that been merged, etc. So that is really important.00:17:25.000 --> 00:17:39.000So shadow for think about it. We are taking a Set. A set state and from that state we're applying some change so we're literally forking an existing chain.00:17:39.000 --> 00:17:51.000Now, Dev Nets are specific networks that are spun up. To test the implementation of changes. And They are not made from an existing network state.00:17:51.000 --> 00:18:00.000That is kind of how I define it. You might hear slightly different definitions, but the idea is that, hey, a Devnet is literally a developer network, right?00:18:00.000 --> 00:18:20.000Like Devnet is typically used with other sort of if you're writing solidity, your DAP developer, you might first deploy your, your, depth to a, Devnet that you create from like, a.00:18:20.000 --> 00:18:24.000You know, if you're using any of the sort of development frameworks, most of them have something that you can spin up and run a Devnet.00:18:24.000 --> 00:18:39.000So you think about this is isn't sort of necessarily taking the set network state. But I think the key and and Rob like you know, showed up on camera.00:18:39.000 --> 00:18:56.000So this is a great time to say like a remix being one of those frameworks. Has options to essentially like if you think about it from that depth developer perspective, you can either like start clean like a true Devnet or you could say, hey, I want to start from a certain fork.00:18:56.000 --> 00:19:07.000Right? So you could deploy and saying like, Hey, I want to deploy under what I would call the, I think, you know, if you, if you use remix, I was using it the other day, it's gonna be, Shanghai is like your default when, when you load it up in your web browser.00:19:07.000 --> 00:19:20.000So, Dev Nets. Once again, same thing, spat like. We're gonna have our.00:19:20.000 --> 00:19:32.000Here's the test. Here's the information. Here's you know, what our analysis of it and then we're gonna have a spec sheet, right? Same sort of deal.00:19:32.000 --> 00:19:46.000We're gonna have a spec sheet that's gonna help guide us. And in this case, you know, these are, these are really important for actually like running, running these, these tests.00:19:46.000 --> 00:19:49.000The results are. Sort of I would say formally reported out from both the shadow forks and the Dev Nets.00:19:49.000 --> 00:20:11.000In the, E. Developer operations. Hack MD. So here you could see a bunch of like different published notes and and information about them so you could like go in and just click on it.00:20:11.000 --> 00:20:18.000We have the links here gives you an overview of. Of all of these different specs and what happened.00:20:18.000 --> 00:20:26.000But what you'll also see is that a lot of times Client teams, other stakeholders.00:20:26.000 --> 00:20:38.000Will report out results. I think in the merge, we saw probably the most reporting out of shadow fork results literally by media outlets and other organizations.00:20:38.000 --> 00:21:00.000You'll see reporting out via Twitter or X or you know, whatever social media. The key though is that you have when you do Shadow Forks and Dev Nets that it doesn't mean that we're necessarily testing all the changes and it might not be inclusive of all the changes that go into the eventual upgrade.00:21:00.000 --> 00:21:07.000It may include changes that are going to go into a next upgrade. It just depends on how that specified, right?00:21:07.000 --> 00:21:13.000So. One big thing to take away from Shadow Forks and Dev Nets, they're useful because it's getting closer and closer to the actual conditions that we'd expect.00:21:13.000 --> 00:21:35.000But we're able to more piecemeal. Test changes under conditions that are looking more and more like what the actual upgrade will look like, but we're not all the way there.00:21:35.000 --> 00:21:42.000So we could be selective. You could, you saw in one of these, there was a Devnet that was really focused on like.00:21:42.000 --> 00:21:57.000Blobs right so you might be testing very specific functionality because you need to do that Like I said, this came after talking about the client implementation and talking about writing the test cases, but Shadow Forks and Dev Nets.00:21:57.000 --> 00:22:06.000Think about this like a circle where these are kind of always happening and we're working to then. Figure out what's going to be in there.00:22:06.000 --> 00:22:15.000So Last week, Matt talked about how ELF was not included. In this upgrade. But EOF was tested in some of the Shadow Forks and Def Nets to be perfectly clear.00:22:15.000 --> 00:22:29.000Like, that did happen during this, during the run up to this upgrade. So understanding what goes in and out is really important.00:22:29.000 --> 00:22:39.000That's where the Shadow Forks and Dev Nets really help us determine whether the the scope of the upgrade is such that it can include certain changes or not.00:22:39.000 --> 00:22:48.000Right? So do not be surprised if you go back to the history and you start to see it's like, Hey, there were implementations even.00:22:48.000 --> 00:23:06.000Correct me if I'm wrong, Matt, I think 4 8 4 4 was being tested in elements of 4 8 4 4.00:23:06.000 --> 00:23:10.000Yeah, the.00:23:10.000 --> 00:23:13.000You'll have to repeat the last part of your question again. Excuse me.00:23:13.000 --> 00:23:24.000Oh, basically we even though that 4 8 EIP 4 8 4 4. You know, proto-dank sharding was gonna, is going into this upgrade.00:23:24.000 --> 00:23:32.000Testing for that was actually occurring. Back before the Shanghai and Capella upgrade, the Shapella upgrade.00:23:32.000 --> 00:23:40.000Yeah, absolutely. So we kind of started developing that at the same time, honestly. So it was.00:23:40.000 --> 00:23:57.000Little bit separated logically because they we had to get withdrawals. Ships first. But yeah, we were, we've We kind of are iteratively testing and developing all of this stuff in tandem and they kind of loop over each other and things like that.00:23:57.000 --> 00:24:02.000So yeah, I think that's the answer to the question is yes.00:24:02.000 --> 00:24:07.000Cool. And here's a great question. Why do we need both Shadow Forks and Dev Nets?00:24:07.000 --> 00:24:16.000The simplest explanation is that it allows you to change the scope of your tests.00:24:16.000 --> 00:24:34.000And what I mean by that is a shadow fork says, let's take existing. So Devnet is probably less close to What, if you think about like main upgrading main net and upgrading main net as the goal.00:24:34.000 --> 00:24:46.000Certain points in time we need to simulate. What a main net upgrade will look like. A shadow for gives us the ability to do that.00:24:46.000 --> 00:24:59.000To give a better approximation of what upgrading main net or test net will look like. Whereas a Devnet allows us to start with maybe more idealized conditions, right?00:24:59.000 --> 00:25:06.000We can sort of change the parameters of where we're starting. Whereas with a shadow for We are saying we need to take in it.00:25:06.000 --> 00:25:11.000We're gonna take an existing state and see how that applies. So basically gives us optionality of what we're testing and how we're testing it, right?00:25:11.000 --> 00:25:31.000Think about the difference between the Shadow Fork and the Devnet. State is the variable and with the Devnet, you're saying that we are going to We were able to manipulate state much more easily because we can sort of set state.00:25:31.000 --> 00:25:46.000Wherever it should be and by state I literally mean the state try or the state tree right whereas with a shadow fork we're saying Let's hit us like a state so we can actually look at whether it's main net or that test net.00:25:46.000 --> 00:25:51.000And then let's manipulate it. And now we have a comparison point of what happens. After that upgrade is applied.00:25:51.000 --> 00:26:03.000So it's that optionality. That's why we need to do both, right? It gives us the ability to test different things and in different ways.00:26:03.000 --> 00:26:07.000And do different comparisons. Great question.00:26:07.000 --> 00:26:13.000Yeah, it's also an approximation of real state. We generate a lot of interestingly fake state.00:26:13.000 --> 00:26:20.000So Shadow Forks basically say, oh, this is what would happen if nodes forked right immediately down.00:26:20.000 --> 00:26:32.000And are the notes prepared because Not only do we shadow for test random stuff, but those nodes kind of they operate on the same memo sometimes in terms of like if you're shadow working girl you'll still pick up transactions.00:26:32.000 --> 00:26:39.000So it's a real approximation of like What would happen if we literally forked a subset of these nodes right now?00:26:39.000 --> 00:26:45.000How would they interact with an existing network? Would they talk to each other? Would they live? Would they agree upon the right state?00:26:45.000 --> 00:26:59.000So there's 2 approaches. Devonets are more build fast break fast kind of thing. And Shadow Forks are expensive because we have to spend real hardware things and on main net shadow forks we have to spend real E.00:26:59.000 --> 00:27:13.000So it's very like those are more legitimate exercises where we feel we're closer to the end goal where we want as Tom said we're testing on a different set of items but we're also testing on a real picture of state and real transactions.00:27:13.000 --> 00:27:18.000So that's definitely valuable.00:27:18.000 --> 00:27:30.000So. Just hammering 2 points that. We set up, said already what didn't make an an upgrade is often tested in the run up to the upgrade, right?00:27:30.000 --> 00:27:39.000So, That is, I think, sometimes a misunderstanding that it's like, hey, the like.00:27:39.000 --> 00:27:51.000The upgrade is scoped so well that we only test what goes into it. The reality is that we're constantly scoping the upgrade and Well, how we're scoping the upgrade.00:27:51.000 --> 00:28:06.000Allows us, you know, how we're doing doing this testing, we are literally scoping the upgrade through an iterative process and That means that you'll often see certain EIPs.00:28:06.000 --> 00:28:10.000Be, tested and maybe not included in upgrade and that might continue on, potentially through multiple upgrades.00:28:10.000 --> 00:28:21.000Also testing is A process. We are in the tens of Dev Nets in Shadow Forks.00:28:21.000 --> 00:28:31.000Respectively for this current. This current upgrade, I think Devnet 12. Was the last one?00:28:31.000 --> 00:28:37.000So, I mean, this gives you a sense of like. This is happening. Iteratively.00:28:37.000 --> 00:28:47.000And so we are, really going. Through this process This is a great question. Matt, maybe you know the answers.00:28:47.000 --> 00:28:54.000What is the budget of real ether to be spent on an upgrade for testing transactions. Maybe you could just give like a rough estimate.00:28:54.000 --> 00:28:59.000I'm sure it's different. Per upgrade, but just trying to get a sense of.00:28:59.000 --> 00:29:03.000Of you know the real the real stakes that are involved.00:29:03.000 --> 00:29:09.000Yeah, so the EF does not release those numbers because they use real validating nodes to test this stuff.00:29:09.000 --> 00:29:14.000And I also don't blame them because it's probably in the tens of thousands of dollars like sustained amount of money.00:29:14.000 --> 00:29:25.000If you're thinking about the amount of EF and the validating stakes that they use, It's because again, we have to pay gas.00:29:25.000 --> 00:29:39.000Or to keep those nodes. Kind of forks like in their own little worlds. Otherwise, they'll because we'll run out of transactions to sustain that fork and they'll say, oh, there's nothing here, something weird is going on and they'll try to connect to new peers.00:29:39.000 --> 00:29:52.000So we basically have to sustain normal main net gas prices from block to block. Let's see what the gas price on Ethereum main net is right now.00:29:52.000 --> 00:30:05.000Yeah, it's like 42. It's like $2 per transaction and blocks can fit tons of transactions and we need to sustain the shadow forks for.00:30:05.000 --> 00:30:12.000Hundreds and hundreds of blocks. So you can see that it will quickly add up in terms of straight up just gas.00:30:12.000 --> 00:30:24.000And since the state that they're using is real and the fork is fake. What we do on the shadow for doesn't impact Ethereum main net state because we are on a fork of the main chain.00:30:24.000 --> 00:30:35.000But it does cost like you have to cost money It's a weird thing to say, right, that we're not actually impacting Ethereum, state, but we need to spend money on gas.00:30:35.000 --> 00:30:40.000So in theory, you think that that would make any sense, right? Because if I just go off on a fork and spend all the gas in the world.00:30:40.000 --> 00:30:48.000It never really makes it back on the main chain. But we have to pay for those. We have to pay to like fork those machines basically and then sustain them with.00:30:48.000 --> 00:31:01.000Stuff and they also run hundreds of nodes like client pairs for the shadow forks which cost money in just like cloud infrastructure costs.00:31:01.000 --> 00:31:02.000Yeah.00:31:02.000 --> 00:31:06.000At the end of the day, it's electricity all the way down and it's and you know, you gotta pay for the electrons one way or the other.00:31:06.000 --> 00:31:15.000And that's a good thing. So let me just like to use an analogy, right? It is it's worth spending.00:31:15.000 --> 00:31:22.000Let's just think about like. The total value locked on Ethereum, not even considerate.00:31:22.000 --> 00:31:28.000Well, and if we consider all the L two's and app chains like L three's.00:31:28.000 --> 00:31:41.000It's worth, you know, spending. Tens to hundreds of thousands of dollars to test this because the value that's at stake literally at stake.00:31:41.000 --> 00:31:57.000If this is wrong would be, you know, it, You're talking about like 0 point 0 0 0 0 0 like 1% of the value spent testing in order to continue to secure an incredible amount of value.00:31:57.000 --> 00:32:08.000So I think that is really key, but what I'll also mention is that like It's not, you know, to test well and to do this and just to do these setups and.00:32:08.000 --> 00:32:18.000Just to go back to, Let's look at the specs. And the like a shadow for like, you know.00:32:18.000 --> 00:32:23.000Running. This is this is00:32:23.000 --> 00:32:29.000A pretty sweet setup, right? Like if we're if we're running. You know, this kind of, like hardware software combination, it has real cost.00:32:29.000 --> 00:32:56.000So, When I mentioned this because I would say have it's always good for us to have some grace and perspective when it comes to, you know, I think I think it's after the merge it's maybe become a little bit more like, oh well, you know, the the client teams and different teams doing this.00:32:56.000 --> 00:32:59.000They're just, oh, they're so good at this. They just do it, but like.00:32:59.000 --> 00:33:07.000There are real costs and If you're running all this infrastructure. There's all the challenges that go along with running that infrastructure just in the day to day as well, right?00:33:07.000 --> 00:33:20.000Keeping machines online. Dealing with outages, dealing with, that the cost of that.00:33:20.000 --> 00:33:21.000So I point that out because it ends up being, it's not a trivial process, but it's worth it.00:33:21.000 --> 00:33:34.000Right? It's very much worth it. So, let's now talk about.00:33:34.000 --> 00:33:48.000I put this as, you know, dress rehearsals and the main event. So we've now crossed over and we want to start testing what I would say would be let's test everything at once, right?00:33:48.000 --> 00:33:56.000Let's test all the dependencies. All the interdependencies at the same time. And that's test net upgrades, right?00:33:56.000 --> 00:34:05.000These are the closest thing to to an upgrade that like can occur because test nets often persist, right?00:34:05.000 --> 00:34:08.000If you run at test and upgrade, the test net will continue to exist after the upgrade and be used.00:34:08.000 --> 00:34:22.000It has real users. So there are real stakes to this. Like there are real users who are, you know, have a deployment of a DAP.00:34:22.000 --> 00:34:39.000Or of a staking setup on a test net that gives them a signal about how their main net setup or depth is gonna operate it is possible to revert these like this isn't as a dire if something goes wrong.00:34:39.000 --> 00:34:46.000And The safest way to do this is start with a test that's going to be deprecated.00:34:46.000 --> 00:34:54.000There isn't always a test that's going to be deprecated. For the merge it was.00:34:54.000 --> 00:35:06.000Rinkby. Thank you for your service, RinkB. There were some other chess sets that were in deprecation process already, Covon.00:35:06.000 --> 00:35:25.000In the case of, it's going to be girly, girly hat is technically, I believe, already officially deprecated or at least it's beyond Q 3 of 2023 which was like the planned deprecation or maybe I'm wrong about that.00:35:25.000 --> 00:35:26.000Already there. It's done.00:35:26.000 --> 00:35:32.000Girlie is going to be deprecated, is already deprecated. So with Den Coon.00:35:32.000 --> 00:35:35.000You're gonna start there because It's like the least is at state with that, but there is still something at stake.00:35:35.000 --> 00:35:44.000And so similar reporting analysis is going to occur as with the De Nets and Shadow Forks and there's often coordination calls and watch parties as that happens to talk through that happening.00:35:44.000 --> 00:35:55.000I'm not saying that there is a coordination of calls with Dev Nets or Shadow Forks, but like.00:35:55.000 --> 00:36:00.000You're sort of seeing like these are the dress rehearsals, like people start showing up for the merge like.00:36:00.000 --> 00:36:13.000I was blown away by how many people were showing up for the calls that were scheduled for not the merge itself but like for the forks like the Test Net Forks and seeing that play out.00:36:13.000 --> 00:36:21.000So Earlier today, we actually had the girly fork. On the thirtieth, we'll have this Apollo one and on the seventh, we'll have the whole ski one.00:36:21.000 --> 00:36:28.000So. Map, what happened? Let's talk about the girly fork that happened earlier, today.00:36:28.000 --> 00:36:38.000Yeah, so this is a non exhaustive. List of post mortem stuff. We uncovered 3 bugs.00:36:38.000 --> 00:36:47.000The biggest issue is that Prism is basically just kind of broken on girly right now. That means that and needs a patch.00:36:47.000 --> 00:36:53.000Some of the gossip blobs that they are receiving from their peers have the wrong index.00:36:53.000 --> 00:37:02.000And that makes the prism note invalidate them improperly, whereas sometimes the index is not needed because of late.00:37:02.000 --> 00:37:14.000Slot late blob stuff. So you don't necessarily know which index of the 0 to 6, well, you know, 0 to 5 because we have 6 max blobs and Things are 0 indexed in computers.00:37:14.000 --> 00:37:20.000So we have those 5 in the season. If you go outside of that, normally you're supposed to be able to handle a non sequential thing.00:37:20.000 --> 00:37:27.000They believe that the bug is that Prison is just handling this case and properly but it's kind of breaking prison on Gurley.00:37:27.000 --> 00:37:38.000Nether mind is not also keeping the max limited blob transactions. In its00:37:38.000 --> 00:37:56.000I believe parsing of these transactions in the pool for per block. So for the slot when prison or another mind is examining the number of maximum blobs they don't keep the correct number between transactions transaction, which could lead to an invalidation of the of the block from Netherlands perspective.00:37:56.000 --> 00:38:10.000I have a feeling this is not an exhaustive list. The fork happened last night we're still investigating, but it's pretty interesting to see that we have, you know, a few bugs, a client called Grand on that's a consensus layer client is proposing in valid blocks as well.00:38:10.000 --> 00:38:18.000This is the first transition that Grand D has been through, so that's probably to be expected. But yeah, we're ongoing investigation.00:38:18.000 --> 00:38:27.000The net is we'll expect kind of a full post Mormon client releases tomorrow on the core Devs execution call.00:38:27.000 --> 00:38:35.000And we will Hopefully have more information after that.00:38:35.000 --> 00:38:38.000Awesome. So follow along. Join the all core devs call if you want to hear more because that's going to be pretty interesting.00:38:38.000 --> 00:38:51.000This is so, what's gonna happen after this one is that new releases have to get cut.00:38:51.000 --> 00:38:59.000To prepare for the Sapolia fork. And new releases then have to get cut again after that.00:38:59.000 --> 00:39:10.000To prepare for the for the whole ski fork. And then after that The main event.00:39:10.000 --> 00:39:18.000May not upgrade. So after all that testing. This is the and implementation part. Now we have to implement.00:39:18.000 --> 00:39:26.000And so it typically takes a few weeks from the last. Test net fork to make final changes if they're needed to the client repos.00:39:26.000 --> 00:39:36.000I said about 2 weeks, that's more based on like just seeing what's happened in the past takes about 2 weeks.00:39:36.000 --> 00:39:41.000And then after then an epoch, an epoch, a date of the upgrade is a set.00:39:41.000 --> 00:39:54.000You scheduled the coordination call the watch party everything else and then it proceeds you know the upgrade occurs And if we've done our jobs right.00:39:54.000 --> 00:40:06.000All the testing will have. Proven to be you know useful and we'll have a custom upgrade.00:40:06.000 --> 00:40:13.000But after you upgrade. We're actually still testing it in production, believe it or not. Right?00:40:13.000 --> 00:40:23.000So, there are behaviors that cannot be identified until00:40:23.000 --> 00:40:28.000We're actually being used, right? Like they're actually being used on, on main net.00:40:28.000 --> 00:40:35.000And so because of that, There might be some unanticipated behaviors or vectors of attack that could happen.00:40:35.000 --> 00:40:46.000So there is teams of security researchers and ethical hackers that are probing the protocol constantly to identify vulnerabilities to identify potential issues.00:40:46.000 --> 00:41:01.000They do receive payment for successfully identifying and appropriately disclosing these. It's always great to have a bug bounty and ethical hacking program because you want to lean into those members of the community who can add value through their, you know, sort of finding behaviors that aren't supposed to occur.00:41:01.000 --> 00:41:12.000So I have an example of right here. Here's a fork choice. So I have an example of right here.00:41:12.000 --> 00:41:14.000Here's that aren't supposed to occur. So I have an example of right here. Here's the fork choice bug fix disclosure.00:41:14.000 --> 00:41:28.000This was, you can see this is also on the Ethereum Hack MD. But basically there was, 4 choice attacks that, were identified that could able to cause long reorgs.00:41:28.000 --> 00:41:35.000You can actually go down. There's quite a bit of documentation and proof. Some of it is actually done via formal verification.00:41:35.000 --> 00:42:04.000So if you go into this, you could see attached proofs about how this is proven and this helps identify how changes you know can Can occurred that need to be implemented and sometimes these changes don't require like a full update you know full upgrade it can be just done with cutting a new client versions, right?00:42:04.000 --> 00:42:12.000Because We're trying to. And to better adhere to the spec, there might be things that need to be changed or removed.00:42:12.000 --> 00:42:25.000But basically, this just because a upgrade is done doesn't mean that testing stops, there's just a different set of group.00:42:25.000 --> 00:42:34.000Different set of use. Users who are testing and in this case it's at security researchers.00:42:34.000 --> 00:42:40.000Ethical hackers, protocol, sort of base people. So What we have then is during this whole process, there's a ton of discussion that's happening.00:42:40.000 --> 00:42:55.000So like I said, we have coordination calls, we have, you know, the Alcore Devs calls, the CL layer calls, there's constant talk, sometimes there's specific calls to discuss EIPs and the testing and implementation of those.00:42:55.000 --> 00:43:07.000There's a testing channel on E. Discord. There's channels related specifically to each upgrade.00:43:07.000 --> 00:43:13.000There's the interop channel that talks between the EL and CL and talks about the engine API.00:43:13.000 --> 00:43:19.000The client teams have their own discords and telegrams where there's talking about testing and test conditions.00:43:19.000 --> 00:43:27.000There's a hack MD. Documents and we haven't even really broached this yet but I think in the run up to.00:43:27.000 --> 00:43:39.000You know, events like Dev Connect or East Denver, you know, this course started after Dev Connect and it's gonna end before East Denver, at least this version of it.00:43:39.000 --> 00:43:54.000But those are opportunities, you know, these different. Community get togethers for actually a lot of the teams to get together in person or in a hybrid setting and actually talk and try stuff out.00:43:54.000 --> 00:44:13.000Prior to the merge, there was a huge amount of progress made in an August. Meet up that was in I think the greater Athens area where client teams researchers aren't you able to get together, and actually spend, I think 3 to 4 days together working specifically on some of the harder aspects of the merch.00:44:13.000 --> 00:44:27.000So as much as we do stuff and sort of this like always a virtual environment. There are times when it does make sense to convene people in person to spend time working to sort of push through and get stuff done.00:44:27.000 --> 00:44:40.000So, anywhere else, Matt, that you think the coordination is kind of happening as well.00:44:40.000 --> 00:44:49.000This is basically it.00:44:49.000 --> 00:44:50.000Yes, the magicians and all of that.00:44:50.000 --> 00:44:54.000Yeah, so a lot of I was gonna say a lot of this happens in the GitHub spec repositories.00:44:54.000 --> 00:45:02.000So the CL specs and the enterprise execution layers are not the enterprise. The execution. Layer specs so the eels.00:45:02.000 --> 00:45:06.000Yeah, the coordination is happening there, but a lot of the actual coordination happens in GitHub discussions.00:45:06.000 --> 00:45:22.000On individual spec PRs. So, EIPs might have one or more individual spec PRs and people will ask for comment and opinion from other teams on those PRs.00:45:22.000 --> 00:45:26.000So the discussion in GitHub actually becomes pretty important and it stays canonical too, which is nice.00:45:26.000 --> 00:45:31.000It doesn't really go anywhere. Yeah, that's the only other thing I would add to this.00:45:31.000 --> 00:45:36.000Everything else is very much covered.00:45:36.000 --> 00:45:41.000Awesome. Alright, so let's open it. We have some good time for QA. Feel free to type it in or come off.00:45:41.000 --> 00:45:53.000Mute, whatever your choice and then I'll try and just wrap it. We'll probably stop with like a minute ago to go over, you know, what's gonna happen.00:45:53.000 --> 00:46:19.000At office hours.00:46:19.000 --> 00:46:27.000If no questions, that's completely, completely fine.00:46:27.000 --> 00:46:28.000I'll give everyone a moment to think and I'll cover this. We go right to the end.00:46:28.000 --> 00:46:36.000So obviously office hours will answer any questions that people have as they go over the material. We can go in depth on any specific.00:46:36.000 --> 00:46:38.000You00:46:38.000 --> 00:47:02.000Questions that you have about any of these processes. So one thing that we won't cover is We will probably we will stay away from the Devnet side because that's like our final week of the class is sort of diving into the Dev Nets and that's specific for when we talk about the Dev Nets, we're going to be specifically talking about how Dev Nets applied to this upgrade.00:47:02.000 --> 00:47:20.000So, That's to more sort of dive into to these changes and. Yeah, and we do have a guest speaker coming up to speak about, so, we're gonna get that scheduled.00:47:20.000 --> 00:47:29.000We'll announce it on Friday. It will be Justin Florentine from the, Hi.00:47:29.000 --> 00:47:43.000And so, we'll put out that information and he will talk about testing probably with, you know, and that's a great person to then ask questions if you have them, not just about the generalized testing process, but like.00:47:43.000 --> 00:48:00.000Decisions around testing specific to that execution layer Ethereum find.00:48:00.000 --> 00:48:07.000Okay, cool. Well, if there are No questions. I don't wanna hold anyone. For any longer.00:48:07.000 --> 00:48:28.000So thank you so much for attending today's. Lecture, we'll see you on Friday if you can attend and just to make a plug, there are some additional Just that forks coming up, so, I will tell you right now.00:48:28.000 --> 00:48:35.000We will probably, we will definitely cover what happens in this a polio one at some point.00:48:35.000 --> 00:48:56.000And we while the course will be, technically in the sort of final project stage at this point, Matt and I are gonna try and pull something together a little special extra one for, we want to, you know, whether that's a debrief about it or depending on the timing, if it works out, something that we can kinda.00:48:56.000 --> 00:49:12.000Do a little bit of a watch. Party. We'll see about that, but we're gonna have there's gonna be at least one or 2 bonus, activities as part of this course to take advantage of the fact that Our timing got lucky, right?00:49:12.000 --> 00:49:23.000So better lucky than good sometimes. And so It's very nice that this course is timing up with the test net forks as we're kind of rolling into the finish.00:49:23.000 --> 00:49:30.000So, we will take it. We will take it. It went as planned. All right, everyone.00:49:30.000 --> 00:49:32.000Have a great rest of your weeks and all will see you, on Friday.00:49:32.000 --> 00:49:37.000Thanks everyoneOffice Hours Office Hours 5 Slideshow▼Office Hours Transcript The following transcript was auto-generated, and may content syntactical errors. 00:00:04.000 --> 00:00:18.000Kick off today is Friday? January nineteenth. And we're gonna have some office hours.00:00:18.000 --> 00:00:30.000So just to review. From last00:00:30.000 --> 00:00:46.000Lexter. We. Are always going through some sort of testing process. Whether we're testing the current fork or testing.00:00:46.000 --> 00:00:53.000Yeah, EIPs for inclusion and another fork. We might also always be testing. We might also be testing.00:00:53.000 --> 00:01:10.000Interpretations of the protocol as well. I think that's something that we kind of missed elaborating on, but what I mean by that is just that Sometimes, particularly when there is.00:01:10.000 --> 00:01:20.000The implementation of an EIP or of some change, other change to the Ethereum protocol, it may not.00:01:20.000 --> 00:01:29.000Be implemented as planned and so there's always testing to make sure that. There's adherence to the specification.00:01:29.000 --> 00:01:36.000So. There's always testing going on, but as we think about coming into any sort of upgrade.00:01:36.000 --> 00:01:55.000There are these initial test that give us signal about what needs to change. Those are, you know, the shadow forks and dev nets will go next week into more specifics about like sort of the nitty gritty of what happened on the Dev Nets.00:01:55.000 --> 00:02:07.000And then. When it's time for like. Prime time, we first start with dress rehearsals of Does, test net forks.00:02:07.000 --> 00:02:20.000And then those give a signal about what worked but didn't work. Make changes and that allows us to go and prepare into the, to the main net upgrade.00:02:20.000 --> 00:02:30.000So that's just an overview. I just wanted to open it up to any questions that folks had or any topics.00:02:30.000 --> 00:02:34.000To that would make sense to just go deeper on.00:02:34.000 --> 00:02:44.000So feel free to unmute. Or just drop any questions in the in the chat.00:02:44.000 --> 00:02:54.000Something that we pointed out, and this is in the slides is that, you know, there is this end-to-end testing suite for a theory.00:02:54.000 --> 00:03:01.000Of Hive and so we put that in there as that will replace retest teeth.00:03:01.000 --> 00:03:11.000Okay, so great. Question one. Which networks are accessible? To the general public.00:03:11.000 --> 00:03:21.000For trying out their own depths. Would this include an appropriate asset faucets? Would there be a page with all relevant information?00:03:21.000 --> 00:03:33.000Great question. So I'll start generally. Let's start outside of an upgrade. The DAP networks right now that folks can test on.00:03:33.000 --> 00:03:36.000There is.00:03:36.000 --> 00:03:48.000There is by. Coordination of the Ethereum Foundation and those who are running it. Only.00:03:48.000 --> 00:04:04.000One network that's really built for testing daps. And that. Would be, at this time.00:04:04.000 --> 00:04:14.000Sipolia. For Ethereum may not. So, Ethereum may not Sapolia because Gurley is deprecated.00:04:14.000 --> 00:04:23.000That is going to be the network for. Testing daps. There are Sepulia faucets.00:04:23.000 --> 00:04:33.000The etherium foundation does a very good job of like keeping up to date on where there are faucets.00:04:33.000 --> 00:04:41.000I would be remiss to say, you know, I, I'm wearing an inferior shirt right now.00:04:41.000 --> 00:04:50.000I work at consensus. We have a Sapolia faucet as well. So many other, our PC providers also have.00:04:50.000 --> 00:05:04.000Faucets as well. So if you want to get any sort of You can debt go to.00:05:04.000 --> 00:05:15.000The Ethereum Foundation website, you just go to the theorem.org and you're There is a like a.00:05:15.000 --> 00:05:19.000A section that says faucets, if you go to infuri. Dot.00:05:19.000 --> 00:05:26.000There's faucets if you go to, if you Google. S, you will find Sipolia faucets.00:05:26.000 --> 00:05:33.000There are a lot of faucets. That is for testing. If you just want to test at any given time.00:05:33.000 --> 00:05:39.000Adap and you wanna test it in a environment as close as possible to production, that's where you do it.00:05:39.000 --> 00:05:49.000Now, if you wanna test your DAP. In a Devnet situation or a shadow fork. That does require joining.00:05:49.000 --> 00:06:00.000The, channel and being in communication with the various teams that are testing it because That, Dev Nets and Shadow Forks.00:06:00.000 --> 00:06:10.000Tend to be a close validator staking set, right? So. If you have a DAP that has been deployed.00:06:10.000 --> 00:06:21.000To that. Network. And there's a shadow fork occurring. You would actually see what happens in that shadow fork.00:06:21.000 --> 00:06:31.000If you had the ability to interact with it, but that those shadow forks are, almost always not like open to the public, right?00:06:31.000 --> 00:06:39.000And that. There is not a, the coordination to like have that be an open test is very hard.00:06:39.000 --> 00:06:52.000So. If there's a reason that it makes sense to like test the DAP and for example, like if the DAP is making use of a certain new opcode that's introduced by EIP.00:06:52.000 --> 00:07:15.000That'd be a great reason to have that one be tested. That would require coordination with. All core devs with the The Ethereum like DevOps team is also sometimes called Panda ops just like if you hear us like throughout that name.00:07:15.000 --> 00:07:29.000Panda Ops is just a reference to the merge some visualization of the merge of like a like a black bear in a polar bear.00:07:29.000 --> 00:07:36.000Doing a little dance together and then becoming a panda bear so that's why that's, what the reference is.00:07:36.000 --> 00:07:48.000But if you wanted to test under the conditions, test adapt under the conditions of a. Upgrade, but wanted to get like a smaller set, you'd have to coordinate with them.00:07:48.000 --> 00:07:57.000Now, if you have a depth that's deployed to S, on the.00:07:57.000 --> 00:08:16.000The Sapolia. Upgrade is gonna happen on thirtieth January and that would be the best sort of like open set of people if you wanted to test out a depth deploy it before then see what happens and then get an opportunity to do it and you could just use Sapolia.00:08:16.000 --> 00:08:28.000So the is is now become the DAP testing layer and, Holeski is like the staking layer. So if you're like that, sorry, the staking sort of base test.00:08:28.000 --> 00:08:34.000So if you're looking to like test your staking validator setup, this test net will, will be the one.00:08:34.000 --> 00:08:44.000This will change over time, because What we're seeing is that, you know, test nets while there's no like actual crude value on them.00:08:44.000 --> 00:08:58.000They do like any other sort of computer system require resources to run. And so I would say the general conclusion is.00:08:58.000 --> 00:09:09.000That. It's easier to run a test net. That is. For DAP developers.00:09:09.000 --> 00:09:26.000At scale, if you have a closed validator set, basically like a closed number of set. Groups that are running the validators for it because it makes a little bit easier for them to ensure that that test that stays online.00:09:26.000 --> 00:09:43.000And if you're running a test net that is more like Testing a staking setup. Focused it's that's the one that you can keep open right it's because then it's an open set you're having validators join and the requirements are different if you think about it.00:09:43.000 --> 00:09:44.000Like adap doesn't need 30 to to deploy, right? 32 test. That's when you could do like a faucet.00:09:44.000 --> 00:09:50.000I's different if you think about it. Like, adap doesn't need 32, to deploy, right?00:09:50.000 --> 00:09:55.00032 test. Whereas like if you want to test your staking setup. You need like to have that 32 test, right?00:09:55.000 --> 00:10:07.000So it's just like a fundamentally sort of different. Setup. So great question. I went into a little bit more detail there, but just kind of wanted to flush it out.00:10:07.000 --> 00:10:17.000And then The page with all the information I can probably pull that up in a second. So let's go on to question 2.00:10:17.000 --> 00:10:24.000I would like to fork a client test modifications to the network framework. Would there be a way to do this without having to run a full-fledged network.00:10:24.000 --> 00:10:45.000I would say the best way to do that. Is to run it locally. Run it locally and The easiest way to do that is just to Spin up like you can spin up a client and also spin up multiple like other iterations of that client and have it peer with itself.00:10:45.000 --> 00:10:53.000And what you would need to decide is like what your starting state would be. So you could either say, I'm going to have my starting state be essentially a snapshot of a certain block number.00:10:53.000 --> 00:10:59.000And, and when I say this like, I'm assuming that you know how to set the configuration of the client, right?00:10:59.000 --> 00:11:15.000Like basically when you're doing startup, whether it's your ingesting that information via like a GAMMO file, any let's just say any sort of configuration file that you know what the starting state wants to be.00:11:15.000 --> 00:11:24.000So if you want to start from like, hey, we're gonna start from. Essentially.00:11:24.000 --> 00:11:32.000A coin like a coin based account just, you know, 0 x 0 0 0 and we're gonna start and try network.00:11:32.000 --> 00:11:45.000Framework that is essentially like recreating everything from scratch or if you want to say, you know what, I'm really interested in starting from the state, like doing a fork from 5 min ago. You could do that.00:11:45.000 --> 00:11:57.000You just need to set up your client and the peers appropriately. And then you can use that information to actually then fork and test modifications within the framework.00:11:57.000 --> 00:12:04.000So you're gonna run a network. But you're gonna run it locally. So it isn't as.00:12:04.000 --> 00:12:25.000Full, we used to do this all the time in the, one of the exercises in the developer boot camp was taking death and starting it and running basically a for node network back under like proof of work, but just doing it locally on.00:12:25.000 --> 00:12:32.000Your machine and you know there was a lot of laughs that would happen as everyone's machines start to you know go the fans spin and it sounds like it's an airplane taking off depending on how powerful their machines are.00:12:32.000 --> 00:12:46.000But now with proof of state, you can do this. You could be pretty strategic and like, wait about it.00:12:46.000 --> 00:13:02.000Additionally, if you want to. Even be more lightweight, there are different development frameworks that ship with a.00:13:02.000 --> 00:13:12.000With the sort of. Forks and you can reach out to those teams and see if that's something that they accommodate.00:13:12.000 --> 00:13:33.000Everything, you know, it's, it wouldn't be, for most of it wouldn't be out of the box because normally you're just trying to test the DAP versus test the client, but there could be some possibilities there and I'm talking about everything from remix, open source flavor of Ganesh opens hard hat Foundry all those they have some sort of version of like a00:13:33.000 --> 00:13:43.000forked version. So I'm not saying that like it would be easy to bolt on testing a client there, but it's something worth to talk to teams too.00:13:43.000 --> 00:13:54.000So those are kind of your options. And then finally during the testing, whether it be any data collection for vital signals of the network, say message is going gossip network, shaped with a blockchain with this data be accessible.00:13:54.000 --> 00:14:06.000Yes, there is. This is probably a good time for me to stop share and break out of my00:14:06.000 --> 00:14:21.000Pull up a page for you. Because we can talk about The, what is called the, the,00:14:21.000 --> 00:14:27.000Let me pull this up. I had this ready and then of course I had it ready and then I clicked off somewhere and I can't find it.00:14:27.000 --> 00:14:42.000Which, you know, always happens. This is the.00:14:42.000 --> 00:14:57.000First. And now, of00:14:57.000 --> 00:15:01.000I'm still sharing my screen. My goodness.00:15:01.000 --> 00:15:09.000Thank you. It's my first time using. This computer, but it is. Actually not.00:15:09.000 --> 00:15:15.000Okay, where did everything go?00:15:15.000 --> 00:15:21.000Come on, Zoom. Help me out. Help me out too. Okay, there we go.00:15:21.000 --> 00:15:26.000Goodness. I did not know it's going on there. Apologize to that. Okay.00:15:26.000 --> 00:15:36.000Here we go. We're back and share. Okay, so Girlie. This should be the, girly blob analysis.00:15:36.000 --> 00:15:52.000So this came out of. Yes. Upgraded. And so, This is the test, this is like the data collection and a write it up in hack MD on the, Panda ops.00:15:52.000 --> 00:15:59.000So it's talking about, hey, you know, we, the girly public test, it underwent the, the denkin fork.00:15:59.000 --> 00:16:09.0006 30 am universal coordinated time. There was a brief non finality incident. Let me just drop this in here, cause I think this is really interesting.00:16:09.000 --> 00:16:15.000This is like an initial report out, like, like we said in lecture, this is kind of what happens.00:16:15.000 --> 00:16:27.000There is like an initial report out and then We, we have some additional, reporting that's done, there is an issue, an issue analysis, which is awesome.00:16:27.000 --> 00:16:45.000So here is the debugging of that. And talks about What happened? Walks through all of it so We have that issue now.00:16:45.000 --> 00:17:02.000Data from precon was collected via centuries running. Eeli think it's like LLL is and you can kinda see this is the.00:17:02.000 --> 00:17:03.000Electric, that's what it is. Yeah. This is a multiplexer.00:17:03.000 --> 00:17:28.000It's additional. It allows multiple consensus clients to connect to a single execution client, which is Really makes me laugh because like when we were doing a bunch of education, work about the merge we.00:17:28.000 --> 00:17:32.000We're like, don't.00:17:32.000 --> 00:17:41.000Do this, right? Because it's suitable for monitoring analytics. Which is great. It's good for testing, but you will miss block proposals.00:17:41.000 --> 00:17:48.000Basically the idea is that you can't have multiple consensus layer clients driving the same execution that the same engine API, which is associated with that.00:17:48.000 --> 00:18:02.000The execution layer. So That's a little bit of a side, but, so we have, you know, the blob analysis.00:18:02.000 --> 00:18:09.000And everything is kind of walk through here, there's spanning of blobs.00:18:09.000 --> 00:18:18.000There was, you can see some screenshots dropped in here, how data was collected. Everything that was done, I think this is a great overview.00:18:18.000 --> 00:18:26.000And so we're collecting those P. 95 statistics. On what happened. There's some.00:18:26.000 --> 00:18:34.000Interesting, more mainly illustrative graphics dropped in there, but there is you know a heat map that you can view.00:18:34.000 --> 00:18:40.000There's the graph. I think this is all done. Yeah, it is done in and These are like the literal dashboard that you go into.00:18:40.000 --> 00:18:51.000So great question, cause this is, this is gonna force me to log in. Okay. I guess you'd have to have a login to do that.00:18:51.000 --> 00:18:56.000But I'm not exactly sure how to, to get that. That is probably a question.00:18:56.000 --> 00:19:00.000And, and I'm guessing it's a close set of people but This is like all the data about what happened.00:19:00.000 --> 00:19:09.000Kind of at a high level. So yes, I guess the short answer to the question is that, all of that data is here.00:19:09.000 --> 00:19:18.000It should be in some form accessible. That, all of that data is here. It should be in some form accessible.00:19:18.000 --> 00:19:24.000It does seem like it could be exported as well. It should be in some form accessible. It does seem like it could be exported as well. And it is shared around.00:19:24.000 --> 00:19:28.000And then there is, and while we don't have the login to see these graphs, it does seem like it could be exported as well, and it is shared around.00:19:28.000 --> 00:19:30.000And then there is, and while we don't have the login to see these graphs, it that these graphs do exist.00:19:30.000 --> 00:19:32.000So they are something that could be, you know, in talking to like, like someone else from that team that it should be accessible.00:19:32.000 --> 00:19:43.000So, yeah, that kind of answer. I hope that answers all 3 questions. Great questions.00:19:43.000 --> 00:19:52.000Happy to jump into, more.00:19:52.000 --> 00:19:53.000Yeah.00:19:53.000 --> 00:20:01.000I got another one. So like the tests are probably run by some script of. What they think are the things they want to test about.00:20:01.000 --> 00:20:11.000But because there's seems like So many different ways of getting in. To, there's so many different factors.00:20:11.000 --> 00:20:16.000I guess that's why they're limiting the network upgrades to just a couple of PRs.00:20:16.000 --> 00:20:29.000But, still there's still so many ways in so. It's amazing they don't miss new ways of of problems.00:20:29.000 --> 00:20:30.000Okay.00:20:30.000 --> 00:20:42.000You know, like in, I don't know, just like in. In remix where like where there's like our standard tests that we every time we add a new feature we add some tests for it but users always find some new way to break it.00:20:42.000 --> 00:20:43.000And, yeah.00:20:43.000 --> 00:20:53.000So, yeah. It is it is very impressive. That the number of issues. Our like I would say like it's an impressive.00:20:53.000 --> 00:21:04.000An exceedingly impressive rate of, catching issues beforehand. Now, to be clear.00:21:04.000 --> 00:21:22.000There are issues that pop up that have talked up. So we go a little bit more into this. Into the for choice bug that was discovered this one really did like this was a deep deep research issue.00:21:22.000 --> 00:21:39.000I'm not going to lie. Like I think that this was an issue that would have likely not been caught if not for like a really adept security researchers who have.00:21:39.000 --> 00:21:51.000The ability to like write proofs in like formal verification languages. Could do this because they basically found like edge cases that00:21:51.000 --> 00:22:02.000Could have been exploitable? But are so deep in there that we require like you know, Quite a bit of talent.00:22:02.000 --> 00:22:16.000To be able to discover them, though who knows with like. So the proliferation of AI, it is possible that a lot of these kind of like what we would once think of like, oh, you know, these are pretty deep issues, they might be more.00:22:16.000 --> 00:22:22.000I think there is an argument to be made that there's gonna be that sort of like.00:22:22.000 --> 00:22:36.000So only gonna accelerate sort of that. That you know finding bugs and fixing them but There actually is a very deep explanation of this issue.00:22:36.000 --> 00:22:46.000Roberto. Sultani, who's one of the. The00:22:46.000 --> 00:22:52.000The researchers on this, I was fortunate enough to get to spend some really nice time. With him.00:22:52.000 --> 00:23:03.000In a small group setting. To really like talk through what, they, what they figured out.00:23:03.000 --> 00:23:12.000And. It was. I am not going to necessarily be able to.00:23:12.000 --> 00:23:19.000Do you do justice? I've probably, I, he's also shared some like a talk that he gave around this.00:23:19.000 --> 00:23:32.000But, what was realized was that There were possibilities of reorging with for a non justifiable reason like.00:23:32.000 --> 00:23:53.000So, so the issue was essentially was. The way that the specification was written. And the way it was implemented did not match essentially but on these like sort of edge cases so that there was an attack vector where there could be the possibility of a reorg.00:23:53.000 --> 00:24:01.000That could occur that would not be a justified reorg. And it one of the results was that.00:24:01.000 --> 00:24:16.000Validators who acted honestly could be slashed. So this was discovered and. There was at you know full00:24:16.000 --> 00:24:23.000I mean, I think this is one that was done by Roberto like at a station. Denlocks and then.00:24:23.000 --> 00:24:36.000That there was the, this is how it's gonna be. Changed so This was a really so you know it is very and impressive that all of this was like figured out.00:24:36.000 --> 00:24:50.000Here is the manual proof list. And this goes like. Super deep. If you want to like start clicking down, I'll share this like again.00:24:50.000 --> 00:24:58.000So you can just see. But as you keep on like going down, if you really want to get into some some.00:24:58.000 --> 00:25:09.000Interesting. Work, it just. You can see the spec versions in there, but this goes into.00:25:09.000 --> 00:25:30.000Sort of the assumptions and then like. How do you prove that this thing but what was supposed to happen was not happening so this whole00:25:30.000 --> 00:25:39.000Real, math and justification going on. But to your point, Yes, it is possible. That we can get something.00:25:39.000 --> 00:25:53.000Live and there's still be some, gaps in there, but we're very fortunate that For the most part.00:25:53.000 --> 00:26:02.000The vast majority of the hit like expected behaviors. Our, are observed. And the upgrade is able to occur.00:26:02.000 --> 00:26:14.000Now to to talk about sort of an interesting situation. Let's talk about it. This one is, this is like an open for debate.00:26:14.000 --> 00:26:29.000Question, but like at some point, right? We have to say. Is there adjusted like, right, is a good example.00:26:29.000 --> 00:26:36.000Was that an intended part of the protocol? Is that an intended part of the protocol is that an intended part of the protocol, is that an intended part of the protocol or not?00:26:36.000 --> 00:26:47.000I think we've We've come to say, well, you know, It's something that exists and there are benefits.00:26:47.000 --> 00:26:51.000There's positive benefits. There's negative benefit. There's, there's not negative benefits.00:26:51.000 --> 00:26:57.000There's, there's negatives to it, but what you can also see is sometimes that.00:26:57.000 --> 00:27:11.000Parts of the protocol, there is a inability to always predict what that would. How users will. End up interacting with the protocol.00:27:11.000 --> 00:27:32.000And so then there has to be decisions around. You know, is that now just part of it and we're saying it's included a lot of times, you know, that is seen the like code is law sort of subset of of folks that are in this space and that is a group that, I think, you know, it's very important to kinda like take that into consideration.00:27:32.000 --> 00:27:57.000On the other hand, there is a like, hey, the intent wasn't for that. So, self-destruct, we're starting to see, you know, all that work around self-destruct and that's gone on and modifying the opcode is basically saying like, hey, this was the original intention and yes, could his law but like because of things that are going on with00:27:57.000 --> 00:28:03.000it, can we modify the opcode? So it's always like there isn't like a hard fast rule.00:28:03.000 --> 00:28:11.000It does feel like there is always this sort of like tension and pull between various groups within all Cordes within the Ethereum community.00:28:11.000 --> 00:28:21.000But what's cool about it is that we end up coming to . A Enough of a consensus that we can move forward to keep on.00:28:21.000 --> 00:28:32.000Keep the protocol going. Now, obviously doing upgrades like this, we have said, you know, there is a certain desire to constantly move.00:28:32.000 --> 00:28:39.000You know, taking information and react. So as much as that like code is law, there is a desire to say like, hey, you're not going to get everything right.00:28:39.000 --> 00:28:51.000You need to be able to. Constantly update and improve. And so that Tension is is a good one because it Makes us respect the fact that like, hey, there are decisions made.00:28:51.000 --> 00:29:06.000We're just not gonna try and not take those decisions into account, but at the same time, like a protocol has to evolve to meet used cases and what people want to do with it.00:29:06.000 --> 00:29:16.000And in the case of Roberto's. Proof. Like, so you found a problem. And then, then there was probably.00:29:16.000 --> 00:29:20.000How long did it take for them to did they address it or did they? Hi, did it have to wait?00:29:20.000 --> 00:29:34.000They did address it. I think that the time horizon to address it was ended up being months. So which sounds about right, you know, that seems, it seems like that seems pretty fast.00:29:34.000 --> 00:29:48.000Like, like pretty reactive for like such a deep. Proof that they that had to do but it see from my memory it was like identified and and you know00:29:48.000 --> 00:29:55.000A solution was come up with and that was all be able to be implemented over the course of months.00:29:55.000 --> 00:30:02.000But it does that it was presented. It was like fairly disclosed. And all that. So it was part of like a.00:30:02.000 --> 00:30:20.000A an an ethical security researcher work group. I think there that there was a compensation received for identifying that and fairly disclosing it under the terms of a better set out.00:30:20.000 --> 00:30:31.000It didn't make a new. Fork network for good. I mean, yeah. Did it or when they up fixed it?00:30:31.000 --> 00:30:32.000I think.00:30:32.000 --> 00:30:35.000It didn't my understanding is not it was more of a client implementation change that could be made. Yeah, yeah.00:30:35.000 --> 00:30:46.000Without going into the deep specifics of it, I think there was like a specific. Field that was able to be some parameters on that were able to be modified.00:30:46.000 --> 00:30:53.000And so that's a little bit quicker because it just requires the client teams to cut a new version.00:30:53.000 --> 00:30:59.000Yeah.00:30:59.000 --> 00:31:08.000So as much as you kind of laid this out linearly, I mean, I would I probably need to get a good make a good diagram.00:31:08.000 --> 00:31:18.000Long term that shows like it's really like circles upon circles of testing. And, and usage and so yeah.00:31:18.000 --> 00:31:33.000Any other questions? Great questions.00:31:33.000 --> 00:31:40.000Okay, well, everyone thinks about if there are any final questions. Do not take the whole time, but did want to.00:31:40.000 --> 00:31:53.000Just give some, final, announcement. So on Monday, this is on the course calendar just in Florentine from the basic team is going to explain testing as it relates to him working on an Ethereum execution layer client.00:31:53.000 --> 00:32:03.000So we'll go in depth. It will be sort of in his focus. And Matt will be running that session with with Justin.00:32:03.000 --> 00:32:06.000Any specific questions you have, if you sort of review anything I shared and you're like, I'm really interested in finding more about this tooling.00:32:06.000 --> 00:32:15.000He's great person asked that recording will be posted. It might be a little delayed in getting it posted.00:32:15.000 --> 00:32:23.000We have that next week off. Matt and I have some prior obligations that are just gonna make it, you know.00:32:23.000 --> 00:32:35.000Tough for us to do the Wednesday in the Friday office hours. But then we come back and we have a, our final week where we talk about, you know, really go in depth on Dev Nets.00:32:35.000 --> 00:32:38.000On the00:32:38.000 --> 00:32:48.000On the thirtieth and then instead of having office hours on the second what we're going to do is push them to Wednesday, February, the seventh to coincide with the Holski upgrade.00:32:48.000 --> 00:32:50.000Hello.00:32:50.000 --> 00:32:57.000So that way we can just keep the conversation going. So we won't have office hours that are typical Friday time.00:32:57.000 --> 00:33:10.000We'll have us will push the office hours on the whole ski upgrade date just so we because at that point we'll have all the test nets upgrade date just so we because at that point we'll have all the testnuts upgraded and that's like a good way to wrap and summarize.00:33:10.000 --> 00:33:21.000Probably it also that one will. Talk about bonus activities. And we are also probably going to at that point.00:33:21.000 --> 00:33:46.000Talk about the opportunities that we might have to do a little. if possible, either a hybrid or in person get together as we get closer to, one of the, what I would call like a real cornerstone event of the Ethereum community, alongside the likes of, you know, the, the global series.00:33:46.000 --> 00:33:55.000Devconnect, Devcon, East Denver. So, many of us will, be there.00:33:55.000 --> 00:33:59.000And so, we're trying to figure out, is there a way to just informally get together, maybe over a coffee.00:33:59.000 --> 00:34:11.000And just say hi to anyone who's in the course. Who might be in the area and talk about what we've learned.00:34:11.000 --> 00:34:19.000And see if we can gather some of our our guest speakers as well just to come by and say hi so it would be very informal but.00:34:19.000 --> 00:34:24.000Yeah, that's right thinking.00:34:24.000 --> 00:34:25.000Sounds good. We can meet at Dandy's house.00:34:25.000 --> 00:34:33.000Alright. Yeah, that would be nice. I'll bring, I'll bring a bunch of coffee and, I'll just.00:34:33.000 --> 00:34:34.000Yeah.00:34:34.000 --> 00:34:46.000Get caffeinated. Pull out a whiteboard and and and write some write some mathematical proofs.00:34:46.000 --> 00:34:58.000Alright, well if no more questions that was really great to hear everyone guest speaker on Monday. Just to, so you should see that in the calendar and, We're really excited for that.00:34:58.000 --> 00:35:08.000We'll have a week off from lecture and then we're back. With our final official lecture of the course on the I said the thirtieth by minute 30 first and then we're pushing back like I said office hours to be the day it's something that Friday I've already moved that that should update it that should have updated in the calendar.00:35:08.000 --> 00:35:28.000All rightGuest SpeakerJustin Florentine - Besu - Consensys ▼Guest Speaker Transcript00:04:37.150 --> 00:04:47.520 Matt Nelson: all right. Thank you for joining folks. Today is week number 4, guest speaker number 3 of our upgrading ethereum network 20 00:04:47.710 --> 00:04:50.759 Matt Nelson: upgrading the ethereum network lecture series. 21 00:04:51.100 --> 00:05:11.770 Matt Nelson: And today's topic is going to be a guest speaker on the week that we discussed testing and implementation of the hardworks. So we have with us today. Justin Florentine is a member of of my team here, consensus on the base. 2 team who was the core developer. It's been a big part of delivering the Cancun upgrade on the execution there. 22 00:05:11.770 --> 00:05:24.069 Matt Nelson: so I won't give too much away. But we'll give let you do a little intro Justin and then we can kind of go into it. I'll start with some initial questions, or if you'd like to start us off in a particular spot. 23 00:05:24.210 --> 00:05:28.840 Matt Nelson: But yeah, again, the context for this this week of lecture was last. We discussed. 24 00:05:29.080 --> 00:05:50.650 Matt Nelson: How? Core devs coordinate around things like Devnett's specification updates testnet forks. We discussed the gorilla fork briefly. So there, there's context already. There there's also context around the eip process. How those are written, how those are coordinated! We taught. Had puja from Eve cat Herters come on and discuss all that stuff. 25 00:05:50.650 --> 00:06:00.260 Matt Nelson: So the the students should have a great kind of understanding of how we get to the point of scope and how we start to put things in practice. But I think your perspective will be super helpful on like. 26 00:06:00.390 --> 00:06:06.380 Matt Nelson: okay, we have kind of a defined list of items. Now we have to get started with. 27 00:06:06.390 --> 00:06:17.980 Matt Nelson: you know, implementation, which is in many cases kind of the step one or the easy step until we start to break stuff with interoperability testing. And then we have, of course, devnets and testnets. 28 00:06:18.060 --> 00:06:27.700 Matt Nelson: so I'll take a pause here. I'll let you intro, and then, if you'd like to start anywhere feel free. If not, I have some questions prepared that we can go through to discuss, but I'll turn over to you. 29 00:06:28.570 --> 00:06:38.200 Justin Florentine (Besu): Cool. Yeah. Hi, Justin Florentine. I'm an engineer on the Baseu team. You'll find me on socials as Robocop's gone mad, or some variant thereof. 30 00:06:38.260 --> 00:06:49.589 Justin Florentine (Besu): I've been working with consensus since 21 right around London. I think London was the first hard fork that I worked on. I worked on pretty much every hard fork since then, except for Shanghai. 31 00:06:49.760 --> 00:06:55.110 Justin Florentine (Besu): So that's you know, London, Paris. And now, cancun 32 00:06:55.680 --> 00:07:11.510 Justin Florentine (Besu): you will find me hanging out in discords various times when it's time for interop testing ethicians, forums, etc., etc. That's where a lot of the hanging out and finding out what needs to be done where we're at, etc., etc. goes on. 33 00:07:11.540 --> 00:07:13.310 Justin Florentine (Besu): So yeah, happy to be here. 34 00:07:14.910 --> 00:07:25.940 Matt Nelson: Awesome, Justin. So let's think this, let's start these questions kind of in terms of of testing apparatus. Right. So zoom, we've had a 35 00:07:26.030 --> 00:07:34.290 Matt Nelson: rousing set of scoping discussions around the Cancun Hard Fork, and you are a core developer on the basic team who's been given 36 00:07:34.510 --> 00:07:43.560 Matt Nelson: a handful of things to implement as well as a date for a first devnet launch. What does that kind of look like when we're going through? 37 00:07:43.790 --> 00:07:54.269 Matt Nelson: again. You know, there's the broad specs, broad strokes applying the specs. But then we actually have to connect the nodes and get that started. What's kind of those first little steps there. And how do we start to kind of 38 00:07:54.470 --> 00:08:02.489 Matt Nelson: iteratively patch the clients to get them all talking to each other? Get them all working on the same page, and, you know, go from there. 39 00:08:03.490 --> 00:08:15.800 Justin Florentine (Besu): So there's a couple of different aspects that you want to think about when you're when you're when you're discussing how you get from decided specs to clients running and processing blocks on a network. 40 00:08:15.870 --> 00:08:28.660 Justin Florentine (Besu): The first is a traditional software engineering challenge. Right? So this is open source software, which means you don't necessarily have quite the rigorous top down software design and implementation that you would have elsewhere. 41 00:08:28.890 --> 00:08:50.330 Justin Florentine (Besu): So you need to have discussions with your community. You need to decide. You know what are the. You know we. We understand what the actual features to be delivered in a hard fork are, and the granularity you're given is the eips themselves. So there'll be a design document that'll express some sort of feature that we wanna add to ethereum? It's scheduled to be delivered in this hard fork. Cool? 42 00:08:50.330 --> 00:09:08.250 Justin Florentine (Besu): What are the software tasks that need to happen for that? Is there an area of the code that needs to be refactored? Is there a place where this obviously is a quick hit and will be easy to implement? Who is already familiar with that part of the code and wants to work on. So all of that needs to happen. As you're preparing for the hard work, and as you start delivering. 43 00:09:08.710 --> 00:09:13.890 Justin Florentine (Besu): you're going to be discussing with other client teams in discord 44 00:09:14.650 --> 00:09:19.049 Justin Florentine (Besu): how we know the various eips work. so there will be 45 00:09:19.090 --> 00:09:47.319 Justin Florentine (Besu): usually a series of tests that arrive in a number of different ways. We have reference tests. We have state tests. The big one these days is the hive tests, which is a full on systems test. It's actually running clients. It's interacting with them over the engine Apis to drive them from a test Genesis state to a head block that is theoretically past the fork that we're designing. It has all the features that we're trying to build. 46 00:09:47.320 --> 00:10:04.150 Justin Florentine (Besu): and there's a lot of discussion amongst the teams around that. So they will say, Hey, here's a tive test. Is this actually valid? Great? Ef does does a lot of investment on that. They spent a lot of time and effort on testing. We have really good test coverage cover coverage in these different dimensions. 47 00:10:04.170 --> 00:10:15.409 Justin Florentine (Besu): And that's and I think that's really the shelling point for a lot of getting this work done is the tests. The tests really do drive it. And they are granular enough that you can discuss it. And we implement these tests when they're passing. 48 00:10:15.410 --> 00:10:39.240 Justin Florentine (Besu): We discuss with folks when they're not most the time it's on us a lot of times we find things in the test that are either inaccurate or ambiguous, and sometimes it'll go back to the spec. So a lot of times. People have been joking lately like we're still making changes to the spec for Dan Coon, you know. Hey, we kind of are but a lot of these things are very, very small things. They're they're reducing ambiguities that developers may have run into. 49 00:10:39.360 --> 00:10:55.329 Justin Florentine (Besu): So that's you know. That's kind of the I think the the high vision of of how that works. You iterate on that stuff until we're all in a discord room somewhere, saying, Yeah, we're passing all these things, we feel good about it. What do we do next? Great, we're gonna build test network 50 00:10:55.450 --> 00:11:01.869 Justin Florentine (Besu): it'll be a completely fake network. The Ef will spin up a whole bunch of their devops stuff, they'd use it. 51 00:11:02.080 --> 00:11:21.590 Justin Florentine (Besu): Excessive automation on all of this. They can create these new networks. They will socialize that and say, like, Hey, we made a fake network. Here's a bunch of tools that we're actually running on it for people to watch. So when a new network comes out I'll be like, Hey, Matt? E stats is up over here. You can see how things are going, and they'll be an E stats implementation, and they will be running all the different versions of our clients 52 00:11:22.040 --> 00:11:41.400 Justin Florentine (Besu): when they run into an issue they'll contact whoever they've been working with as the point of contact for that for that hard fork. So some. It's nice for us to have one team member kind of quarterback that, and and try to coordinate that information and keep it moving. That could be anybody, any release? And so you know, that happens. 53 00:11:41.730 --> 00:11:48.869 Justin Florentine (Besu): we work the work, the issues that we run into on the debnets, we find the bugs. And again, it's really your standard Sdlc process at that point 54 00:11:49.640 --> 00:11:55.869 Justin Florentine (Besu): find an issue, report an issue. If you can fix it, fix it. Somebody else could help you fix it. Great, awesome 55 00:11:56.220 --> 00:12:03.200 Justin Florentine (Besu): and that's that's usually the first step is like the fully synthetic test nets the completely constructed one. 56 00:12:03.350 --> 00:12:25.389 Justin Florentine (Besu): Once we've done a few of those we feel really good about it, we'll say, like cool. Let's Shadow fork, a test network like a proper test network. So like girly a named test network, polesky sepolia, those are the those are the ones that we have today. Ripe girly. Bye, bye, this is the last. This is the last for off for girly, so say your goodbyes. Raise a glass. 57 00:12:25.890 --> 00:12:28.970 Justin Florentine (Besu): It's served as well. and 58 00:12:29.090 --> 00:12:40.569 Justin Florentine (Besu): we will take a shadow fork, which is a technique that we developed during the merge. Actually, I think, was the first time that we started doing that where we'll intentionally fork the network. So you'll have 59 00:12:41.090 --> 00:12:54.219 Justin Florentine (Besu): a much smaller subset that'll get real transactions, and it'll create its own blocks with the new rules for the Hard Fork while the other network is still running on the old rules. So, for instance, during the merge Mainet would keep 60 00:12:54.260 --> 00:12:56.440 Justin Florentine (Besu): chug it along on London specs. 61 00:12:56.540 --> 00:13:17.960 Justin Florentine (Besu): and we'd set up a whole bunch of networks, a whole bunch of nodes that are on that network, but now have paras specs, and they will intentionally fork off, and they'll continue to run valid transactions against valid main net state for as long as they can like. At some point these things decay. Because you're running contracts that they don't know about the new features. And they're gonna they're gonna start breaking it. 62 00:13:17.960 --> 00:13:27.270 Justin Florentine (Besu): But it's a really really helpful technique that we've developed that we've we're so happy with how it informed the merge that we're just gonna do it all the time. Now. 63 00:13:27.920 --> 00:13:32.120 Justin Florentine (Besu): So there was a lot of that for Shanghai. There's a lot of that going on right now. 64 00:13:32.140 --> 00:13:46.009 Justin Florentine (Besu): We have shadow forked girly a number of times until we were happy with it. And now we've actually fully forked girly. If you're running a note on gorily, and it's keeping up with head congratulations. You're running Dan Coon software. 65 00:13:46.600 --> 00:13:53.869 Justin Florentine (Besu): We then repeat that process for subsequent networks. Next up will be hulesky, and then sepolia 66 00:13:54.380 --> 00:13:57.940 Justin Florentine (Besu): I don't know if I have the more the order, right? It's the other way around. Yeah. 67 00:13:58.510 --> 00:14:03.830 Justin Florentine (Besu): Sopolo and Hillesky, and then we'll do shadow forks for me net and 68 00:14:04.190 --> 00:14:08.550 Justin Florentine (Besu): wash rinse. Repeat until we're super super confident that we're ready to get to main net. 69 00:14:09.190 --> 00:14:22.380 Justin Florentine (Besu): yeah. So that's that's a big, complicated, very thorough process that we use to deploy new features on a very expensive network without without dropping 70 00:14:22.840 --> 00:14:24.520 Justin Florentine (Besu): customer transactions. 71 00:14:26.870 --> 00:14:37.970 Matt Nelson: Yeah, that's that's awesome overview. Tell me a little bit more about how the kind of collaborative testing process works. Mostly being because, you know, we're all kind of core developers 72 00:14:38.150 --> 00:14:50.030 Matt Nelson: to outside observers. It might seem that we're all we're we're kind of creating the tests we're creating the specs we are doing all of the testing ourselves. It's kind of like, you know. 73 00:14:50.540 --> 00:15:12.110 Matt Nelson: could seem like a single group is doing all this. But in reality it's a lot of different people. How does that kind of collaborative testing environment come about? You know, what are the orgs involved? And how does it kind of look from a perspective where some might say, Okay, no. It's actually just the same groups doing the same things kind of patting themselves on the back when in reality there's a huge diversity of 74 00:15:12.560 --> 00:15:14.849 Matt Nelson: stuff that goes into that testing in reality. 75 00:15:16.120 --> 00:15:17.780 Justin Florentine (Besu): Well, 76 00:15:17.980 --> 00:15:23.589 Justin Florentine (Besu): yeah. So we get contributions from the community. Each person 77 00:15:23.700 --> 00:15:45.899 Justin Florentine (Besu): providing what they're what they're best at. Right? So a lot of folks that are, familiar with driving Json, Rpc. Api's would be contributing to the hive tests right? Folks that are more interested or well versed with lower level Evm tests could be submitting reference tests. And those are those are very different. Kind of opposite ends of the spectrum. 78 00:15:47.100 --> 00:16:15.379 Justin Florentine (Besu): You know. So so you you could get contributions from just about anyone and it they just sort of get socialized and filtered amongst the other teams through, mostly discord. But all this stuff is tracked in version control. You know, there's a project out there for reference tests. There's a project out there for the hive tests, and those are really just traditional software management problems, right? So as folks create new issues or find issues or fix ambiguities. 79 00:16:15.530 --> 00:16:38.930 Justin Florentine (Besu): they create Prs, they get on to main. And then someone will say, Hey, Justin, that test that you were complaining about, that you thought we implemented wrong. Well, you were wrong, and here's how, and they'll have a test that you know clarifies the spec or fixes and ambiguity, etc., etc. And you just pull those tests. You run them against your your client software, and you move forward from there. And so you do that, you you. 80 00:16:39.380 --> 00:16:58.850 Justin Florentine (Besu): if or whoever's running the test, knows that that that was done, they'll pull main branch of your client, run it against their tests to confirm, and they'll continue doing that. So they, whoever whoever's running these tests is always staying very current on the bleeding edge of the corresponding clients that are being tested. So you have a pretty tight feedback loop. 81 00:16:58.850 --> 00:17:10.039 Justin Florentine (Besu): You'll hear something the next day, if like something on nightly went out and broke broke a test, because they're just constantly and automatically usually pulling those new client updates and running existing tests on. 82 00:17:11.920 --> 00:17:23.369 Matt Nelson: yeah, that's great. So I mean, I know that in context of what we've seen on main net, there have been a few bugs and clients, consensus issues that have actually stopped kind of nodes in the network. 83 00:17:23.390 --> 00:17:33.950 Matt Nelson: The reason I'm asking about this is because I think in Devnet's and testnets, it's a little bit different. How do we necessarily kind of look at consensus issues when there may not be 84 00:17:34.010 --> 00:17:42.649 Matt Nelson: a consensus among the clients? Do we always kind of defer to the tests? And in some cases right. The tests are written from perspective of 85 00:17:42.760 --> 00:17:59.230 Matt Nelson: kind of one team, often, or one clients implementation. How do we start from the building block where every block isn't valid, and then kind of slowly progress? These client specs because it's really very much like you said it. It might even be from one commit to the next. 86 00:17:59.250 --> 00:18:04.410 Matt Nelson: as far as, like the tip of a branch is concerned with Github that can cause an issue. 87 00:18:04.570 --> 00:18:26.680 Justin Florentine (Besu): So that happens through the the redundant design of the testing pipeline right? The fact that we're actually running these hive tests like I run high test locally against no network. Sometime it'll just be what the hive test run against a single instance of base, which is a very, very, very simple form of doing that, but it's easy to reproduce, and it's easy to iterate on. It's easy to repeat it over and over and over again. So 88 00:18:26.970 --> 00:18:39.910 Justin Florentine (Besu): once we go through that we go through the next layer and the next layer, and then you're doing. You know your shadow forking things, your shadow, forking testnets, etc., etc. You've done it so many times that it's very rare that 89 00:18:40.120 --> 00:18:48.909 Justin Florentine (Besu): You found a bug that halts the network right? Or you've you found like a really weird edge case or a 90 00:18:49.680 --> 00:18:59.779 Justin Florentine (Besu): spec issue like, those are the things that we really worry about is like it's the spec wrong. And in most of those cases, apart from like a proper test network. 91 00:19:00.270 --> 00:19:08.309 Justin Florentine (Besu): it doesn't matter if it's a shadow fork fine, you can. Just, you know. Let the Shadow Fork die and start over with your adjusted specs. 92 00:19:08.660 --> 00:19:10.520 Justin Florentine (Besu): So 93 00:19:11.040 --> 00:19:25.209 Justin Florentine (Besu): it it's it's a good question, because the stakes are really really high. And I think the answer is redundancy, like the answer is, we're just re really, really redundant. It's really, really important. And we're kind of fortunate that an open source community that we can just do that. I think if we were in a commercial sense. 94 00:19:25.210 --> 00:19:41.199 Justin Florentine (Besu): the cost benefit analysis would not fly past some being calendar and accounting, and they would tell you like, Oh, you're you're this is too redundant. You're you're you're you're wasting money, etc., etc. But you know it's it's live. There's no, there's no better way to do that. Redundancy is important. 95 00:19:44.450 --> 00:19:52.059 Matt Nelson: Yeah, redundancy is key, and we we definitely have redundancy among these test test nets, where we have 96 00:19:52.400 --> 00:19:56.709 Matt Nelson: tens of clients of each combination size, shape. 97 00:19:57.350 --> 00:20:26.729 Matt Nelson: very cool. I wanted to take 2 s to see if we have any questions that folks on the phone wanted to provide. Again, the context for today's discussion is testing and implementation for the hardworks. And as I mentioned the intro, some of you might not have been on by that point that Justin has been working on the basic team to get cancun developed, tested and shipped. So anything in that context. But you also have a core developer on the phone. Anything else you wanna chat about is is likely fair game here. 98 00:20:26.900 --> 00:20:30.229 Matt Nelson: If not, I will keep probing, probing with questions. 99 00:20:31.790 --> 00:20:41.889 Matt Nelson: also the eips that we discussed where there was potentially some ambiguity. I know we went through the eips themselves in depth. So you have more detailed spec questions that I was either Iffy 100 00:20:41.990 --> 00:20:43.090 Matt Nelson: or 101 00:20:43.340 --> 00:20:49.679 Matt Nelson: you know we we have Justin here who worked on a lot of those specs themselves, so that'd be great. 102 00:20:50.060 --> 00:20:51.170 Matt Nelson: But hmm. 103 00:21:00.000 --> 00:21:02.929 Matt Nelson: alright! I got another question for you while we wait here, Justin. 104 00:21:03.090 --> 00:21:09.810 Matt Nelson: tell us about the girly fork post mortem. We very briefly walked through what happened. 105 00:21:10.010 --> 00:21:20.889 Matt Nelson: As far as the prism client bug and potential nethermine client bug for the last fork. But we actually didn't have enough time for a proper analysis, because girly was forked, maybe 106 00:21:21.040 --> 00:21:31.449 Matt Nelson: a handful of hours before our call last time. So do you have anything that any lessons learned or anything that would be, you know, interesting in context of this discussion that we learned from that hard fork, how we're going to apply those. 107 00:21:32.010 --> 00:21:33.520 Yeah. So 108 00:21:34.040 --> 00:21:43.039 Justin Florentine (Besu): now you forgive me. I was on vacation this weekend and the day before, so I didn't go too deep into the root causes of what we ran into. But this was 109 00:21:43.390 --> 00:21:57.420 Justin Florentine (Besu): the important question is is like, Oh, how do we miss something right here, because I just went on and on about redundancy, and we're still found something that broke on a test net with, you know, one of our one of our clients. That was interesting, because 110 00:21:59.030 --> 00:22:12.330 Justin Florentine (Besu): it it probably should have showed up in one of the shadow forks. We're a little suspicious as to why it didn't, but the nature of the bug is such that it would not have showed up in hive. It would not have showed up in 111 00:22:12.670 --> 00:22:17.539 Justin Florentine (Besu): The. You know, the ephemeral networks that we're spinning up because my understanding of what 112 00:22:17.800 --> 00:22:27.489 Justin Florentine (Besu): prism ran into had to do with historical receipts that were older than a certain point. So so they had a situation where. 113 00:22:28.100 --> 00:22:53.959 Justin Florentine (Besu): when in our smaller networks, we were creating the 4.4 dancoon, a little too close to origin or to to Genesis. And so we weren't hitting certain paths in the code that are a little bit more relevant when you have a much longer distance there. So that was kind of a a weird edge case that we're gonna have to really consider how we cover with our tests because you don't wanna have like. 114 00:22:54.200 --> 00:23:14.820 Justin Florentine (Besu): you don't wanna have days between your genesis for your test network when you actually hit the thing that you want to test, you know. So we'll have to take a closer look at that, and just a and and realize well, was it a length of the blockchain issue that exacerbated this, or was it a state sized thing, or whatever? It's probably the former, because it had to do the 115 00:23:14.820 --> 00:23:25.940 Justin Florentine (Besu): something with the receipts. I don't. I don't quite recall off the top of my head. But but it would be a new test scenario that we would have to introduce there. To specifically target that. 116 00:23:26.340 --> 00:23:27.949 Justin Florentine (Besu): so yeah, interesting. 117 00:23:28.900 --> 00:23:52.459 Matt Nelson: Yeah, I think that's the the prism. One was especially interesting because this it was sort of an optional spec implementation detail around the way we index blobs, which is part of the reason why we have redundancy, because the spec is intentionally left a little bit ambiguous in certain instances to allow client diversity to solve some of these problems just by virtue of 118 00:23:52.560 --> 00:24:09.070 Matt Nelson: whomever is thinking over the spec in a slightly different way. We do this in many ways by the ways that the Cls will handle Mev blocks or block production in general. Some of them have the. They've implemented different kind of censorship, resistant overrides for these things that have different timeouts. 119 00:24:09.070 --> 00:24:23.010 Matt Nelson: different ways that they kick in and then different present present presentations to the user. But the I remember specifically at a a core Devs call where we said, no, we do not want to codify this in the spec intentionally, to give 120 00:24:23.030 --> 00:24:36.379 Matt Nelson: openness, so that the clients can be a little different than each other, because if we had an extremely rigid spec, they would all look and act identical, which means they would all present potential spec challenges in the same way. 121 00:24:36.630 --> 00:24:49.389 Matt Nelson: so in this instance, it can create kind of issues where clients will have a little bit, maybe too ambiguous or loose interpretation of the spec, and can cause issues. But in in general it's a kind of a good thing for sure. 122 00:24:51.970 --> 00:25:10.409 Matt Nelson: Okay, we got a question in chat from Tj. The question is, what is the process for a new client like ref to be considered legit enough to be an official client. Couldn't any random group of individuals create an execution or consensus client? Obviously we don't want pro poor performing clients to drag down the rest of the teams during the upgrade process. 123 00:25:10.490 --> 00:25:14.150 Matt Nelson: So I'll give a 2 s version. Then I'll kick to Justin 124 00:25:14.750 --> 00:25:16.549 Matt Nelson: In many instances 125 00:25:16.720 --> 00:25:33.780 Matt Nelson: we kind of just say, Hey, if you can keep up with the status quo welcome, we're seeing new consensus and execution clients like ethereum js and ref, and on the consensus side we have a new client called Grandine, which actually produced an invalid block 126 00:25:33.780 --> 00:26:02.259 Matt Nelson: that tripped up a lot of the other clients on on some of these forks. So in reality, I think we we test against a common denominator. We don't really have a criteria. So if people are willing to run your node that can follow the tip of the chain. You basically have a good enough client in many instances. That's an oversimplification. But in the case of things like theium js, and Ref, I think that's absolutely the case. And they continue to push their performance, profile and functionality profile 127 00:26:02.270 --> 00:26:03.460 Matt Nelson: over time. 128 00:26:03.640 --> 00:26:15.049 Matt Nelson: But both of them went through the Fork Transition, and if users want to run that client they can, and if they want to validate with that client they can. I think that a lot of the kind of voice on the core development 129 00:26:15.060 --> 00:26:20.580 Matt Nelson: space comes from expertise, but also representation right? I think the Get team 130 00:26:20.900 --> 00:26:38.890 Matt Nelson: has a big portion of voice on core devs because they represent on the execution layer at least a hugely outsized proportion of users that they want to protect and make sure, have the best functionality available to them, running the client, etc. So that's my take, Justin. Maybe you have a different or more nuanced one there. 131 00:26:39.300 --> 00:26:58.009 Justin Florentine (Besu): II think, might take us fairly similar to that. To add a little nuance, I would say that if you're producing a new client. The good news is that there is a ton of stuff that you can do up to, and prior to actually proposing new blocks on the chain. 132 00:26:58.050 --> 00:27:06.709 Justin Florentine (Besu): and you can do that stuff on all the networks and all the way up to May that and you can measure your performance, and you can measure it against all the other clients 133 00:27:07.040 --> 00:27:09.380 Justin Florentine (Besu): to go further than I think 134 00:27:09.630 --> 00:27:37.869 Justin Florentine (Besu): you know. If I was, if I was endorsing a new client I would wanna see all of that. I would wanna see all of them participating the way Matt suggested, and keeping it up with it. And then I would wanna see all of the lower level testing things that I spoke about before executed and including proposals and including and creating new things for the blockchain. So that includes results from Edm tests running all the state tests and making sure that it was super. You know. 135 00:27:38.540 --> 00:27:52.639 Justin Florentine (Besu): it's making sure that it had been exposed to all the weird edge cases and stuff that we bake into those tests as well as checking its performance to making it to to make sure that you know some of the attacks that we're worried about that would occupy gas 136 00:27:52.940 --> 00:28:00.010 Justin Florentine (Besu): are analogous to all the other clients right like you don't want to find a gas mispricing issue that's specific to one client. 137 00:28:00.110 --> 00:28:06.779 Justin Florentine (Besu): And so there's there's a lot of tests that you would want to do. And then, you know, you've got the proposal stuff, too. 138 00:28:07.070 --> 00:28:15.020 Justin Florentine (Besu): so how do you? How do you create blocks? How long does it take you to create a block all all that stuff. So I think that 139 00:28:15.510 --> 00:28:27.719 Justin Florentine (Besu): it would be. I don't think it would be that difficult to say like, Oh, this client, yeah, this is a legit client like we have all the processes in place we have where we're used to running hive tests against everything. The results are quite comparable. 140 00:28:27.720 --> 00:28:51.289 Justin Florentine (Besu): I know that there's been a discussion ongoing recently about performance. And how do we compare it? And I don't think that's gonna be too hard. We're gonna come to a consensus pretty quickly on that, it'll be documented and reproducible across clients. The hive tests are really good test bed for that. And so once. That's once that exists. New clients. You just gotta you will draw bar somewhere. Says you must be this high to ride on main net and 141 00:28:51.560 --> 00:29:13.619 Justin Florentine (Besu): there's nothing we can do about it. If they don't, you know. We'll we'll there'll be a social consensus. Maybe some social prefer like, hey? Don't run that because it actually, when your turn comes up to propose, might not get a legit block, etc. So that's just common evaluating open source software and waiting for that to shake out through social consensus. 142 00:29:14.920 --> 00:29:24.849 Matt Nelson: There's also the gas limit to consider in this sense that and other clients can shift the gas limit, but they cannot outright change it. So if you're 143 00:29:25.320 --> 00:29:27.469 Matt Nelson: if your client cannot 144 00:29:27.490 --> 00:29:34.789 Matt Nelson: execute roughly 30 million gas and 12 s, you probably aren't going to be able to keep up with the chain. 145 00:29:34.990 --> 00:29:47.210 Matt Nelson: That's why I think it's kind of fun when you look at the python specs to like run those like State tests against the execution layer specifications, which is just like python pseudocode. And you can see, like, you know. 146 00:29:47.380 --> 00:30:01.539 Matt Nelson: the base suit and Geth numbers in the number of milliseconds. And then you have, like, 15 s for some of the python code. So I think that it's kind of like, yeah to Justin's point, what what works, as far as social consensus is concerned, what works in terms of the specs, and how 147 00:30:01.750 --> 00:30:07.930 Matt Nelson: performant is your client to follow the chain, and the rest is kind of social stuff. 148 00:30:08.790 --> 00:30:14.250 Matt Nelson: That was a great question. again, please feel free to post more in the chat. 149 00:30:15.240 --> 00:30:24.919 Matt Nelson: okay, let me take a look back at my question list here. 1 s. 150 00:30:32.100 --> 00:30:34.250 Matt Nelson: Okay, here's a good one, I think. 151 00:30:35.080 --> 00:30:46.619 Matt Nelson: So you've mentioned kind of late breaking spec changes how, as as kind of developers with these late breaking spec changes. 152 00:30:48.250 --> 00:30:56.559 Matt Nelson: how are we kind of squaring up the modifications that need to be done at the implementation level versus 153 00:30:56.720 --> 00:31:00.599 Matt Nelson: the specification. If if you're following my question. 154 00:31:02.590 --> 00:31:08.070 Matt Nelson: is it all the clients think this is this is silly, let's change it, or 155 00:31:08.240 --> 00:31:18.639 Hmm, because I think that a lot of the problem that ethereum had before the merge was that the specification was primarily driven by developments in the Geth client in general 156 00:31:19.140 --> 00:31:21.529 but now we have this different landscape. 157 00:31:22.040 --> 00:31:27.420 Matt Nelson: How does that kind of how does that new blurred line make sense to to a core? Dev! 158 00:31:29.120 --> 00:31:42.350 Justin Florentine (Besu): That's one of those questions that's hard to answer. Unless you've got your elbows in the code like, if you're super deep in the code, you can maybe in like 5, 10 min of discussion with your team. You can say like 159 00:31:42.370 --> 00:31:46.010 Justin Florentine (Besu): that would really be hard for us to implement, and 160 00:31:46.020 --> 00:31:54.399 Justin Florentine (Besu): it would be by hard. That generally means we would have to refactor a bunch of stuff to actually implement that change in the spec. 161 00:31:54.650 --> 00:32:01.450 Justin Florentine (Besu): And here's the additional testing and performance impacts. We would want to be clear about 162 00:32:01.650 --> 00:32:06.989 Justin Florentine (Besu): before before submitting that back and merging that in and changing the way that that works. 163 00:32:07.910 --> 00:32:11.060 Justin Florentine (Besu): Does that answer your question, Matt. 164 00:32:12.110 --> 00:32:30.599 Matt Nelson: Yeah, it does. I think that. Yeah, the in my mind, it's like we have a spec that is intended to be somewhat inflexible. like, where and how does our the teams choose to kind of bend or push on that spec versus bend or push on the client? Because I think you're right like 165 00:32:31.400 --> 00:32:49.190 Matt Nelson: us. Let's think of it this way, right? Us saying we need to refactor a bunch of code in order to meet a spec that is actually well designed versus the opposite, you know. We don't wanna say that that's not a fun thing to say. No, of course not, of course not. 166 00:32:49.420 --> 00:32:55.209 Justin Florentine (Besu): Cause. Cause if you don't, if if that's not really a concern, or the answer is almost always sure, go ahead, change it. 167 00:32:55.420 --> 00:33:06.590 Justin Florentine (Besu): It's like there's a there's a a question being kicked around about renaming some fields for block gas use or data gas user. I forget what I forget what it was. 168 00:33:06.860 --> 00:33:15.930 Justin Florentine (Besu): it's it's kind of just a you know, naming things is important, and it's hard. But once that decision is made, it's not gonna be that difficult for anybody anything to get changed, you know. 169 00:33:16.660 --> 00:33:18.420 Matt Nelson: Yeah. Got you there? 170 00:33:19.130 --> 00:33:38.089 Matt Nelson: This is more, a little bit of a more fun question for you. Where do you see, the community kind of coming into play and coming into this testing and implementation process. Because I think we, we have a lot of the initial stuff is done in the core devs kind of sphere a little bit 171 00:33:38.520 --> 00:33:59.579 Matt Nelson: you know. Hush! As we work things out. But then we slowly start to layer in more and more groups, like institutional stakers, communities like East Staker, and even regular users who are like, I'm experiencing some weird client bugs on girly. You know what kind of fun or cooperative or kind of uplifting 172 00:33:59.720 --> 00:34:10.080 Matt Nelson: interactions did you see there? And where do you see? Also incentives to have community members get involved in that process just as individuals or small groups. 173 00:34:11.510 --> 00:34:22.420 Justin Florentine (Besu): I wish I had a better answer for that. But the the real incentive is social. if you're if you're nice enough to be running a test set client. 174 00:34:22.440 --> 00:34:25.179 Justin Florentine (Besu): Then 175 00:34:25.210 --> 00:34:49.049 Justin Florentine (Besu): you got some interesting people to talk about it with on on the various discords, etc. You're gonna be much further down the stack. You're gonna be, probably in more contact with core developers things like that. It's really really awesome. When our users choose to do that, they have no financial incentives to do that. They're using compute resources to S to sync up with. 176 00:34:49.080 --> 00:35:06.339 Justin Florentine (Besu): you know, girly, which is a very large chain. You know, moving forward. That's gonna get sunset. No, it'll still be. Blobs will be there. They're they're still gonna have the same requirements. Hopefully, that main that's gonna have. The state management will be a lot, you know, a lot smaller. But they're still gonna have to, you know. 177 00:35:06.580 --> 00:35:17.410 Justin Florentine (Besu): They're gonna have to consider their their, their hardware resources. And that's a nice donation. for them to be giving back to the community. And 178 00:35:18.120 --> 00:35:45.129 Justin Florentine (Besu): it's it's it's super super helpful. Right? Because we get we get into these. We get. We get blinders on sometimes, and it's hard to. It's hard to avoid that. And it's very easy for us to automate all of our testing and deployment of our nodes, etc., etc., and they all have the exact same spec out machine, because they're all running in azure aws or whatever. And you don't really have the hey? I tried to run girly on my old laptop the other day. 179 00:35:45.400 --> 00:35:57.809 Justin Florentine (Besu): And this is some weird stuff that happened like, Oh, well, that's cool. Cause that would actually happen on main net. And that's actually kind of hard for us to to to simulate and reproduce. So it's so it's like super helpful. 180 00:35:58.430 --> 00:36:12.190 Justin Florentine (Besu): I would love to see like a stupid girlie tricks award, or a Po app, or something like that, where, like somebody just tried to do something weird and silly on on our test nuts. And you know, build a little bit of a community about around that. 181 00:36:12.430 --> 00:36:23.550 Justin Florentine (Besu): You know, people try to run doom on refrigerators. Why aren't we trying to run, you know, ethereum on on weird stuff? I think that would be that would be fun. That would be that would be kind of 182 00:36:24.220 --> 00:36:29.170 Justin Florentine (Besu): I don't know. I think it would be fun. I think our community would think it would be fun, but that's and that's really what would matter. 183 00:36:29.500 --> 00:36:33.610 Matt Nelson: Zoom in the ebm. Sure. 184 00:36:34.100 --> 00:36:42.690 Justin Florentine (Besu): so yeah, if you're if you're running, if you're already going through the trouble to to to solo, stake and run things from home. Throw another network on there. 185 00:36:44.810 --> 00:36:59.439 Matt Nelson: Awesome, awesome. Alright. We got another question in the chat here from Tj. This one's a fun one, because I think it's actually very revealing. Declined teams start working on implementing future eips, even if they're not necessarily going to be included in upcoming upgrades. 186 00:36:59.590 --> 00:37:05.740 Matt Nelson: or even sorry parentheses, or even eips that do not have popular support from other teams. 187 00:37:07.300 --> 00:37:23.959 Matt Nelson: As an example, euf could be shipped in Prague, Electra, and the Ap. Specs are mostly written will. Most of the clients have done preliminary work to implement eips such as those needed for ef in advance for do client teams have code written or possible future apips that aren't up for inclusion. 188 00:37:26.230 --> 00:37:28.320 Justin Florentine (Besu): Yeah. Yep. 189 00:37:28.460 --> 00:37:32.080 Justin Florentine (Besu): Yup, eip is very often 190 00:37:32.100 --> 00:37:38.740 Justin Florentine (Besu): coexist alongside implementations that are that are in flight, and that's a good thing, because it kind of reinforms the design. 191 00:37:39.140 --> 00:37:52.980 Justin Florentine (Besu): you know, there there will. Eips are notoriously hard to finalize. In fact, there's you could probably go through the eip database and find tons of them that are, you know, marked still in draft, or whatever that have already been implemented and shipped like 192 00:37:53.410 --> 00:38:02.649 Justin Florentine (Besu): they're probably just marked incorrectly, but I think the spirit of that is not too inaccurate. Like, I said, these things keep changing over over time. 193 00:38:02.960 --> 00:38:15.280 Justin Florentine (Besu): so yeah, so yeah, we we will campaign for something. I mean, that's a good way to put pressure on eip, you think is good for the good, for the network is to implement it. I mean being able to say. 194 00:38:15.660 --> 00:38:25.860 Justin Florentine (Besu): You know, basic team thinks we should ship eof. By the way, we've already got it implemented is a it's a great way to to to to get what you want out there. 195 00:38:26.520 --> 00:38:32.439 Justin Florentine (Besu): so so yeah, that that definitely happens, I think what's really important, though, is that 196 00:38:33.100 --> 00:38:37.519 Justin Florentine (Besu): prototyping an eip as it's being developed 197 00:38:37.620 --> 00:38:46.610 Justin Florentine (Besu): gives clients first, mover advantages right? So if everybody that's designing eips prototypes them all in Geth. 198 00:38:46.890 --> 00:38:51.369 Justin Florentine (Besu): Geth will always be able to say we already have an implementation for this, we should ship it. 199 00:38:51.810 --> 00:39:13.519 Justin Florentine (Besu): and that's a really subtle way to encourage client diversity is to not just run other clients, but prototype on other clients. Give that first mover advantage to other clients so that they can say, Yeah, we've got this implemented already. Have other clients coming to that team to say like, how did you implement it? What tips do you have. 200 00:39:13.520 --> 00:39:22.640 Justin Florentine (Besu): you know? Did did implementing Verko go better for you because you already have a bonsai implementation in your your data store. 201 00:39:23.060 --> 00:39:25.760 Justin Florentine (Besu): So yeah, it's 202 00:39:26.240 --> 00:39:31.989 Justin Florentine (Besu): it's a, it's a, it's a bit of a weird flywheel feedback loop that happens there. 203 00:39:38.510 --> 00:39:43.150 Matt Nelson: Awesome. Great question. Yeah, I'd have to add as well that I think 204 00:39:44.280 --> 00:39:51.329 Matt Nelson: it's not necessarily an addition. But the punctuation is that like clients will work on pick these Ips that they're passionate about, or that their users are. 205 00:39:51.400 --> 00:40:08.879 Matt Nelson: you know, products are passionate about and like they will just go ahead and build stuff in order to push things through. We've discussed this many times on this this series is that in order to champion stuff, you need a little more than just a good idea. So client teams will absolutely work with external contributors and other groups to 206 00:40:09.390 --> 00:40:15.190 Matt Nelson: oftentimes provide advice so that others can implement the Ips that they're working on. 207 00:40:15.370 --> 00:40:16.390 Matt Nelson: And 208 00:40:16.810 --> 00:40:33.280 Matt Nelson: a lot of the times these reference implementations end up in Geth. But sometimes you've have users proposing Ips that have expertise in Java, or, you know, Dotnet, or go, or different languages. And so we end up with a variety of reference implementations. But those are definitely super interesting here. 209 00:40:33.780 --> 00:40:39.220 Matt Nelson: Any other questions in the chat? I 210 00:40:39.490 --> 00:40:48.420 Matt Nelson: mostly it will just keep going down my list here. We also have about 20 min left. It could probably duck out a little bit early. 211 00:40:49.670 --> 00:40:53.990 Matt Nelson: give another chance to the peanut gallery. 212 00:41:01.920 --> 00:41:07.220 Matt Nelson: Okay, here's a good question, Justin. What has been the most challenging. 213 00:41:07.430 --> 00:41:16.639 Matt Nelson: frustrating, or rewarding part of your participation in this testing experience. So you can either go good good news, bad news, or 214 00:41:16.850 --> 00:41:17.820 Matt Nelson: we're capable. 215 00:41:22.110 --> 00:41:26.179 Justin Florentine (Besu): Well, testing in particular. 216 00:41:26.320 --> 00:41:30.250 Justin Florentine (Besu): you know, it's weird cause I don't really consider myself a tester like 217 00:41:30.390 --> 00:41:37.249 Justin Florentine (Besu): I consider myself sort of like the customer for the tests. I think that 218 00:41:37.390 --> 00:41:47.899 Justin Florentine (Besu): you know, for as far as like a really rewarding technical experience was seeing, seeing the utility of the hive tests improve and the quality of them improve over time. 219 00:41:48.530 --> 00:42:11.780 Justin Florentine (Besu): There is quickly becoming kind of the de facto way to send somebody a bug report. I think I could see like in a year from now there'll be very, very easy ways to do that, and to say, like, Hey, I think this is an issue in in the bug. Okay, where's the hive test right? And you can run it. You can isolate your clients. It spins up kind of a small little docker network. 220 00:42:11.890 --> 00:42:22.760 Justin Florentine (Besu): It's it's it's a really good piece of infrastructure. I think that has a lot of legs, and it's so easy to automate your testing and so many. so many different ways, but also so many different layers of precision. 221 00:42:23.220 --> 00:42:40.870 Justin Florentine (Besu): And and and that's really the the general purpose. One. II think the hive tests really are the right balance between granularity, repeatability, modularity, being able to kind of point it easily at different clients and different networks, and how they're set up. 222 00:42:40.870 --> 00:43:09.319 Justin Florentine (Besu): and that's not to belittle or say like, Oh, the state tests are not less important. There's those are actually just as important because they do the opposite right. They isolate just the evm, and it's very easy to get a very specific trace back out of it when you run into an issue. And it's very easy to inject very specific pieces of evm coder transactions, or whatever, and point them in a specific piece of state. There's there's other types of testing that have a very different granularity that are equally useful for totally different reasons. 223 00:43:09.320 --> 00:43:17.950 Justin Florentine (Besu): So that I think the diversity of scope of the testing techniques that we have is really really cool to see and really cool, to see how 224 00:43:18.560 --> 00:43:25.899 Justin Florentine (Besu): how, is just making all the developers better. I know. I mean, I should just speak for myself, like I think it's making me better as a developer because 225 00:43:25.900 --> 00:43:49.729 Justin Florentine (Besu): I get frustrated personally when I have to load too big of a model into my head, and I'm always looking for ways to narrow it down. Narrow down, narrow it down just because you know the way that I think I don't have a great memory. I don't have as much experiences of a lot of other people, and it takes time for me to build up those sense memories and intuitions about where to find code or where a problem could be, etc. So, having these different granularities of tests. 226 00:43:49.730 --> 00:44:05.370 Justin Florentine (Besu): speed me up as a developer, and I think speed us up as a team. So I think that's that's been really, really cool to see. I think that the the the shadow forking. I still don't I? Still, sometimes I think I'm like, Oh, man, that's such a good idea like when we started doing that for for main net. I was like. 227 00:44:05.440 --> 00:44:26.370 Justin Florentine (Besu): man. That's that's really a great, great test. Cause it's it's like doing it live. It's like just running it test. It's it's testing in production. And so that's that's really cool. I was really excited when we when we did that it gave me a ton more confidence that the merge was gonna go. Well, and I'm glad to see that it's gotten baked into every hard forks since. And it's just standard operating procedure for us now. 228 00:44:27.260 --> 00:44:35.270 Matt Nelson: So yeah, yeah. I think that the even since the for even since I started working on network upgrades with the merge 229 00:44:35.710 --> 00:44:42.820 Matt Nelson: Shanghai and then cancun have increasingly become more automated and more procedural, which I think is 230 00:44:44.050 --> 00:44:59.020 Matt Nelson: like nice to see for a decentralized community, because there's so many procedures and things like you said, that have to be redundant. That could be missed. That could be gaps in testing that, I think. As we kind of get used to working with 10 client teams and more. And all this other stuff. 231 00:44:59.350 --> 00:45:07.050 Matt Nelson: That it's it's very heartening to see the progress. Okay, let's see, here 232 00:45:08.260 --> 00:45:09.380 Matt Nelson: we have 233 00:45:09.840 --> 00:45:15.570 Matt Nelson: again a little over 15 min left. We don't need to use all that time. I mostly have more 234 00:45:15.960 --> 00:45:21.390 Matt Nelson: basic questions, if we have any, from the group here for Justin. 235 00:45:28.190 --> 00:45:40.209 Matt Nelson: alright, you gave us a good, a good, a good one there, like a a positive note. What about what do you find the most difficult or frustrating part of 236 00:45:40.250 --> 00:45:43.540 Matt Nelson: working on these network upgrades as a developer? 237 00:45:45.660 --> 00:45:58.190 Justin Florentine (Besu): I mean, there's always that fear right? The stakes are the stakes are just really high. I don't know that that's specific to the network upgrades or how they're gonna work. It's just working on like a 238 00:45:58.200 --> 00:46:05.930 Justin Florentine (Besu): you know. What are we at? Like a 200, some 1 billion dollar network? I don't even know where we're at as far as Market Cap these days. 239 00:46:05.980 --> 00:46:08.540 Justin Florentine (Besu): you know. So that kind of sucks. 240 00:46:08.930 --> 00:46:10.110 Justin Florentine (Besu): I think. 241 00:46:13.000 --> 00:46:15.200 Justin Florentine (Besu): Gosh! I don't know 242 00:46:16.020 --> 00:46:18.090 Justin Florentine (Besu): what what could be better. 243 00:46:19.580 --> 00:46:30.849 Justin Florentine (Besu): I you know that. So so the the diversity of testing is, you know, probably cuts both ways right? It's like, well, where? Where you think you find an issue. Okay, can you send me a test cool. In what format? 244 00:46:31.020 --> 00:46:39.829 Justin Florentine (Besu): Okay? Well, where do I find the tools to do that? I don't know. So now I need to like walk you through. I got a report the other day that was 245 00:46:40.330 --> 00:47:07.450 Justin Florentine (Besu): it was. I forget exactly the test format, but it was a it was a I think it was a more guest specific test format, and it hadn't been translated into something that bases evm tool could execute. And that was, that's kind of a that's friction, right? That's friction in your testing process, friction in your development process where you need to be like. Oh, now I need to go learn this other tool. So I can convert its output into something that I can run through my tool before I can even start doing my job and analyzing. 246 00:47:07.450 --> 00:47:23.489 Justin Florentine (Besu): What is the problem that you found analyzing the traces? Comparing the traces like comparing traces between Cvm's is still not something that we've really standardized. And II think that's a problem. I think that's a huge area for improvement, because the point of traces is to compare them like 247 00:47:23.890 --> 00:47:46.239 Justin Florentine (Besu): it's it's you. Can. You can. The trace will tell you a lot about your evm in isolation, and that's fine. But so did the code that you were looking at. It's even more useful to be able to say like very quickly, I don't have to read and understand Guest code. I can pull that trace. And literally I should be able to do a diff on it like I should be able to compare them with a tool that says like, Hey, here's where they diverge, and how they diverge, and why? 248 00:47:46.290 --> 00:47:47.640 Justin Florentine (Besu): So 249 00:47:48.560 --> 00:47:58.100 Justin Florentine (Besu): getting that next step? To really make that process better, I think, would be, would be a big improvement. That's not. That's not fun to do when you're when you're actually trying to chase down an Evm bug. 250 00:48:00.260 --> 00:48:10.199 Matt Nelson: And we all know those notorious knowing you. That is as bad as it probably will be on this call. So that's fine. 251 00:48:10.840 --> 00:48:19.710 Matt Nelson: okay, folks, I'll give one more chance for any last questions. If not, I think we'll give everybody an additional 10 min back to their day. 252 00:48:30.510 --> 00:48:33.990 Matt Nelson: Anything coming in chat? We'll give you just a second. 253 00:48:45.710 --> 00:49:02.920 Matt Nelson: Okay, it looks like that, is it? Thank you again, Justin, for coming and bringing your perspective. It's super valuable. This course, I'm hoping will remain relevant for tons of time to come, because we will continue to upgrade Aetherium, and we will continue to have to test those upgrades. 254 00:49:03.400 --> 00:49:05.589 Matt Nelson: So thank you again, Justin, for your time. 255 00:49:05.760 --> 00:49:07.460 Justin Florentine (Besu): Happy to be here. It was a blast. 256 00:49:07.850 --> 00:49:15.429 Matt Nelson: Alright! Some housekeeping folks, I believe. Our next lecture is the 257 00:49:17.470 --> 00:49:26.949 Matt Nelson: February, January 30 first, so we have nothing this week, and then we will return on the Wednesday, the 30 first at 90'clock, Pacific time, or 12 Eastern. 258 00:49:27.040 --> 00:49:37.369 Matt Nelson: and we will have some guest speakers the following week. There. This will be our last lecture. So make sure to mark your calendars. We'll be discussing 259 00:49:38.270 --> 00:49:51.529 Matt Nelson: maybe some more technical details on how to kind of get literally involved in testing on fork to networks so we might even take some look at early code or girly clients. I might have to sync up some early client for 260 00:49:51.570 --> 00:49:52.770 Matt Nelson: the lecture 261 00:49:53.060 --> 00:50:05.599 Matt Nelson: either way. Thank you again for your time today. And if you have any additional questions that pop in your mind that you want answered later on. I know where to find Justin. So please feel free to shoot me an email, and we can get an answer. 262 00:50:05.910 --> 00:50:07.279 Matt Nelson: But thanks again. Everybody 263 00:50:09.160 --> 00:50:10.000 Justin Florentine (Besu): see folks. Supplemental Resources Testing Parameters that should exist for a testnet or devnetProto-danksharding: What It Is and How It Works Proceeds from collectibles editions of this post will be distributed as follows: 100% to Education DAO Optimism Treasury (0xbB2aC298f247e1FB5Fc3c4Fa40A92462a7FAAB85)

## Publication Information

- [Wiki of Web3](https://paragraph.com/@education-dao/): Publication homepage
- [All Posts](https://paragraph.com/@education-dao/): More posts from this publication
- [RSS Feed](https://api.paragraph.com/blogs/rss/@education-dao): Subscribe to updates
- [Twitter](https://twitter.com/education__dao): Follow on Twitter

## Optional

- [Collect as NFT](https://paragraph.com/@education-dao/dencun-5): Support the author by collecting this post
- [View Collectors](https://paragraph.com/@education-dao/dencun-5/collectors): See who has collected this post