<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <channel>
        <title>Rolfy.eth</title>
        <link>https://paragraph.com/@rolfy-eth</link>
        <description>Protocol engineer on the Teku consensus-layer client.</description>
        <lastBuildDate>Fri, 20 Nov 2026 21:37:12 GMT</lastBuildDate>
        <docs>https://validator.w3.org/feed/docs/rss2.html</docs>
        <generator>https://github.com/jpmonette/feed</generator>
        <language>en</language>
        <copyright>All rights reserved</copyright>
        <item>
            <title><![CDATA[Graffiti Watermarks]]></title>
            <link>https://paragraph.com/@rolfy-eth/graffiti-watermarks</link>
            <guid>vMqhroQBN7xvPiT64w6I</guid>
            <pubDate>Wed, 01 May 2024 03:00:54 GMT</pubDate>
            <description><![CDATA[Title Image by MMT from Pixabay For a while now, we’ve been looking at client diversity, and sometimes beating the drum pretty hard admittedly, to try to make sure everyone understands what it is and why we care… Sites like clientdiversity.org have done a good job of socialising some of the data, and a lot of people are sitting up and taking notice, which is awesome. The gap, as always with these kinds of things, is a lot of data is ‘opt in’ - where people go out of their way to volunteer dat...]]></description>
            <content:encoded><![CDATA[<p>Title Image by <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://pixabay.com/users/mmt-649797/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=569265">MMT</a> from <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://pixabay.com//?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=569265">Pixabay</a></p><p>For a while now, we’ve been looking at client diversity, and sometimes beating the drum pretty hard admittedly, to try to make sure everyone understands what it is and why we care…</p><p>Sites like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://clientdiversity.org/">clientdiversity.org</a> have done a good job of socialising some of the data, and a lot of people are sitting up and taking notice, which is awesome.</p><p>The gap, as always with these kinds of things, is a lot of data is ‘opt in’ - where people go out of their way to volunteer data in order for us to get the level of data that we have today, and still there’s a large gap - over 30% of execution layer clients we don’t actually know about.</p><p>With the consensus layer, several tools like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://blockprint.sigp.io/">blockPrint</a> from sigp have helped gather data, and as it turns out that seems to be fairly reliable. The remaining gap is on the execution layer side where is also currently (in theory) the largest disparity in diversity, with geth occupying an estimated 63% (very very rough) of all execution layer clients.</p><p>Recently, there was a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/ethereum/execution-apis/pull/517">proposal</a> suggested to try to surface some of this information. Basically, the idea is that clients can use some spare graffiti space to pack information into without the need for a hard fork, as the space is already available.</p><p>Obviously we don’t want to stop people being able to set graffiti, but at the same time the feeling is that this would be super useful information to have, if people have space available for us to use.</p><p>In teku land, we’ve taken the position that this data should be gathered in an ‘opt-out’ fashion, because we really do believe it’s important information, so as long as there’s space available in proposal graffiti, we’re going to try to pack some information.</p><p>If a user sets no graffiti, this might then appear something like `TKcdcb1773GE87246f3c`, and admittedly this is not super pretty, so lets unpack it a little…</p><p>Basically this string has 2 sections, “TKcdcb1773” and “GE87246f3c”. The first section starts with TK, so that indicates teku was the consensus-layer client, and the data after TK is the git hash of the client. Similarly, the second part starts with “GE”, which indicates geth, and the rest is the version of geth in use. Each client will use their own 2 character code.</p><p>If a user specifies a string in graffiti, and we have space left for the full identifier, we’ll go ahead and add it. If we do run out of room, we’ll start reducing the length of the git hash to the point where we may end up with just the client types (TKGE).</p><p>Some people may prefer that version information isn’t surfaced for security reasons etc, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.teku.consensys.io/reference/cli#validators-graffiti-client-append-format">so it’s possible to specify in teku</a> that you just want the client strings without version information, and if people could use that at a minimum that’d be very very helpful in gathering client diversity information. If people do feel super strongly about it, it’s possible to disable this reporting completely. Please be aware the data is just for diversity information, so it’d be very helpful if it is possible to leave at least the client types being reported when blocks are produced.</p><p>If you fill more than 28 bytes of graffiti, that’s totally fine, we’ll not overwrite your graffiti to gather this data, we can appreciate that it’s ultimately yours to do with as you will!</p><p>Very similar to gathered data currently, this is purely an effort to understand network diversity, and it’s ultimately possible that people want to falsely report for whatever reason, but if nodes can report honestly it would allow us to get a much clearer overall picture of the network health in terms of diversity, and maybe we can reduce that 30% or more that are un-reported…</p><p>Because this is still in the process of being implemented by various teams, it’s possible only the CL might be reporting, but over the coming weeks it’s likely this data will become a lot more common in graffiti. It’s my hope that explorers may choose to display the information in some way and filter it from graffiti, but I’m not sure if there’s plans around that, or if it’s just something I think would be a good idea :)</p><p>Hopefully this helps shed some light on the mystery of this new data appearing in your proposals!</p>]]></content:encoded>
            <author>rolfy-eth@newsletter.paragraph.com (Rolfy.eth)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/5ffd79f998c009cab606bcfde28cdca617655d89696e00af958111e292021ce8.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Synchronised functions vs Locks]]></title>
            <link>https://paragraph.com/@rolfy-eth/synchronised-functions-vs-locks</link>
            <guid>fDNBMvse8Psp3RebVkN6</guid>
            <pubDate>Thu, 21 Dec 2023 23:32:47 GMT</pubDate>
            <description><![CDATA[Cover photo by Life Of Pix: https://www.pexels.com/photo/brass-colored-metal-padlock-with-chain-4291/ The year is winding down, so why not take a moment to revisit a bugbear. We have this really simple concept in Teku around slashing protection, where we’ve ensured someone can’t be slashed with quite a simple set of functionality and data. It all comes down to 1 validator, 1 file, 3 numbers. This works really really well. It’s backed by code that uses a synchronized function - the implication...]]></description>
            <content:encoded><![CDATA[<p>Cover photo by Life Of Pix: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.pexels.com/photo/brass-colored-metal-padlock-with-chain-4291/">https://www.pexels.com/photo/brass-colored-metal-padlock-with-chain-4291/</a></p><p>The year is winding down, so why not take a moment to revisit a bugbear.</p><p>We have this really simple concept in Teku around <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.teku.consensys.io/concepts/slashing-protection">slashing protection</a>, where we’ve ensured someone can’t be slashed with quite a simple set of functionality and data. It all comes down to 1 validator, 1 file, 3 numbers.</p><p>This works really really well. It’s backed by code that uses a <code>synchronized</code> function - the implication here is we can only run one instance of this function at any one time. It’s a good decision in a lot of ways and scales fairly nicely.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/b8d3c2bc227fc5d9344254ee5d054262e08612eb9035359bbde11101325bf608.png" alt="32 validators, minimal network  - profiler. This looks decent pretty good at this point" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">32 validators, minimal network - profiler. This looks decent pretty good at this point</figcaption></figure><p>I noticed an issue though, back when I was doing performance testing. If we scale up enough this synchronised function does actually start to hurt our timings. Visually, the profiler shows it taking up a large amount of a code path, and practically some of our metrics show increased delays in attestation production times.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/8599de5366a14d237a549f145ab681e013b325ffe4af809b3837d4a4a6218869.png" alt="4096 validators, minimal network  - profiler. Time is ramping up a lot!" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">4096 validators, minimal network - profiler. Time is ramping up a lot!</figcaption></figure><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/2167d9f6eb271be6e391ef30772543ae6f509a42ec9cc01504d584b2e77c2312.png" alt="16k validators, minimal network  - profiler" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">16k validators, minimal network - profiler</figcaption></figure><p>On holesky during some testing, I was running with 20k validators, admittedly there’s more slots so the signing load does get spread, but this becomes an actual issue as we continue to scale up, as attestation production delays in some cases reduce our overall effectiveness.</p><p>To look at why, you could look at our slashing protection files as a database. It’s a stretch, but in real terms we have a ‘table’ of records of slashing protection data, and effectively there’d be 1 row per validator key.</p><p>The synchronized functions mean any 1 of those validator records being updated is effectively like locking the entire table. This is a fairly common kind of concept (table level locking), so I won’t go deeply into it, but the synchronised function is fairly analogous. With 20k validators, on average we’re looking at about 625 attestations per slot, and all of these checks need to happen one at a time.</p><p>To look at a solution, we can look to some common principles of databases. As Table level locking is effectively the problem, how do relational databases often solve this issue? Row level locking…</p><p>That would be a fairly fundamental change, and with fundamental change is a level of risk.</p><p>This would mean maintaining locks for each validator, so that we’re only restricting access at a validator level, not at the entire system level, you can see how that may improve things… The advantage would be that these 625 attestations could be verified ok over several threads concurrently, as long as they’re not for the same validator (which they shouldn’t be!). It’s unlikely that we’d want that many concurrent threads but we’d be able to make better use of whatever we had available.</p><p>I did a fairly tiny <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/Consensys/teku/pull/7819">PR</a> to attempt to implement that concept. We often use feature flags for things like this, it’s a way of ensuring we don’t do something bad and push that to master, so that’s basically the pattern I followed, but the critical changes were around using <code>ConcurrentHashMap</code>, and a new <code>validatorLocks</code> member that allows a lock per validator (row level locking effectively).</p><p>Lets see what happens with that same example above with 16k validators…</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/c4b15ecb7d2cdb00d95cc44fab4cf58893fdcaef59faf92c197a392ef461bf77.png" alt="16k validators, minimal network, row locks - profiler" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">16k validators, minimal network, row locks - profiler</figcaption></figure><p>The time has gone up a fair amount, so thats interesting - both sample times were roughly 6 minutes, and this one is more than double the previous time as an overall measurement. Interestingly (not shown) the call stack changes drastically, but theres no real easy way to demonstrate that that’s readable. This is likely explained by the signing not being a choke point, it’s now likely that a lot more functions find their way into this kind of pattern.</p><p>Lets look at Grafana output (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://grafana.com/grafana/dashboards/16737-teku-detailed/">Teku Detailed</a>) on a node on holesky running 20k validators:</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/c117d0121613dafe0cad3b7ad2593c3a36c60604ee131e25fbda50ec461fd702.png" alt="Feature switched on in the middle roughly" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Feature switched on in the middle roughly</figcaption></figure><p>When the feature flag was toggled, all of a sudden the vast majority of attestation delays drop to the 0-1ms (delay) bucket… That’s interesting! Also in the above graph you can roughly see our delays are significant sometimes before we enable this feature, which is why this area initially drew my eye. This is likely more of a ‘worst case’ scenario than likely in the wild, as 20k validators on 1 node is a lot.</p><p>Lets look at some other panels of the Grafana board:</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/a1915108989810375bd015ac4fcef45877f170595b52a1756bffbfd4ef0c44b6.png" alt="Same timescale, feature came on roughly in the middle" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Same timescale, feature came on roughly in the middle</figcaption></figure><p>Inclusion distance (max) improved a lot, as did the Attestation inclusion detail. This is significant!</p><p>I’m pretty happy with this improvement. There’s likely more to be done, and possibly better solutions to this problem, but its surprising how often we find a bottleneck because of changes of how we’re using something, and first principles lets us implement a relatively simple solution.</p><p>I still have questions regarding some specifics, but the idea definitely had merit, and now it’s just about time to soak (testing) and making sure theres no weird corner cases. I’m not sure we’ll have this on by default in the next release, but its certainly tempting.</p><p>The thing with bottlenecks, is now we get to find the next one :)</p>]]></content:encoded>
            <author>rolfy-eth@newsletter.paragraph.com (Rolfy.eth)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/2d747c6016f1b78c4bb8ddf8759259d1653a394f29ee9e877ee5cc970c40a724.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Caching and Memory]]></title>
            <link>https://paragraph.com/@rolfy-eth/caching-and-memory</link>
            <guid>54L1Qeo48tsihaGaL5Ey</guid>
            <pubDate>Mon, 24 Jul 2023 00:06:41 GMT</pubDate>
            <description><![CDATA[Teku has always made heavy use of caching data, which is one mechanism for making sure we have the data available that we need, without having to load it from disk. The upside of caching is that we get fast data access. Within reasonable bounds, it’s a great trade off, but there comes a time when things seem to grow unbound, and we need to re-assess what we’re storing.Beacon State CacheSince the May non-finality issue, we’ve been acutely aware of the heavy reliance we have on our state cache....]]></description>
            <content:encoded><![CDATA[<p>Teku has always made heavy use of caching data, which is one mechanism for making sure we have the data available that we need, without having to load it from disk.</p><p>The upside of caching is that we get fast data access. Within reasonable bounds, it’s a great trade off, but there comes a time when things seem to grow unbound, and we need to re-assess what we’re storing.</p><h2 id="h-beacon-state-cache" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Beacon State Cache</h2><p>Since the May non-finality issue, we’ve been acutely aware of the heavy reliance we have on our state cache.</p><p>This cache gave us fast access to all of the hot states. In fact, one of the answers we give most often on socials when people have issues with memory, is to look at the Objects in memory graph on the teku detailed grafana board, and it should look like a saw-tooth pattern.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/aff31b1d33beab65c17e1ecd1de5075b1bd94c673e5d49337ae92173ab788b5d.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><p>The number of objects being graphed in this are Beacon States available to load from our cache.</p><p>Our primary fix for the May non-finality was to not access this cache each time we need to validate attestation gossip from the network. Afterwards, we still had 1 access periodically to this large cache, and so it was an unknown, and we left it alone.</p><p>With the validator set size increasing at break-neck speed, we were going to have to revisit this decision, and in the last couple of weeks, we’ve done just that. It turns out that we access 3 states consistently, and somewhat unsurprisingly they’re the epoch transition states, and the head state.</p><h2 id="h-epoch-states-cache" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Epoch States cache</h2><p>A new cache got added for epoch transition states. This can be a fairly small cache with a little room to grow, in case of forks. The intent of this cache is to ensure we can access finalized and justified states, and if there’s any epoch transitions of forks ideally, without having to regenerate a state. Regenerations do tend to be expensive, and it’d be nice if they only occur very rarely…</p><p>Tracking epoch states is mostly complicated to know which state we’ll access for justified and finalized states, but it’s a problem we’ve solved elsewhere so the challenge was basically to ensure we add that same management to this new scenario.</p><p>The size of the cache will grow and shrink - periods of non-finality, forks etc may influence what gets tracked in this cache. The minimum number of items required will be 2 (finalized and justified), but the size of the cache as a limit can comfortably be more like 6 or so, giving room for when different network events happen.</p><p>The other state that we do track in this cache is the current epoch transition, which typically will become the justified epoch, so it just saves computing it for the majority case when everything happens as expected.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/97ed24fdfda8451c67c328899af772605f2a57a1efa15bbd87ac84d335f1c05a.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><p>Looking at the graph (Mainnet), we see that 3 is indeed the typical number, and over a few hours sometimes ours does grow to 4. On Goerli sometimes it goes higher, particularly in some instability over the last little while, so it’s worth having spare capacity in case it’s required in this cache.</p><h2 id="h-non-finalized-states-cache" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Non-finalized States Cache</h2><p>The states cache doesn’t really need to be large. It is somewhat dependent on how many forks are in flight as to the tuning of this cache, so that may need to be tweaked potentially if things are going poorly on a given network.</p><p>The main task for this cache is to ensure we can get the head state of any chain, to avoid having to replay to track the head of that chain.</p><p>The other thing the states cache is handy for, is if people are referencing a lot of non finalized state data as part of their operations. That really means this option may want to be a visible command line option, so that people can freely adjust it depending on their requirements.</p><p>This cache will basically always be full at lower numbers, but that’s ok. The cache self-manages what’s in it by getting rid of the oldest items, and if it’s very small that becomes a self managing limit. The reason for the old ‘saw tooth’ pattern, was because this cache got cleaned up by transaction updates, where we explicitly removed items to reduce the size of the cache before it hit its capacity limit, where as if the cache is very small, we may never end up needing to remove stale items from this cache manually.</p><h2 id="h-okay-but-memory" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Okay, but Memory…</h2><p>All of these cache changes resulted in a reduction of the amount of memory required, but it actually wasn’t huge. In situations like this we can sometimes look for an ‘easy win’.</p><p>In Java, it’s fairly easy to take a dump of the heap (that <code>Xmx</code> setting people are so familiar with is the heap space), and look over it with a memory analysis tool.</p><p>Interestingly, we had a lot of duplicated objects. Looking at the data, basically we had a lot of duplication of some numbers - <code>32_000_000_000</code> (32 ETH), <code>0</code>, <code>-1</code> (which is max uint64…). In all there was over 200MB of cache on Goerli referencing these numbers, and it’s actually a fairly easy task to cleanup.</p><h2 id="h-the-result" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">The result</h2><p>After making that little change, I pushed to some of our non-validating nodes, and the heap space came down quite a lot! Below is the graph of heap space usage of one of our nodes I’ve been testing on (that <code>Xmx</code> setting, we were on 5g max initially).</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/1f15f6ae05828556038b048a224eb78394f294db3d80d7c938230ec20aed40b7.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><p>The first ‘peaky’ bit of the graph is our old states based cache with no epoch states, basically the heap space when we’re employing that saw-tooth kind of pattern we saw above.</p><p>The change to using less states and the addition of the epoch-states (6 in states cache, and 6 in epoch states cache ) gave us the middle section, which already is a big improvement on part one.</p><p>The final section from the 23rd is after fixing the numeric duplication we had, and though it’s not a huge change, it’s definitely a worth while ‘easy win’.</p><p>The result was being able to comfortably run on a 3g heap (-Xmx3g) without the need for frequent garbage collection, down from 5g previously.</p><h1 id="h-wen-cache" class="text-4xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">WEN Cache?</h1><p>The next challenge will be rolling out these changes without causing too much chaos, and finding the next conversation to have when people are struggling with their memory, as this has fairly substantially changed that landscape. These changes aren’t small, but it’s something I’d like to get in general use relatively soon due to the impact.</p><p>The numeric ‘easy win’ change we’ve put in and will definitely be in the next release.</p><p>Regarding the cache sizings, I’m still toying with how to best roll this out. The one complication is for those that access non-finalized states a lot, as they may find they need to tweak the settings, and documenting that scenario.</p><p>The next release contains Java 17, so that’s a pretty huge change for some people that may be on older images.</p><p>This change would mean people are able to reduce some memory pain, so it may be worth including also, but the discussion is ongoing.</p><p>The command line arguments are present, so it’s possible that we just document how to make the changes, but it will definitely become default settings very soon.</p><h2 id="h-cover-image" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Cover Image</h2><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.freepik.com/free-photo/motherboard-circuit-technology-remixed-media_17122419.htm#query=memory%20computer&amp;position=36&amp;from_view=search&amp;track=ais">Image by rawpixel.com</a> on Freepik</p>]]></content:encoded>
            <author>rolfy-eth@newsletter.paragraph.com (Rolfy.eth)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/14ddb46f0da90bfa80d704dc786b0b0f46ffac398d276446cb4245b93c9e842c.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Attestation Gossip Processing]]></title>
            <link>https://paragraph.com/@rolfy-eth/attestation-gossip-processing</link>
            <guid>7xmTRCf0r7ZJTFW2qbIa</guid>
            <pubDate>Tue, 13 Jun 2023 01:49:30 GMT</pubDate>
            <description><![CDATA[The opinions in this are mine, I’m not writing this on behalf of anyone, and ahm.. any other disclaimers you think should be here probably are… So anyway… There was a network issue on Mainnet, and the Prysm team wrote up a great post-mortem. From a Teku perspective, we got a release out relatively quickly largely due to the initial event the day before, and we’d been working on it ‘in case it happened again’, and I’m glad we took that approach. It took time to understand the issue, and circli...]]></description>
            <content:encoded><![CDATA[<p><em>The opinions in this are mine, I’m not writing this on behalf of anyone, and ahm.. any other disclaimers you think should be here probably are… So anyway…</em></p><p>There was a network issue on Mainnet, and the Prysm team wrote up a great <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://offchain.medium.com/post-mortem-report-ethereum-mainnet-finality-05-11-2023-95e271dfd8b2">post-mortem</a>.</p><p>From a Teku perspective, we got a release out relatively quickly largely due to the initial event the day before, and we’d been working on it ‘in case it happened again’, and I’m glad we took that approach.</p><p>It took time to understand the issue, and circling back on it recently and reviewing the post-mortem written by the Prysm team, I still had niggles with the fix we had produced. It did fix our issue, but ultimately we shouldn’t have needed nearly as much <code>state</code> in memory as we have in the first place.</p><p>When digging, I noticed something subtle, but interesting, and I added some metrics, then based on that was later able to produce a fix to the actual problem that teku was experiencing during that time - more on that after some context.</p><h2 id="h-attestation-gossip" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Attestation Gossip</h2><p>To understand the issue, and how subtle it is, I’ll try to give a little background.</p><p>Attestations are a crucial part of our network. They get produced every epoch, by every active validator, and they get produced and propagated at basically the same time around the network - 4 seconds into the slot.</p><p>Every node sees these Attestations depending on what they’re listening to, and each node reads them and decides whether to pass them on.</p><p>These Attestations have a relatively simple structure (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#attestationdata">spec</a>). Your vote includes:</p><ul><li><p>source: validators opinion of the best currently justified checkpoint</p></li><li><p>target: block at the start of the current epoch</p></li><li><p>beacon_block_root: your current head on your chain.</p></li><li><p>slot: the slot you produced the attestation at.</p></li></ul><p>This is all wrapped up and signed.</p><p>Another better description can be found in <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://eth2book.info/capella/part3/containers/dependencies/#attestationdata">eth2book</a>.</p><h2 id="h-the-incident" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">The incident</h2><p>A number of attestations were arriving that had some interesting properties. The slot was current, but the target was an epoch old, and the head was also old. Typically they were on the canonical chain.</p><p>When validating attestation data, a certain amount of information is needed from a relatively recent state, and a logic process is followed to determine the ideal state to use.</p><p>If the head matches, this is ideal, and we can just use our head state - that covers over 90% of the cases that are seen for validation. After this we start a process of deciding what’s going to be the best state.</p><p>Given that we have the <code>target</code>, we can attempt to use that to find a state, and we can also attempt to use things such as the <code>finalized state</code> as a common ancestor.</p><p>Our fix straight after the incident was basically to ignore attestations to a <code>target</code> prior to the current justified epoch if the <code>target</code> of the attestation is on the canonical chain, and we can’t find the state in memory. This fix was in the <code>23.5.0</code> teku release, and was very effective at avoiding the problems seen during non finality.</p><h2 id="h-post-incident-testing" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Post-Incident Testing</h2><p>One cool thing about this community is the sense of ‘team’ and ‘us’ - it’s actually very supportive and inclusive.</p><p>This is a super power, because it lets us do things that would never be possible if it was always a huge competitive field.</p><p>So we spun up a testing network, a big one… 600k validators. The devops people we have in the group are amazing, and made it super easy for us. We ran with similar setup to mainnet in terms of ratios of clients, and proved we could break it in the way that was understood to be the issue. It was almost the issue, so we upped the anti, and things went boom. This is huge. Replicating the circumstances shows you understand what actually happened.</p><p>Afterwards we spun up the new versions of clients put out to rectify the issue (23.5.0 for teku), and saw the problem was no longer present. Then we pushed the understood circumstances, and gained confidence that even given a significantly worse set of circumstances, the network would be healthy.</p><p>This was a huge community win, I can’t speak highly enough of this group of people and their willingness to extrapolate and test more.</p><h2 id="h-2360" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">23.6.0</h2><p>After <code>23.5.0</code>, it was a lingering thought that we still needed to lookup states without a great description of why we needed them, and I did end up revisiting this area.</p><p>The metrics I added during investigation were very effective at showing percentages of our state selection choices, and I noticed a peculiarity, because basically the target that was in these problematic attestations were still ultimately canonical, and should have actually been able to be verified very cheaply (ancestor of head).</p><p>Prior to our initial fix, after we discounted the chain head as a candidate state, we would attempt to find the state where we processed the block that’s referenced as target of the attestation. Unfortunately we made the assumption that the <code>slot</code> of the attestation would match the <code>target</code> slot. While this does occur, it’s actually relatively common for the target to not be the <code>slot</code>. If you think about it, only 1 slot in 32 is the start of any given epoch. This was the core of the issue that caused the chaos for us in May, as we weren’t able to determine a good match here and went further down into trying to match scenarios, ran out of things to try, and regenerated states.</p><p>In <code>23.6.0</code>,the root cause of the bug that was present during non finality has been removed, by looking up the slot of the target rather than using the slot of the attestation.</p><p>We also extended the theory of using a chain head for validation, since we generally have head states present at all times, they’re usually a cheap way to validate, if we can locate them. Our selector now also checks for non canonical chains and if the attestation is an ancestor of any of those chains, and uses that chain head to verify against in that scenario.</p><h2 id="h-still-outstanding" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Still Outstanding</h2><p>While we’ve come a long way in reducing the number of states accessed during gossip validation, there’s other cases where we still need to investigate.</p><p>One desire I’ve had for a while is to bring memory under control, and we’ve reduced state access a lot, but there’s still regenerations happening when we drop the heap size down, so no changes to our recommended minimum at this stage, as it may impact validating nodes. If we can rectify another couple of accesses, we should be able to actually reduce our heap requirements, and that would be a huge improvement.</p><p>The other investigation that probably is notable is why state regeneration is taking a long period of time, because that impacts <code>tick</code> processing, which can have flow on effects.</p>]]></content:encoded>
            <author>rolfy-eth@newsletter.paragraph.com (Rolfy.eth)</author>
        </item>
        <item>
            <title><![CDATA[Withdrawals]]></title>
            <link>https://paragraph.com/@rolfy-eth/withdrawals</link>
            <guid>mq3RcnB2MwCHOUvOCYDh</guid>
            <pubDate>Wed, 15 Mar 2023 22:00:39 GMT</pubDate>
            <description><![CDATA[Ethereum - WithdrawalsSince the advent of the Beacon Chain, we asked for a stake for validators. At the time our story was basically "Once the merge is done, you&apos;ll be able to get these funds back". So, the merge is done, and it was successful, but the elephant in the room was that it didn&apos;t actually include the ability to get your funds back. It turns out that the merge was pretty complicated, and withdrawals are a tiny bit complicated, and to add another feature was potentially in...]]></description>
            <content:encoded><![CDATA[<h1 id="h-ethereum-withdrawals" class="text-4xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Ethereum - Withdrawals</h1><p>Since the advent of the Beacon Chain, we asked for a stake for validators. At the time our story was basically &quot;Once the merge is done, you&apos;ll be able to get these funds back&quot;.</p><p>So, the merge is done, and it was successful, but the elephant in the room was that it didn&apos;t actually include the ability to get your funds back. It turns out that the merge was pretty complicated, and withdrawals are a tiny bit complicated, and to add another feature was potentially introduce bugs or to delay the merge... So we shipped without it.</p><p>Bellatrix was the Fork of the Consensus-layer that included all the logic for the merge to work.</p><p>Capella is next! And the headline feature included in Capella will be withdrawals. It&apos;s pretty significant, because it&apos;s a cross-cutting feature that requires changes in both the Execution-layer (EL) and the Consensus-layer (CL).</p><h2 id="h-how-will-withdrawals-work" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How will withdrawals work?</h2><p><strong>TLDR;</strong> once you qualify for withdrawals, it&apos;s an automated process, and to qualify, your withdrawal key just needs to be an ethereum address.</p><h3 id="h-withdrawal-key" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Withdrawal Key</h3><p>To break down how this will work, we need to know a little bit about validators, specifically their withdrawal credentials.</p><p>There&apos;s 2 types of withdrawal credentials that a validator may have been configured with.</p><ul><li><p>BLS Keys - Validators at genesis were mostly set up with a BLS key for their withdrawal credentials. At the time we didn&apos;t really have a vision of how it would all slot together, and BLS credentials was a safe starting point.</p></li><li><p>ETH address - Validators for a while now have been able to use regular ethereum addresses for their withdrawal credentials.</p></li></ul><p>As of Bellatrix, it was only possible to set the credentials for withdrawals at the time of validator creation, and they could never be updated after that.</p><p>As of Capella, if you have configured your validator with BLS withdrawal credentials, you&apos;ll be able to update your credentials to use an ethereum address for your withdrawals.</p><h3 id="h-updating-my-withdrawal-key" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Updating my Withdrawal key</h3><p><strong>NOTE:</strong> Once the address has been updated on your validator, you will not be able to make further modifications to the withdrawal credentials, so it&apos;s important that you update this address to an address you own, and have the keys for.</p><p>As of Capella, you will be able to alter your validator withdrawal credentials to an Ethereum address, if it has been configured with a BLS address initially.</p><p>The change is a <strong>one-time</strong> opportunity to update your withdrawal credentials, and you will need either your MNEMONIC or your withdrawal BLS private key to sign a message to update your withdrawal credentials.</p><p>Updates are submitted to a Consensus-layer (CL) client, and they will be distributed to the network, and need to be included in a block in order to update your validator credentials. This is a similar concept to the Execution-layer (EL) concept of a transaction pool, where all clients basically access a pool as a source of information when building blocks.</p><p>It&apos;s possible that they aren&apos;t picked up by block producers, and there&apos;s no penalty for re-submitting the request periodically; for example if nothing has happened in 6 hours; if it has not been processed.</p><h3 id="h-getting-my-rewards" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Getting my rewards</h3><p>Validators have been accruing rewards since genesis. The system is reliant on 32 ETH to secure the network, but everything in excess of that is not needed by the CL.</p><p>This excess is what will be returned to active validators that have an ethereum address as their withdrawal address.</p><p>Over time, the validator list will be iterated on by validator index (from 1 to N, then loop), and up to 16 withdrawals can be included in each block. This excess of balance is called a partial withdrawal, and full withdrawals (exited validator balance) are also processed in the same way, except they draw down to zero eth.</p><p><strong>NOTE:</strong> At 16 withdrawals per block, and <code>7_200</code> blocks per day, roughly <code>115_200</code> withdrawals are possibly daily. This will mean that it&apos;ll take around 5 days to iterate the validator set (roughly <code>550_000</code>) if all validators are candidates for withdrawals, so you will get frequent deposits.</p><p>Importantly, the transfer process does not run any EVM code, it simply increases balance of the ethereum address that it&apos;s been told to allocate funds to. The advantage in doing this is that it is a free transaction (new operation), so balances aren&apos;t reduced by gas cost. Further, it&apos;s not a normal transaction might be reverted, the balance transfer is it&apos;s own special operation in the EL.</p><p>This process is automatic, and as long as you have an ethereum address as your withdrawal key, you will periodically be considered. BLS withdrawal credentials are still completely valid to use, so if you don&apos;t wish for periodic withdrawals, then staying on your BLS key (or creating a new validator with a BLS key) would be a good option for you to consider.</p><h3 id="h-capella-fork" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Capella Fork</h3><p>At the time of the Capella fork, it&apos;s possible that a small number of people will &apos;need&apos; to get changes in for BLS credentials, and it&apos;s possible that there may be a bit of a rush.</p><p>A number of people believe their mnemonic was compromised, and some people desparately want their funds back as quickly as possible for exited validators, and probably some other reasons as well. These are all valid reasons to potentially try to get a change in as quickly as possible to get your credentials updated.</p><p>For everyone else, I put it to you that it&apos;s not a rush. There&apos;s no real reason to be in the initial set of changes, at least as far as I can see, unless you&apos;re part of that small subset of people. We do expect a tiny bit of congestion on the change, but it&apos;s possible we&apos;re just projecting, it&apos;s hard to know how many people fall into the above buckets, or if there&apos;s other reasons.</p><p>Given that the BLS changes are included at a rate of 16 per block, the queue will progress quite quickly, and some validators will be prioritising their own changes as block builders, which is completely valid.</p><p>After 2-3 days though, the queue will likely be completely empty, even if every single validator wants to change their credentials. Similar to withdrawals per block, <code>115_200</code> BLS changes can be processed daily, and at the time of writing there&apos;s roughly <code>310_000</code> validators that could potentially wish to change to ethereum addresses for their withdrawal keys.</p><p>The reason I&apos;m saying this, is the funds aren&apos;t disappearing, and there&apos;s no real rush to get the initial allocation, as your validator balance will keep accruing regardless of what else happens on the network. So if you&apos;re not in that small subset of urgent changes, I would suggest holding off a few days, then your BLS change will go through quickly and you won&apos;t need to wait hours to check if it got into a block.</p><h3 id="h-tracking-my-balance" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Tracking my balance</h3><p>Previously, a good indicator for whether your validator is performing well was the balance. People would use this as a guide, and the balance would only ever go up, so looking at todays balance vs. yesterdays would be a good indicator for how your validator is doing in terms of activity on the network.</p><p>Once you&apos;re using withdrawals, your balance graph will look more like a saw-tooth, dropping down to 32 ETH periodically (roughly every 4 days). You can still use it to see how you&apos;re going, but it&apos;s worth understanding the difference in the pattern you&apos;ll see, as it&apos;ll initially be an adjustment to see your balance periodically reset to 32 ETH.</p><p>Your ethereum address will also have periodic eth deposits which will be potentially a good way to map over time if the address is specific to the validator balance.</p><h2 id="h-wen-withdrawals" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">WEN Withdrawals?</h2><p>Withdrawals are coming very soon to ethereum, with the Capella fork. We&apos;re in the final stages of testing, and it&apos;s looking very promising so far! Obviously if issues are found there may be delays, but it&apos;s very very close, and the spec is currently frozen.</p>]]></content:encoded>
            <author>rolfy-eth@newsletter.paragraph.com (Rolfy.eth)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/04d1e6c9c5091f022b655bed852c11b3646254604802d40f3d5af2957a1c0d78.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Hello!]]></title>
            <link>https://paragraph.com/@rolfy-eth/hello</link>
            <guid>mBAbdY9fSUDvFqLBk4tr</guid>
            <pubDate>Wed, 15 Mar 2023 21:55:07 GMT</pubDate>
            <description><![CDATA[I’ve setup this blog as a space to share ideas. Currently it’ll be consensus-layer centric, as that is my expertise. Maybe people will find it insightful, ultimately I’m representing my opinions, and it may not be 100% accurate. If there’s issues let me know and I’ll attempt to rectify any inaccuracies that you may have found.]]></description>
            <content:encoded><![CDATA[<p>I’ve setup this blog as a space to share ideas. Currently it’ll be consensus-layer centric, as that is my expertise. Maybe people will find it insightful, ultimately I’m representing my opinions, and it may not be 100% accurate. If there’s issues let me know and I’ll attempt to rectify any inaccuracies that you may have found.</p>]]></content:encoded>
            <author>rolfy-eth@newsletter.paragraph.com (Rolfy.eth)</author>
        </item>
    </channel>
</rss>