'If you wish to make an apple pie from scratch, you must first invent the universe'.

-Carl Sagan

Note: This article dives directly into assembly code and explores advanced bit operations. If you're an experienced developer or researcher, you're in the right place. However, if you're not familiar with these topics, I recommend reading the article “Solidity and EVM: Bit Shifting and Masking in Assembly(YUL)“ by 0xweiss first before continuing with this one. This background knowledge will help you fully understand the content presented here!

This article is all about showing how much gas you can save by using Yul to build complex Solidity data structures, instead of sticking with Solidity's built-in features. Keep in mind that what you'll see in the following article and code examples are for educational purposes. If you plan to use low-level assembly code in a real-world production environment, it's crucial to have it thoroughly audited for security, feel free to reach out for auditing services!

To keep things practical and hands-on, we will dive straight into real-world examples without getting bogged down in theory. This article will follow a common Solidity use case (creating a new instance of a complex struct and storing it in a mapping), in order to explain the mechanics of bit masking:

The following elements are given to us in order to create an instance of the struct Pool and store it on a mapping:

A packed optimized Struct type Pool:

 struct Pool {
    /// Slot 0
    uint16 poolRatio; // uint16 is enough to store BPS
    uint16 poolFee;
    uint48 poolActivation; //uint48 is enough to cover timestamps
    uint48 poolLastUpdate;
    //uint128 is enough to store up to 340282366920938463463e18 amount
    uint128 poolMaxDebt;
    
    /// Slot 1
    uint128 poolMinDebt;
    uint128 poolGain;

    /// Slot 2
    uint128 poolDebt;
    uint128 poolLoss; 
 }

A mapping to store pools:

 mapping(address => Pool) public pools;

And the following cached values:

 uint16 poolRatio = 300;

 uint16 poolFee = 500;

 uint48 poolActivation = block.timestamp;

 uint48 poolLastUpdate = block.timestamp;

 uint128 poolMaxDebt = 99999999e18;

 uint128 poolMinDebt = 999e18;

Note: The rest of the elements would be 0 on a new pool.

The following code snippet gives a solution for the use case we have discussed above. It's important to note that as the size and complexity of the struct increase, the compiler tends to perform more behind-the-scenes operations, resulting in increased gas costs (we don’t want that).

 // New pool struct is initialized with the required values
 Pool memory pool = Pool({
      poolFee: poolFee,
      poolRatio: poolRatio,
      poolActivation: block.timestamp,
      poolLastUpdate: block.timestamp,
      poolMaxDebt: poolMaxDebt,
      poolMinDebt: poolMinDebt,
      poolDebt: 0,
      poolGain: 0,
      poolLoss: 0
 });

 // The pool created above is stored in the mapping
 pools[poolAddress] = pool;

Now is when the Optimizoor Auditor Alpha comes into play. Our goal is to create a new Pool instance without all the overhead generated by the compiler.

To achieve this, it's crucial to understand how the EVM handles storage and the packing of structs:

• Storage Slots: Struct fields are stored in slots. In our case, the Pool struct is already packed and occupies three slots.

• Storage Allocation: An instance of a struct is stored from slot "n" to slot "n + (number of slots - 1)". For our purpose, we want to create a struct where the fields of the first slot are at slot "n," the second at "n + 1," and the third at "n + 2."

• Field Packing Order: Elements within a slot are packed from right to left. For example, in the Pool struct, the first uint16 (poolRatio) occupies the 16 Least Significant Bits (LSB), and the uint128 (poolMaxDebt) resides in the 128 Most Significant Bits (MSB) of the 32-byte slot.

Let’s move to the next step, which involves creating the required 32-byte slots with their correctly padded elements. To optimize this struct packing process, we will rely on the two techniques mentioned earlier:

• Bit-shifting: This involves shifting a group of bits either to the right or left within a 32-byte slot.

• Bit masking: This technique allows us to accurately select the required bits within a 32-byte slot.

Mastering assembly provides you with the freedom to choose your preferred technique. In my opinion, a balanced approach that combines both bit shifting and bit masking is essential.

As a general guideline, I tend to employ bit shifting when dealing with selections exceeding 64 bits, and I rely on bit masking for selections involving fewer than 64 bits. This combination allows for flexibility and efficiency in assembly coding.

This is how the field packing of the first slot of the Pool struct would look like, using assembly and the techniques mentioned above, now you now why the article is called “Degen Bit Masking“:

 or(
       shl(128, poolMaxDebt),
             or(
                   shl(80, and(0xffffffffffff, timestamp())),
                   or(
                         shl(32, and(0xffffffffffff, timestamp())),
                         or(
                              shl(16, and(0xffff, poolFee)),
                              and(0xffff, poolRatio) // 4 LSB bytes
                         )
                   )
             )
       )
 )

Let’s break down this process:

We work from the MSB to the LSB using or gates to combine the fields sequentially:

// The or gate combines both operators into the same 32-byte slot
 or(   
       // this shift moves the poolMaxDebt value all the way to the
       // left, leaving the 128 LSB free for the rest of the fields
       shl(128, poolMaxDebt), 
       ///....////     
 )

We do this recursively, until hitting those lengths of 64 bit lengths mentioned above, we use the bitmask 0xffffffffffff in order to select the 48 bits of the current timestamp making sure the rest is 0

///....///,
or(    // we shift poolLastUpdate 80 bits to the left
       // 80 bits = 48 + 16 + 16
       shl(80, and(0xffffffffffff, timestamp())),//
       ///....////
 )

Then for the fields poolActivation, poolRatio and poolFee we do the following:

///....////,
 or(
       shl(32, and(0xffffffffffff, timestamp())), // poolActivation
       or(.  // same masking than above but with 16 bits 
             shl(16, and(0xffff, poolFee)),  
             and(0xffff, poolRatio) // 4 LSB bytes
       )
  )

Nice!, so we got out first slot packed, bit-wise it would something like this:

128-bits      48-bits          48-bits          16-bits   16-bits
poolMaxDebt | poolLastUpdate | poolActivation | poolFee | poolRatio

For the second slot, since the last three fields are 0, we only have to allocate poolMinDebt to the 128 LSB, cleaning the 128 upper-bits:

shr(128, shl(128, poolMinDebt))

Now that we have our struct created we should store it correctly on the pools mapping. In order to know which storage slot is “n”, we need to calculate the keccak hash of the address of the new pool and the slot of the mapping. In solidity it will look something like this keccak(poolAddress,pools.slot) but since this is supposed to be the degen solution we are going to calculate it in assembly using the scratch space 0x00-0x40:

 mstore(0x00, poolAddress)
 mstore(0x20, pools.slot)
 slot := keccak256(0x00, 0x40)

Finally we can store the two 32-byte slots that we created in the bit masking step:

// store the contents of the first slot at "n"  
sstore(
       slot,
       or(
             shl(128, poolMaxDebt),
             or(
                   shl(80, and(0xffffffffffff, timestamp())),
                   or(
                         shl(32, and(0xffffffffffff, timestamp())),
                         or(
                              shl(16, and(0xffff, poolFee)),
                              and(0xffff, poolRatio) // 4 LSB bytes
                         )
                   )
             )
       )
 )
 // store the contents of the second slot at "n + 1"  
 sstore(add(slot, 1), shr(128, shl(128, poolMinDebt)))

While this degen optimization may appear complex and challenging to the average solidity developer, it offers a remarkable gas-saving advantage, reducing gas costs by around 5,000 units when compared to the Solidity version.

Embrace low-level assembly anon.

Hope you have enjoyed this article and it piqued your interest in delving into low-level EVM code. If you or you project require assistance with Smart Contract gas optimisation, code reviews, or security assessments, don't hesitate to reach out to me via my Twitter account.

'If you wish to make an apple pie from scratch, you must first invent the universe'.

-Carl Sagan

Note: This article dives directly into assembly code and explores advanced bit operations. If you're an experienced developer or researcher, you're in the right place. However, if you're not familiar with these topics, I recommend reading the article “Solidity and EVM: Bit Shifting and Masking in Assembly(YUL)“ by 0xweiss first before continuing with this one. This background knowledge will help you fully understand the content presented here!

This article is all about showing how much gas you can save by using Yul to build complex Solidity data structures, instead of sticking with Solidity's built-in features. Keep in mind that what you'll see in the following article and code examples are for educational purposes. If you plan to use low-level assembly code in a real-world production environment, it's crucial to have it thoroughly audited for security, feel free to reach out for auditing services!

To keep things practical and hands-on, we will dive straight into real-world examples without getting bogged down in theory. This article will follow a common Solidity use case (creating a new instance of a complex struct and storing it in a mapping), in order to explain the mechanics of bit masking:

The following elements are given to us in order to create an instance of the struct Pool and store it on a mapping:

A packed optimized Struct type Pool:

 struct Pool {
    /// Slot 0
    uint16 poolRatio; // uint16 is enough to store BPS
    uint16 poolFee;
    uint48 poolActivation; //uint48 is enough to cover timestamps
    uint48 poolLastUpdate;
    //uint128 is enough to store up to 340282366920938463463e18 amount
    uint128 poolMaxDebt;
    
    /// Slot 1
    uint128 poolMinDebt;
    uint128 poolGain;

    /// Slot 2
    uint128 poolDebt;
    uint128 poolLoss; 
 }

A mapping to store pools:

 mapping(address => Pool) public pools;

And the following cached values:

 uint16 poolRatio = 300;

 uint16 poolFee = 500;

 uint48 poolActivation = block.timestamp;

 uint48 poolLastUpdate = block.timestamp;

 uint128 poolMaxDebt = 99999999e18;

 uint128 poolMinDebt = 999e18;

Note: The rest of the elements would be 0 on a new pool.

The following code snippet gives a solution for the use case we have discussed above. It's important to note that as the size and complexity of the struct increase, the compiler tends to perform more behind-the-scenes operations, resulting in increased gas costs (we don’t want that).

 // New pool struct is initialized with the required values
 Pool memory pool = Pool({
      poolFee: poolFee,
      poolRatio: poolRatio,
      poolActivation: block.timestamp,
      poolLastUpdate: block.timestamp,
      poolMaxDebt: poolMaxDebt,
      poolMinDebt: poolMinDebt,
      poolDebt: 0,
      poolGain: 0,
      poolLoss: 0
 });

 // The pool created above is stored in the mapping
 pools[poolAddress] = pool;

Now is when the Optimizoor Auditor Alpha comes into play. Our goal is to create a new Pool instance without all the overhead generated by the compiler.

To achieve this, it's crucial to understand how the EVM handles storage and the packing of structs:

• Storage Slots: Struct fields are stored in slots. In our case, the Pool struct is already packed and occupies three slots.

• Storage Allocation: An instance of a struct is stored from slot "n" to slot "n + (number of slots - 1)". For our purpose, we want to create a struct where the fields of the first slot are at slot "n," the second at "n + 1," and the third at "n + 2."

• Field Packing Order: Elements within a slot are packed from right to left. For example, in the Pool struct, the first uint16 (poolRatio) occupies the 16 Least Significant Bits (LSB), and the uint128 (poolMaxDebt) resides in the 128 Most Significant Bits (MSB) of the 32-byte slot.

Let’s move to the next step, which involves creating the required 32-byte slots with their correctly padded elements. To optimize this struct packing process, we will rely on the two techniques mentioned earlier:

• Bit-shifting: This involves shifting a group of bits either to the right or left within a 32-byte slot.

• Bit masking: This technique allows us to accurately select the required bits within a 32-byte slot.

Mastering assembly provides you with the freedom to choose your preferred technique. In my opinion, a balanced approach that combines both bit shifting and bit masking is essential.

As a general guideline, I tend to employ bit shifting when dealing with selections exceeding 64 bits, and I rely on bit masking for selections involving fewer than 64 bits. This combination allows for flexibility and efficiency in assembly coding.

This is how the field packing of the first slot of the Pool struct would look like, using assembly and the techniques mentioned above, now you now why the article is called “Degen Bit Masking“:

 or(
       shl(128, poolMaxDebt),
             or(
                   shl(80, and(0xffffffffffff, timestamp())),
                   or(
                         shl(32, and(0xffffffffffff, timestamp())),
                         or(
                              shl(16, and(0xffff, poolFee)),
                              and(0xffff, poolRatio) // 4 LSB bytes
                         )
                   )
             )
       )
 )

Let’s break down this process:

We work from the MSB to the LSB using or gates to combine the fields sequentially:

// The or gate combines both operators into the same 32-byte slot
 or(   
       // this shift moves the poolMaxDebt value all the way to the
       // left, leaving the 128 LSB free for the rest of the fields
       shl(128, poolMaxDebt), 
       ///....////     
 )

We do this recursively, until hitting those lengths of 64 bit lengths mentioned above, we use the bitmask 0xffffffffffff in order to select the 48 bits of the current timestamp making sure the rest is 0

///....///,
or(    // we shift poolLastUpdate 80 bits to the left
       // 80 bits = 48 + 16 + 16
       shl(80, and(0xffffffffffff, timestamp())),//
       ///....////
 )

Then for the fields poolActivation, poolRatio and poolFee we do the following:

///....////,
 or(
       shl(32, and(0xffffffffffff, timestamp())), // poolActivation
       or(.  // same masking than above but with 16 bits 
             shl(16, and(0xffff, poolFee)),  
             and(0xffff, poolRatio) // 4 LSB bytes
       )
  )

Nice!, so we got out first slot packed, bit-wise it would something like this:

128-bits      48-bits          48-bits          16-bits   16-bits
poolMaxDebt | poolLastUpdate | poolActivation | poolFee | poolRatio

For the second slot, since the last three fields are 0, we only have to allocate poolMinDebt to the 128 LSB, cleaning the 128 upper-bits:

shr(128, shl(128, poolMinDebt))

Now that we have our struct created we should store it correctly on the pools mapping. In order to know which storage slot is “n”, we need to calculate the keccak hash of the address of the new pool and the slot of the mapping. In solidity it will look something like this keccak(poolAddress,pools.slot) but since this is supposed to be the degen solution we are going to calculate it in assembly using the scratch space 0x00-0x40:

 mstore(0x00, poolAddress)
 mstore(0x20, pools.slot)
 slot := keccak256(0x00, 0x40)

Finally we can store the two 32-byte slots that we created in the bit masking step:

// store the contents of the first slot at "n"  
sstore(
       slot,
       or(
             shl(128, poolMaxDebt),
             or(
                   shl(80, and(0xffffffffffff, timestamp())),
                   or(
                         shl(32, and(0xffffffffffff, timestamp())),
                         or(
                              shl(16, and(0xffff, poolFee)),
                              and(0xffff, poolRatio) // 4 LSB bytes
                         )
                   )
             )
       )
 )
 // store the contents of the second slot at "n + 1"  
 sstore(add(slot, 1), shr(128, shl(128, poolMinDebt)))

While this degen optimization may appear complex and challenging to the average solidity developer, it offers a remarkable gas-saving advantage, reducing gas costs by around 5,000 units when compared to the Solidity version.

Embrace low-level assembly anon.

Hope you have enjoyed this article and it piqued your interest in delving into low-level EVM code. If you or you project require assistance with Smart Contract gas optimisation, code reviews, or security assessments, don't hesitate to reach out to me via my Twitter account.