Blockchain protocols suffer from scalability trilemma such that optimal solution to achieve decentralization, higher transaction throughput and security at the same time is still a research topics.
Some protocols compromise scalability over decentralization like ethereum while other protocols like solana, tron compromise decentralization over scalability to have more tps .
Among all these protocol , L2’s trying to find a sweat spot to achieve high scalability while inheriting security and decentralization of ethereum . However it comes at cost of technical naunces and complexity .
One of the major bottlenecks in decentralized architectures is every nodes/particpant have to download/store full copy of data to participate in a system. This limit decentralization to only select few who can afford good hardware specs.
To solve this issue various scalaing solutions tried like state channels , plasma etc . However ethereum resort to sharding techniques to achieve scalibity.
Sharding is a technique where data is divided into chunks and assign to each particpant to perform specific sets of task . Once they acheive their task , it then communicated to other particpant/nodes which then validate and added to block.
There are two main types of sharding
Execution based sharding
Data based sharding
In Execution based sharding , the execution of smart contracts are divided and assigned to different nodes . Each node will then execute a transaction on their chains and once done will be cross communicated to other shards/chains which will then be synced to the main blockchain
However, later it is dropped for data-based sharding which is explained below
The main crux of data sharding is to focus on data availability rather than data execution. It is based on the assumption that not everyone needs to store a full copy of data however it should be available to anyone who seeks it. That means data have to stored smartly such that at any given time, any data can be verified and available if required.
So in Data sharding data are split in the form of chunks among nodes. In this way every node doesn't have to store a full copy of data rather they can just store chunks of data and with some clever math to ensure data integrity and availability.
If every node doesn't store all data then how does it maintain data integrity?
To solve this problem data is encoded and extended using polynomial interpolation and by doing some clever math it is made sure that the node only needs to verify 50% of the chunk instead of the full chunk to ensure data availability and integrity. That’s call Data Availability Sample (DAS) . Node only do necessary work to satisfy DAS needs
What If a malicious actor/node put some random data in between chunks? How will it ensure that dosent happen especially in financial and other critical transactions?
To solve this problem , another mathmetical trick KGZ ccomittments are done to make sure every data is in correct polynomial format . Anyone can read about this mechanism here https://dankradfeist.de/ethereum/2020/06/16/kate-polynomial-commitments.html . It is sort of merkel tree mechanism in tradiitional blockchain world to ensure data integerity.
Bottom line if malicious actor puts some random data in a chunk then they will have to fill more than 50% of the chunk with data and if that happens it will cause DAS to fail and probability of doing that is 2 power -30.
How it will help in achieving scalability?
Currently ethereum L2's have to execute transaction on their own chain and then submit proof as call data on L1 chain . How much data they can post as call data on L1 is limited due to block size limitation . However with dank sharding , block size will be lifted to 1 mb or higher due to the fact that every node dosent have to download all data so overhead on node to particpate in a system will be reduced.
This also means L2s can add more data on L1 in the form of blobs which means they can execute more transaciton on L2 and attach more.
This will cause tps to increase 20x over time.
Development Stages:
Proto DankSharding (EIP-4844):
The main purpose of this improvement protocol or proto-dank sharding is to do the major heavy lifting before actually implementing dank sharding thus easing off pressure from the development side.
The main feature introduced will be a new transaction type, which we call a blob-carrying transaction. A blob-carrying transaction is like a regular transaction, except it also carries an extra piece of data called a blob. This will help clients in implementing necessary changes on their end before full sharding.
Full DankSharding Specs:
Once it's done, changes on the consensus level will be executed which include the implementation of polynomial interpolation and KGZ commitments.
https://members.delphidigital.io/reports/the-hitchhikers-guide-to-ethereum
https://research.thetie.io/danksharding-ethereums-scalability-killer/
https://research.thetie.io/blockchain-trilemma/#on-chain