What is a Syncer? It's a scheduled job to retrieve transactions from one contract regularly and persist them locally.
The Syncer has two main parts:
Syncer: responsible for retrieving transactions from the contract and persistence handling.
Syncer Server: repsonsible for job scheduling and monitoring.
Parser：Parse the transactions.
Syncer is useful, because it formats and caches on-chain data, allowing various dApps and services to read on-chain data more quickly, improving the experience.
The traditional Syncer is implemented by connecting blockchain nodes, which has the advantage of being more stable, but we also have a lighter option -- to implement Syncer by calling the API of the Etherscan-like browser! Etherscan provides an interface to get all transactions of a specific contract, so we can do Syncer with lower complexity. This doesn't sound so Web3 because Syncer relies on the browser service to function properly, but in most scenarios, it works well!
The processing logic of Syncer contain below three main steps:
Retrieves the latest block number of the blockchain.
Based on the last locally persited block number, retrieves transactions from the chain using Etherscan API and persist locally.
Update last block number of the contract locally.
# https://github.com/WeLightProject/tai_shang_nft_gallery/blob/main/lib/tai_shang_nft_gallery/nft/syncer.ex
defmodule TaiShangNftGallery.Nft.Syncer do
alias TaiShangNftGallery.NftContract
alias TaiShangNftGallery.TxHandler
alias TaiShangNftGallery.Chain.Fetcher
alias TaiShangNftGallery.ScanInteractor
require Logger
@api_keys %{
"Moonbeam"=> System.get_env("MOONBEAM_API_KEY")
}
def sync(chain, %{last_block: last_block} = nft_contract) do
# Step 1
best_block = Fetcher.get_block_number(chain)
# Step 2
do_sync(chain, nft_contract, last_block, best_block)
# Step 3
NftContract.update(
nft_contract,
%{last_block: best_block +1}
)
end
def do_sync(%{name: name} = chain, %{addr: addr} = nft_contract, last_block, best_block) do
# Step 2.1- Retrieves Txns by Etherscan API
{:ok, %{"result"=> txs}}=
ScanInteractor.get_txs_by_contract_addr(
chain,
addr,
last_block,
best_block,
@api_keys[name]
)
# Step 2.2- Persist transactions locally
handle_txs(chain, nft_contract, txs)
end
def handle_txs(chain, nft_contract, txs) do
Enum.each(txs, fn tx->
Logger.info("Handling tx: #{inspect(tx)}")
tx_atom_map = ExStructTranslator.to_atom_struct(tx)
TxHandler.handle_tx(chain, nft_contract, tx_atom_map)
end)
end
end
Syncer Server
Syncer Server is implemented by GenServer. For those who do not know about it, you can imagine that it's some kind of process that runs forever. If there is any error, it can automatically restart.
The GenServer operates by handling the messages received, from itself or others. There are two main functions here:
init: The initialization function. It loads the Contract locally, store it in the process state and sends itself a :sync message.
handle_info: The :sync message handler that calls the API of the Syncer to do the actual work.
After each processing, the Process.send_after/3 function sends itself a :sync message after 10 minutes.
# https://github.com/WeLightProject/tai_shang_nft_gallery/blob/main/lib/tai_shang_nft_gallery/syncer_server.ex
defmodule TaiShangNftGallery.SyncerServer do
@moduledoc """
Genserver as Syncer
"""
alias TaiShangNftGallery.Nft.Syncer
alias TaiShangNftGallery.NftContract
use GenServer
require Logger
@sync_interval 600_000 # 10minutes
# +-----------+
# | GenServer |
# +-----------+
def start_link(state) do
GenServer.start_link(__MODULE__, state, name: :"#nft_syncer")
end
def init([nft_contract_id: nft_contract_id]) do
Logger.info("SyncerServer started yet.")
nft_contract =
nft_contract_id
|> NftContract.get_by_id()
|> NftContract.preload()
state =
[nft_contract: nft_contract]
send(self(), :sync)
{:ok, state}
end
def handle_info(:sync, [nft_contract: nft_contract] = state) do
Syncer.sync(nft_contract.chain, nft_contract)
sync_after_interval()
{:noreply, state}
end
def sync_after_interval() do
Process.send_after(self(), :sync, @sync_interval)
end
end
Decode Transaction
Different transactions actually represent different contract method call. In order to identify what exactly the transaction is doing, we have to decode the transaction input because the input is a hex value like 0x379607f5000000000000000000000000000000000000000000000000000000000000026d. TxHandler.handle_tx/3 uses ABI related APIs encapsulated in ABIHandler.find_and_decode/2 to accomplish that.
# https://github.com/WeLightProject/tai_shang_nft_gallery/blob/main/lib/tai_shang_nft_gallery/tx_handler.ex
defmodule TaiShangNftGallery.TxHandler do
@moduledoc """
Handle Ethereum Tx.
"""
alias TaiShangNftGallery.ABIHandler
alias TaiShangNftGallery.NftContract
def handle_tx(chain, nft_contract, tx) do
%{
from: from,
to: to,
value: value,
input: input
} =tx
do_handle_tx(chain, from, to, value, input, nft_contract)
end
def do_handle_tx(chain, from, to, value, input, %{type: type} = nft_contract) do
input_handled =
nft_contract
|> NftContract.preload(:deep)
|> Map.get(:contract_abi)
|> Map.get(:abi)
|> ABIHandler.find_and_decode(input)
"Elixir.TaiShangNftGallery.TxHandler.#{type}"|> String.to_atom()
|> apply(:handle_tx, [chain, nft_contract, from, to, value, input_handled])
end
end
# https://github.com/WeLightProject/tai_shang_nft_gallery/blob/main/lib/tai_shang_nft_gallery/abi_handler.ex
defmodule TaiShangNftGallery.ABIHandler do
alias Utils.TypeTranslator
def find_and_decode(abi, input_hex) do
abi|> ABI.parse_specification
|> ABI.find_and_decode(TypeTranslator.hex_to_bin(input_hex))
end
end
In the exact contract handler (type Web3Dev), we can see that only below methods are handled. Other than those, it returns {:ok, "pass"} directly.
safeTransferFrom / trasnferFrom: Used to transfer the ownership of the NFT
Below is Syncer implemented in JavaScript. In this example, I persist the data using sqlite but you can choose anything you are familiar with. And I simply use setTimeout and Catch-All-Error approach to simulate a forever running process.
importWeb3from'web3';
importgotfrom'got';
importsqlite3from'sqlite3';
importabiDecoderfrom'abi-decoder';
const db =new sqlite3.Database('./sqlite.db');
const web3 =new Web3(new Web3.providers.HttpProvider('https://rpc.api.moonbeam.network'));
const INTERVAL =1*60*1000; // 10 minutes
const API_KEY ='Y6AIFQQVAJ3H38CC11QFDUDJWAWNCWE3U8';
const CONTRACT_ADDRESS ='0xb6FC950C4bC9D1e4652CbEDaB748E8Cdcfe5655F';
const CONTRACT_ABI = []; // ABI can be retrieved from https://moonbeam.moonscan.io/address/0xb6FC950C4bC9D1e4652CbEDaB748E8Cdcfe5655F#code
abiDecoder.addABI(CONTRACT_ABI);
async functionsync() {
try {
const contract = awaitgetContractInfo(CONTRACT_ADDRESS);
// Step 1 - Retrieves the latest block number of the blockchainconstlatestBlock = awaitweb3.eth.getBlockNumber();
console.log(`Latest blockNumber is: ${latestBlock}`);
constlastBlockNumber = contract.lastBlockNumber || latestBlock - 100000;
// Step 2.1 - Retrieves Txns by Etherscan APIconsttransactions = awaitgetTransactionsOfContract(contract.address, lastBlockNumber, latestBlock);
console.log(`${transactions.length} transactions retrieved from Block ${lastBlockNumber} to ${latestBlock}.`);
if (transactions.length > 0) {
// Step 2.2 - Handle transactions
await handleTransactions(contract, transactions);
}
// Step 3 - Persist the latest block numbercontract.lastBlockNumber = latestBlock + 1;
awaitupdateContractLastBlockNumber(contract);
console.log(`lastBlockNumber of contract ${contract.id} is updated to ${contract.lastBlockNumber}.`);
} catch (error) {
console.error('Failed to sync transactions.', error);
}
nextRun();
}
functionnextRun() {
setTimeout(sync, INTERVAL);
}
async functiongetTransactionsOfContract(contractAddress, fromBlock, toBlock) {
const url = `https://api-moonbeam.moonscan.io/api?module=account&action=txlist&address=${contractAddress}&startblock=${fromBlock}&endblock=${toBlock}&sort=asc&apikey=${API_KEY}`;
const response = await got(url).json();
return response.result;
}
async function_getContract(address) {
returnnew Promise(function (resolve, reject) {
db.get('SELECT id, lastBlockNumber, address FROM contract WHERE address = $address', { $address: address }, function (err, result) {
if (err) {
return reject(err);
}
return resolve(result);
});
});
}
async functiongetContractInfo(address) {
let contract = await_getContract(address);
if (contract) {
returncontract;
}
contract = {
lastBlockNumber: 0,
address
};
const contractId = await _runSQL(
'INSERT INTO contract (lastBlockNumber, address) VALUES ($lastBlockNumber, $address)',
{
$lastBlockNumber: contract.lastBlockNumber,
$address: contract.address
}
);
contract.id = contractId;
returncontract;
}
asyncfunctionupdateContractLastBlockNumber(contract) {
return _runSQL(
'UPDATE contract SET lastBlockNumber = $lastBlockNumber WHERE id = $id',
{
$lastBlockNumber: contract.lastBlockNumber,
$id: contract.id
}
);
}
async functionexecute() {
// SQLite DB tables should be initialized first.
await sync();
}
execute();
Decode Transactions
In order to decode the transaction, we need to unitilize the APIs provided by Web3.
encodeFunctionCall
web3.eth.abi.encodeFunctionCall(jsonInterface, parameters);Encodes a function call using its JSON interface object and given parameters.
Parameters
jsonInterface - Object: The JSON interface object of a function.
parameters - Array: The parameters to encode.
Function - (optional) Optional callback, returns an error object as first parameter and the result as second.
Returns
String - The ABI encoded function call. Means function signature + parameters.
web3.eth.call(callObject [, defaultBlock] [, callback])Executes a message call transaction, which is directly executed in the VM of the node, but never mined into the blockchain.
Parameters
Object - A transaction object, see web3.eth.sendTransaction. For calls the from property is optional however it is highly recommended to explicitly set it or it may default to address(0) depending on your node or provider.
Number|String|BN|BigNumber - (optional) If you pass this parameter it will not use the default block set with web3.eth.defaultBlock. Pre-defined block numbers as "earliest", "latest" and "pending" can also be used.
Function - (optional) Optional callback, returns an error object as first parameter and the result as second.
Returns
Promise returns String - The returned data of the call, e.g. a smart contract functions return value.
