# How to deploy a prod-grade Starknet node By [Clemlaflemme](https://paragraph.com/@clemlaflemme) · 2022-10-03 --- [Equilibrium](https://equilibrium.co/) develops a RPC node for Starknet and provides a Docker image to run it. However, when it comes to running a container in production, some extra settings (like a load balancer, automatic reboot, CPU scaling, etc.) are required. In this tutorial, I will explain how to deploy such production-grade nodes in AWS, Azure and GCP. All the files used in this tutorial are available in the [corresponding repo](https://github.com/ClementWalter/starknet-nodes). Note: some of the snippets use [jq](https://stedolan.github.io/jq/) for easily parsing and manipulating json data from the console. If you don't want to install it, just break the command and copy/paste manually the selected entry. AWS --- ### AWS CLI setup To deploy programmatically to AWS, you first need to install and configure the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). If you are okay with this step, just make sure that your configured credentials have the appropriate [policy](https://github.com/ClementWalter/starknet-nodes/blob/ea49f1a6da8950c4628e89b1e9b3c63ccf1023bd/docs/aws/policy.json) and jump to the next section. It is recommended not to use your own root user credentials with the CLI. Instead, you should create an specific policy for the task and then an IAM User granted only these accesses. Eventually, you can generate credentials for this _user_ and create a _profile_ in the CLI for using them. Ouch. But we've got you covered! #### TL;DR copy/paste curl https://raw.githubusercontent.com/ClementWalter/starknet-nodes/main/docs/aws/policy.json -o policy-document.json aws iam create-policy --policy-name docker-ecs-context --policy-document 'file://policy-document.json' > policy.json rm policy-document.json aws iam create-group --group-name docker-ecs-users aws iam attach-group-policy --group-name docker-ecs-users --policy-arn `cat policy.json | jq ".Policy.Arn" -r` aws iam create-user --user-name docker-ecs-user aws iam add-user-to-group --group-name docker-ecs-users --user-name docker-ecs-user aws iam create-access-key --user-name docker-ecs-user > access_key.json aws configure --profile docker-ecs #### Details * create an AWS policy using this [policy file](https://github.com/ClementWalter/starknet-nodes/blob/ea49f1a6da8950c4628e89b1e9b3c63ccf1023bd/docs/aws/policy.json) * either using the [AWS console](https://us-east-1.console.aws.amazon.com/iam/home#/policies$new?step=edit) * or with the cli: aws iam create-policy --policy-name docker-ecs-context --policy-document 'file://docs/aws/policy.json' > policy.json * create an AWS user group and add it the above created policy * either with the [AWS console](https://us-east-1.console.aws.amazon.com/iamv2/home?region=eu-west-1#/groups/create) * or with the cli: aws iam create-group --group-name docker-ecs-users aws iam attach-group-policy --group-name docker-ecs-users --policy-arn `cat policy.json | jq ".Policy.Arn" -r` * create an AWS User, select only "Access key - Programmatic access", and add it to the above created group * either using the [AWS console](https://us-east-1.console.aws.amazon.com/iam/home#/users$new?step=details) * or with the cli: aws iam create-user --user-name docker-ecs-user aws iam add-user-to-group --group-name docker-ecs-users --user-name docker-ecs-user aws iam create-access-key --user-name docker-ecs-user > access_key.json * create an AWS profile for deploying the node using the generated credentials aws configure --profile docker-ecs ### CloudFormation deployment Now that the hard part is done, let us focus on the easy one: deploying the [eqlabs/pathfinder docker image](https://hub.docker.com/r/eqlabs/pathfinder) to AWS. Indeed, we use the [Docker<>ECS integration](https://docs.docker.com/cloud/ecs-integration/) to deploy a whole [CloudFormation stack](https://aws.amazon.com/cloudformation/) right from a simple `docker-compose.yml` file. This deployment only requires to change the [docker context](https://docs.docker.com/engine/context/working-with-contexts/) to use ECS. So basically a simple `docker compose up` is enough. Which means that you need to have docker compose installed on your machine. Refer to [their doc](https://docs.docker.com/get-docker/) for your own configuration. #### TL;DR copy/paste curl https://raw.githubusercontent.com/ClementWalter/starknet-nodes/main/docker-compose.yml -o docker-compose.yml curl https://raw.githubusercontent.com/ClementWalter/starknet-nodes/main/docs/aws/docker-compose.aws.yml -o docker-compose.aws.yml docker context create ecs starknet-ecs docker context use starknet-ecs docker compose -f docker-compose.yml -f docker-compose.aws.yml up #### Detailed story The creation of the `docker context` lets "update" all the docker commands to use AWS services in a pre-defined manner. For instance, it creates a [AWS Cloudformation](https://aws.amazon.com/cloudformation/) stack with the following main components: * [Elastic File Systems](https://aws.amazon.com/efs/) for the volumes * an [ECS cluster](https://docs.aws.amazon.com/AmazonECS/latest/userguide/clusters.html) with goerli and mainnet services and tasks. * [AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html) to run the containers The deployed stack can be monitored on the [AWS CloudFormation home page](https://eu-west-3.console.aws.amazon.com/cloudformation/home). All the `aws cloudformation` commands are available for managing the stack. Indeed, you can also use the `docker compose convert` tool to generate the corresponding Stack configuration and eventually use the aws cli instead: docker compose convert > stack.yaml aws cloudformation deploy --template-file stack.yml --stack-name starknet-nodes --capabilities CAPABILITY_IAM --profile docker-ecs Eventually, deploying the nodes requires: * to create a docker ecs context: `docker context create ecs * for example, `docker context create ecs starknet-ecs` * use the above created profile * and use it: `docker context use ` * or pass the `--context ` to every following commands * to execute `docker compose --project-name -f docker-compose.yml -f docs/aws/docker-compose.aws.yml up` * for example, `docker compose --project-name starknet-nodes -f docker-compose.yml -f docs/aws/docker-compose.aws.yml up` * ignore the `WARNING services.scale: unsupported attribute` * the `--project-name` option lets define the name of the stack but creates bugs for later use of `docker compose logs/down/etc.` so I don't recommend to use it for now. The created endpoints can be found in the ECS Cluster page: * Cluster > Services > Networking > DNS names or directly using docker compose: You can export those URLs easily for later user with the `--format json` option: docker compose ps --format json > nodes.json You can then check that the nodes are running using for example `curl`: curl `docker compose ps --format json | jq ".[0].Publishers[0].URL" -r` \ -H 'content-type: application/json' \ --data-raw '{"method":"starknet_chainId","jsonrpc":"2.0","params":[],"id":0}' \ --compressed | jq .result | xxd -rp # SN_GOERLI curl `docker compose ps --format json | jq ".[1].Publishers[0].URL" -r` \ -H 'content-type: application/json' \ --data-raw '{"method":"starknet_chainId","jsonrpc":"2.0","params":[],"id":0}' \ --compressed | jq .result | xxd -rp # SN_MAIN ### Troubleshooting The backup download time before the node actually starts can be quite long. If the service keeps restarting because of health check, you can extend the [health check grace period](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecs/update-service.html). In the following snippet, we put 3000s because it should be enough time to download ~100Gb at relatively low rate of ~33Mb/s: aws ecs list-services --cluster starknet-nodes | \ jq '.serviceArns[] | select (. | contains ("goerli") )' -r | cut -d '/' -f 3 | xargs -L1 \ aws ecs update-service \ --cluster starknet-nodes \ --health-check-grace-period-seconds 3000 \ --service $1 ### Monitoring The deployed stack can be monitored on the [AWS CloudFormation home page](https://eu-west-3.console.aws.amazon.com/cloudformation/home). All the `aws cloudformation` commands are available for managing the stack. The `docker compose logs` command will output the logs otherwise found in CloudWatch. ### Cleaning To delete your nodes and clean everything, just run: docker context use starknet-ecs docker compose down docker context use default docker context rm starknet-ecs aws efs describe-file-systems | jq ".FileSystems[].FileSystemId" | xargs -L1 aws efs delete-file-system --file-system-id $1 export POLICY_ARN=$(aws iam list-attached-group-policies --group-name docker-ecs-users | jq ".AttachedPolicies[0].PolicyArn" -r) aws iam detach-group-policy --group-nam docker-ecs-users --policy-arn $POLICY_ARN aws iam delete-policy --policy-arn $POLICY_ARN aws iam remove-user-from-group --group-name docker-ecs-users --user docker-ecs-user aws iam delete-group --group-name docker-ecs-users aws iam delete-access-key --user-name docker-ecs-user --access-key-id $(cat access_key.json | jq ".AccessKey.AccessKeyId" -r) aws iam delete-user --user-name docker-ecs-user Azure ----- ### Azure setup There is also a [Docker<>Azure ACI integration](https://docs.docker.com/cloud/aci-integration/) using docker context and the strategy is consequently similar to the one above. However, Azure made it easier by removing the need for using their CLI nor generating credentials. In short, the whole AWS CLI Setup section boils down to: However, it may be convenient to still install the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/?view=azure-cli-latest) for later manipulations. We are almost done with Azure config. You just need to [create a Resource group](https://portal.azure.com/?quickstart=true#create/Microsoft.ResourceGroup) for the nodes and chose an appropriate region for your application. If you installed the cli, just run: az configure --defaults location=francecentral az group create --name starknet-nodes ### Cloud deployment #### TL;DR copy/paste docker context create aci starknet-aci docker context use starknet-aci docker volume create goerli-data --storage-account starknetnodes docker volume create mainnet-data --storage-account starknetnodes curl https://raw.githubusercontent.com/ClementWalter/starknet-nodes/main/docker-compose.yml -o docker-compose.yml curl https://raw.githubusercontent.com/ClementWalter/starknet-nodes/main/docs/azure/docker-compose.azure.yml -o docker-compose.azure.yml docker compose -f docker-compose.yml -f docs/azure/docker-compose.azure.yml up #### Detailed story The `docker volume` command creates [File shares](https://learn.microsoft.com/en-us/azure/storage/files/storage-files-introduction). The [storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview) (`starknetnodes`) is created if it does not exist. These volumes are then used in the `docker-compose.azure.yml` azure overriding configuration. When running `docker compose up`, the `aci` context creates [Azure Container Instances](https://azure.microsoft.com/en-us/products/container-instances). Eventually, the commands are: * create a docker aci context: `docker context create aci * use the above created resource group * `docker context use ` * or pass the `--context ` to every following commands * create volumes: * `docker volume create goerli-data --storage-account ` * `docker volume create mainnet-data --storage-account ` * execute `docker compose --project-name -f docker-compose.yml -f docs/azure/docker-compose.azure.yml up` You can then retrieve the node urls using : You can then check that the node are running using curl: curl $(docker ps --format json | jq '.[] | select( .ID | contains("goerli") ) | .Ports[0]' -r | cut -d "-" -f 1) \ -H 'content-type: application/json' \ --data-raw '{"method":"starknet_chainId","jsonrpc":"2.0","params":[],"id":0}' \ --compressed | jq .result | xxd -rp # SN_GOERLI curl $(docker ps --format json | jq '.[] | select( .ID | contains("mainnet") ) | .Ports[0]' -r | cut -d "-" -f 1) \ -H 'content-type: application/json' \ --data-raw '{"method":"starknet_chainId","jsonrpc":"2.0","params":[],"id":0}' \ --compressed | jq .result | xxd -rp # SN_MAIN ### Monitoring You can find info about your deployment and containers in the Resource group's page. ### Cleaning To delete your nodes and clean everything, just run: docker context use starknet-aci docker compose down docker volume rm starknetnodes/goerli-data docker volume rm starknetnodes/mainnet-data docker context use default docker context rm starknet-aci GCP --- ### GCP setup There is no native Docker<>GCP integration and the usual way to deploy a containerized stack to GCP is to use the [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine). You first need to install the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install). If you have already a gcloud project, you can simply use it. Otherwise — or if you prefer — you need to create one: gcloud projects create gcloud config set project Then, we set for the project the region and zone we want to deploy in (see [regions/zones](https://cloud.google.com/compute/docs/regions-zones)). For example: gcloud config set compute/region europe-west1 gcloud config set compute/zone europe-west1-b ### Cloud deployment #### TL;DR copy/paste gcloud container clusters create starknet-nodes gcloud container clusters get-credentials starknet-nodes export $(xargs <.env) kubectl run starknet-mainnet \ --image=clementwalter/pathfinder-curl \ --port=9545 \ --env="PATHFINDER_ETHEREUM_API_URL=${PATHFINDER_ETHEREUM_API_URL_MAINNET}" \ --command -- /bin/bash -c 'curl https://pathfinder-starknet-node-backup.s3.eu-west-3.amazonaws.com/mainnet/mainnet.sqlite --output /usr/share/pathfinder/data/mainnet.sqlite && pathfinder' kubectl expose pod starknet-mainnet --port=9545 --target-port=9545 --type=LoadBalancer kubectl run starknet-goerli \ --image=clementwalter/pathfinder-curl \ --port=9545 \ --env="PATHFINDER_ETHEREUM_API_URL=${PATHFINDER_ETHEREUM_API_URL_GOERLI}" \ --command -- /bin/bash -c 'curl https://pathfinder-starknet-node-backup.s3.eu-west-3.amazonaws.com/goerli/goerli.sqlite --output /usr/share/pathfinder/data/goerli.sqlite && pathfinder' kubectl expose pod starknet-goerli --port=9545 --target-port=9545 --type=LoadBalancer #### Detailed story We first create a Kubernetes cluster on GCP. You can check all the options using `gcloud container clusters create --help`. Especially, by default, the cluster uses 3 nodes (param `--num-nodes=NUM_NODES; default=3`). Then we log into this cluster using the gcloud cli to have the `kubectl` command working directly on GCP. Depending on the versions of gcloud used, you may need to follow [this doc](https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke) to be able to login correctly. We also export all the env variables defined in a `.env` file (where we put the `PATHFINDER_ETHEREUM_API_URL_MAINNET` and `PATHFINDER_ETHEREUM_API_URL_GOERLI` definition). Eventually, `kubectl run` lets define the pods to run, and `kubectl expose` expose the pods (the _starknet nodes_). You can then retrieve the node urls using : kubectl get services and eventually check that the nodes are running using curl: curl $(kubectl get service starknet-goerli --output=json | jq ".status.loadBalancer.ingress[0].ip" -r):9545 \ -H 'content-type: application/json' \ --data-raw '{"method":"starknet_chainId","jsonrpc":"2.0","params":[],"id":0}' \ --compressed | jq .result | xxd -rp # SN_GOERLI curl $(kubectl get service starknet-mainnet --output=json | jq ".status.loadBalancer.ingress[0].ip" -r):9545 \ -H 'content-type: application/json' \ --data-raw '{"method":"starknet_chainId","jsonrpc":"2.0","params":[],"id":0}' \ --compressed | jq .result | xxd -rp # SN_MAIN ### Troubleshooting You can connect to your cluster using [Cloud Shell](https://console.cloud.google.com/cloudshelleditor), from browser or using the gcloud cli: gcloud cloud-shell ssh It may happen that the initial backup download fails. This does not prevent the node to start but can make its warmup time several days long. So if it happened, just `ssh` into the container and re-run the download command: kubectl exec -it starknet-goerli -c starknet-goerli -- sh curl https://pathfinder-starknet-node-backup.s3.eu-west-3.amazonaws.com/goerli/goerli.sqlite --output /usr/share/pathfinder/data/goerli.sqlite ### Monitoring You can find info about your deployment and containers in the [Kubernetes cluster page](https://console.cloud.google.com/kubernetes/list/overview) ### Cleaning To delete your nodes and clean everything, just run: kubectl delete pods --all kubectl delete services --all gcloud container clusters delete starknet-nodes gcloud projects delete starknet-nodes Conclusion ---------- While running a container is rather easy, running a prod-grade micro-service requires some more configurations (load balancing, auto restart, auto scaling, vpc, etc.). In this tutorial, I wanted to show how to deploy prod-grade Starknet nodes using managed services of the three main cloud providers. These guidelines may be taken "as is" to start and eventually updated depending on your needs. Please fell free to reach out to me directly on [Twitter](https://twitter.com/ClementWalter) or [Discord](https://discord.gg/BtuD5HguDX) should you have any comment or feedbacks! --- *Originally published on [Clemlaflemme](https://paragraph.com/@clemlaflemme/how-to-deploy-a-prod-grade-starknet-node)*