✨ TechOps News

• We’ve got out tickets to ETHDenver!

• We’re proud sponsors of Pragma Denver! You can win a free ticket.

• We published a new Blog post about our experience building Blockchain Keepers.

• We’ve launched our partnership program that helps us and our trusted partners expand our networks and reach even more people. If you’re interested in hearing more, come talk to us on Discord.

🚀 Terraform Cloud Workspaces Monitoring

During one of our team’s knowledge-sharing sessions, Sasha, from our engineering team, showed how he solved a long-lasting issue that many DevOps teams suffer from. We asked him some questions:

What issue were you trying to solve?

We observed issues with Terraform workspaces sometimes being incorrectly attached to GitHub or not updated to the master branch. This could be because someone is working on a test branch or because they forgot to switch back. At TechOps, where we bill by the ticket resolution time and we often pick up where someone left off, this causes delays and complications in software development, as engineers have to spend time rectifying these discrepancies instead of working on new tickets.

What solution did you find so far?

For now, I have created an automation that checks Terraform Cloud's workspaces connection to GitHub, its attachment to the master branch, and whether there are any errors or pending actions. It then creates a report that gets automatically posted to one of our internal Discord Channels at 08:00 am CET as we hand over from APAC to EU work hours.

Are you satisfied with the result?

It’s a step in the right direction and has already proven itself useful by reminding us of something we may have forgotten; or sparked a conversation during our Standup meetings about why something has changed. You can’t beat clear facts with human memory.

What would you like to build next?

I don’t know yet but we’re constantly looking out for ways of improving our efficiency. And it’s often small tools like this that bring us the most value because they take care of the mundane and allow us to focus on bigger and harder things.

If you would like to know more, you can reach Sasha or anyone from our engineering team through the TechOps Discord.

💌 Content worth sharing

• 2023 Anual Crypto Report

• Bitcoin ETF Approval by the SEC

• OpenAI’s custom GPT’s and GPT Store

• Kaspersky Employees IPhone’s Hack

• DevOps Memes

🙌 Meetups

You can find us at ETHDenver between the 24th of February and the 4th of March and at Pragma Denver on the 28th of February (Free Tickets). If you’re not able to make those events, we can meet you in Copenhagen, Melbourne, or online.

-

Thank you for staying connected with TechOps! If you have any questions or feedback or would like to explore collaborations, you can chat with us on Discord or any of our socials or simply reply to this email.

In the evolving landscape of blockchain technology and smart contracts development, one of the key components that plays a pivotal role is the "Keeper."  A Keeper is a programmable bot or script that automatically performs tasks on a Blockchain Network, such as executing smart contract functions when certain conditions are met, with the result of maintaining efficiency and stability in decentralized systems. At TechOps Services, crafting custom keepers is part of our regular day-to-day activity, providing essential bridges between on-chain and off-chain worlds. So far we have built many keepers with different uses. Here are some examples:

• Governance Keeper - executing the latest governance passed proposal at the proper time when it has to be executed

• Poker Keeper - posting the latest Oracle prices

• CDP Keeper - actively manage open Collateral Debt Positions

• Starknet Teleport Keeper - the bridge between Ethereum and Starknet networks

In this post, we describe one instance of this process when developing these specialized components for a customer. Here’s our first-hand experience resolving the challenges we had to tackle with this project. Let’s dive in!

Initiating the Keeper Development

The development process started with ensuring all necessary environment variables were set. Since we planned on Running this keeper in Docker, it meant this would be a trivial process. One of the first critical steps was establishing a connection to Etherscan to fetch ABIs (Application Binary Interfaces) for the smart contract in question. Choosing a dynamic retrieval of ABI, rather than static storage,  ensured that the keeper would always interact with the latest version of the smart contract, accommodating updates and changes without manual intervention, thereby enhancing flexibility and reducing maintenance effort, which streamlined our design and ensured efficiency right from the get-go.

Designing the Keeper Workflow

To begin we worked with smart contracts developers to clearly understand the desired flow keeper should follow. We usually end up with a very clear and easy to read flow diagram, we confirm it with all stakeholders and always keep it up to date. It would look a bit like this:

We need the keeper to operate in a continuous loop, constantly checking for new Ethereum block confirmations. After some monitoring and testing, the choice of a five-second sleep interval struck the right balance between effective monitoring and resource optimization, especially in the context of our Infura usage limitations. (Although, you will see in a moment that we opted for a more cost effective solution.)

Carefully managing our Infura request quota was essential for long-term operation considering resources and costs. We fine-tuned our polling intervals to maintain an effective balance between active monitoring and quota conservation, a critical aspect of a keeper's sustainable operation.

The Use of WebSockets with Infura

To maintain real-time updates, we considered using Websocket with Infura. This would have established a persistent connection for event consumption, reducing latency and network load, and enabling a more responsive and efficient operational model. However, it necessitated a shift to asynchronous programming, which we decided to reserve for a future iteration, considering our existing progress and time constraints.

Self-Hosted Ethereum Nodes as an alternative

Utilizing our own Ethereum nodes offered significant cost savings compared to relying solely on Infura, so we decided to go with our own nodes. This strategy enhanced our operational flexibility and scalability, allowing us to manage a higher volume of requests without the constraints of third-party quotas. The move towards self-hosting was a strategic decision to optimize for our use case with the added benefit of supporting a more decentralized blockchain infrastructure.

Embracing Python’s Multithreading Capabilities

A highlight of this project for us was leveraging the enhanced multi-threading capabilities in Python 3.11. This advancement brought us closer to the multi-threading efficiency of languages like Java, allowing each keeper to run in separate threads with improved performance. This makes sense because it can significantly improve the efficiency and responsiveness of the keeper. Multi-threading allows the keeper to perform multiple operations concurrently, such as monitoring different contracts or executing various tasks simultaneously. Enhanced multi-threading can lead to better resource utilization, faster execution, and overall improved performance of the keeper. As an example, this level of efficiency can be a huge win for DeFi when performing multiple market order liquidations.

Keeper Flow, Event Management and Exceptions in Logical Flow

Our keepers start by aggregating all relevant past events from the blockchain, processing each sequentially. This approach required an in-depth understanding of blockchain's unique operational environment, different from the more traditional architectures.

In traditional programming, exceptions are errors that occur during the execution of a program. Software Developers try to minimize exceptions because they often indicate something has gone wrong. However, in the context of keepers interacting with smart contracts, exceptions can be reinterpreted in a more nuanced way.

Intriguingly, smart contract exceptions in our keeper's flow weren't indicators of errors but signals of timing for function execution. For us, this novel use of exceptions exemplified our adaptive approach to utilizing blockchain's functionalities effectively.

Understanding the Nuances of Solidity Functions

Differentiating between static and dynamic functions in Solidity was vital. While static functions can't initiate transactions (and don’t cost any gas), dynamic functions can. This understanding was crucial for our interactions with the blockchain, ensuring cost-effectiveness and functional accuracy.

Rigorous Testing for Optimal Performance

Testing was a cornerstone of our development process. Tools like CodeCov helped us identify untested code segments, guiding our testing strategy. Mock calls were used to simulate blockchain interactions, ensuring the reliability and accuracy of our keepers.

Deployment and Monitoring Strategies

Leveraging Kubernetes, we were able to deploy our keeper service efficiently using GitHub Actions and managing them with Terraform for any future continuous deployments. However, the real challenge lies in monitoring our wallet activities on-chain. Having developed a monitoring Lambda function to track success (or failure) of all the transactions via Etherscan API, we ensured the operational integrity of this service by closely monitoring its behavior over time.

This venture into blockchain keeper development wasn't just about understanding blockchain intricacies; it was about integrating modern software engineering practices into the decentralized world. Through this post, we aim to provide some valuable insights for those looking to navigate similar paths in the blockchain and smart contract arena.

If you're considering developing your own keepers and need expert guidance or assistance, TechOps Services is here to help. Our experienced team is well-equipped, up to date and ready to support projects in this innovative and rapidly evolving field. Feel free to reach out to us for collaboration, advice, or services tailored to your blockchain development needs. info@techops.services

Summer.fi & TechOps: Business as usual

TechOps has been providing DevOps, 24/7 monitoring and IT support services to Summer.fi since August 2022. So when this project was brought up at one of the monthly planning meetings and a deadline was given, there was no delay in starting to research, plan, and brainstorm how to pull it off effectively.

Part of the plan was understanding what kind of downtime different services could accommodate, based on discussions with Summer.fi Engineers, and making sure that the end users wouldn’t see any impact.

Since communication is key to everything in life, TechOps began by listing what tasks and elements would need to be changed and collecting all insights into one place. In this case a collaborative document in the popular note-taking app Notion. This would allow both teams to collaborate and contribute, giving a clear overview and resolving any questions that came up.

Executing the Plan

Success would mean that everything was deployed and running without impacting the usability of the Summer.fi website and WebApp, welcoming users with a nice surprise the next time they returned to the site.

The roadmap was straightforward:

1. Identify all the elements that need changing, including text references, static assets, domains, URLs, and specific variables.

2. Plan how to pull off a smooth final switch. Which parts of the infrastructure don’t require migration and which parts can be migrated in stages.

3. Divide the process into smaller tasks that can be picked up by and assigned to anyone within TechOps.

4. Prepare a staging environment to test the production migration process and make sure the monitoring, alerting, and other testing systems run smoothly.

5. Thoroughly assess and test all aspects to prevent glitches or downtime.

Guiding the backend symphony was Cristi from the TechOps team, a DevOps engineer with over 10 years of experience who works from Northern Spain. He would work alongside TechOps’ Project manager Arti to handle the plan, steps to be taken, and their timelines for completion. Further discussions could take place on the already existing and regularly used communication channels that Summer.fi has with TechOps and, as usual, in person at the weekly check-in meetings between TechOps and Summer.fi.

Crossing the Finish Line

Anticipating challenges and potential snags had been foreseen with solutions proactively placed. With progressive switching of certain elements so that the front end wouldn’t be affected by changes made in the back end allowed the front end to be the last thing to be changed. That way, when it came to the transition, the TechOps team had all redirects in place and planned the final switch just before Summer.fi would make their press release and announcements.

The result was a stunning new look, celebrated for its design with a couple of new features to drive home the power of Summer.fi to users. All without missing a beat.

These kinds of projects become routine when everyone on the team has a minimum of 10 years of experience. So In the ever-shifting digital landscape, a project like Summer.fi's rebranding proves that simplicity in effective collaboration and precise execution doesn’t have to be overly complicated, exhausting, or feared. When you have top engineers available 24/7, teamwork, planning and dedication lead to seamless execution, leaving a lasting impression on the tech horizon for how modern work should be done.

Thank you for reading this use case story between TechOps Services and Summer.fi. If you would like to know more please visit techops.services and summer.fi.