Since 2020, TE Academy has established a reputation system leveraging on-chain proofs of knowledge and achievements. Moreover, by 2024, this system has evolved to enable reputation-based decision-making applications, such as TE Academy’s Reputation-based Weighted Voting, developed by Angela Kreitenweis, Octopus (8arms9brains), and Vitor Marthendal.

This article explores the motivation, development, and impact of the TE Academy’s NFT reputation system. We examine the growth of achievements in our community through detailed data analysis of Token Engineering Fundamentals (TEF), the bachelor-level, comprehensive curriculum in token engineering. We explore the evolution of student engagement and share trends we observed for the development of the discipline. Additionally, the article discusses the levels of experience achieved within TE Academy’s community, the role of NFTs in classifying expertise, and how we address challenges posed by imbalances in an early-stage ecosystem.

Through these insights, we present how TE Academy has built a meritocratic system that incentivizes collective progress and grants decision-making power to those with proven expertise. Through reputation-based voting, we enable our community members to actively steer the development of our sector, a governance model with enormous potential for any knowledge-driven ecosystem.

Developing a New Engineering Discipline

TE Academy, founded by Angela Kreitenweis in 2020, is an educational organization dedicated to advancing the field of token engineering—the design, verification, and optimization of cryptoeconomic, token-based systems. We view token systems as complex, emerging systems that must be designed with rigor. Drawing inspiration from the traditions of established engineering disciplines, we take responsibility for building public infrastructure and value engineering ethics.

Looking at other engineering fields, such as mechanical or chemical engineering, we see parallels in how an engineering discipline comes to life. It often begins with a need to solve actual, current challenges sparked by innovations in science and technology. From there, pioneers begin experimenting, gradually shaping a foundational understanding of the field until a cohesive body of knowledge emerges.

In the case of token engineering, this journey began in 2018 when a group of pioneers started hosting meetups to explore concepts related to tokens, and token-based economies. At that time, a few key whitepapers and articles were emerging, discussing the early ideas of what was then called “cryptoeconomics.” The ICO boom helped ignite the first wave of cryptoeconomic mechanism design, with notable works like the Bancor whitepaper, or Curation Markets driving early exploration. This pioneering work culminated in the first mention of “token engineering” by Trent McConaghy and the influential paper “Foundations of Cryptoeconomic Systems” by Shermin Voshmgir and Michael Zargham. This paper profoundly impacted the perception of token engineering as a cross-disciplinary, multi-faceted field. It also highlighted the glaring gap in higher education, as no university offers a curriculum to address this emerging challenge—something TE Academy sought to remedy.

As a common understanding began to take shape, the first curriculum concepts began to form - combining elements from systems engineering, mechanism design, mathematical optimization, and operations research, along with the idea of certifying students' knowledge.

For token engineering, our goal has been to build a digital system that reflects this journey. We started experimenting with POAPs (Proof of Attendance Protocol) and gradually developed a system of cryptographic proofs to verify achievements step-by-step.

In 2020, TE Academy offered its first online education course, Ecosystem Value Flows, laying the foundation for TE Academy’s future. In response to the growing demand for structured learning, TE Academy launched the comprehensive Token Engineering Fundamentals (TE Fundamentals) curriculum in 2022. Course authors Andrew Penland, Kris Paruch, Jessica Zartler, and Angela Kreitenweis took the lead in creating a bachelor-level program, which typically requires about four months of study time.

In addition, TE Academy offered complimentary live tracks and online events, providing space for specialization and live discussions. Since 2020, TE Academy has organized more than 100 online events and run ten cohort-based live programs. As of December 2024, over 5,000 students have enrolled. TE Academy has successfully onboarded more than 20,000 individuals globally, all benefiting from and contributing to the development and growth of token engineering.

Establishing an Immutable, Public Record

Learning Achievements: Empowering a Global Community

The most tangible achievement at TE Academy is the successful completion of a learning program. To earn a certificate, students undergo various methods of knowledge assessment, including online exams and graded homework. TE Academy ensures that each accomplishment is both meaningful and measurable, allowing us to track student's learning progress with credibility. However, TE Academy's vision extends far beyond: students play an active role in teaching token engineering. Through our Study Group Program, students lead study groups in 40+ locations across 14 languages, playing a vital part in establishing token engineering as a global discipline.

Knowledge Creation: Pioneering the Field

The second key contribution to developing token engineering is the creation of knowledge, alongside the formulation, challenge, and sharing of this body of work.

Creating TE Fundamentals as the first comprehensive curriculum in token engineering was a monumental effort. Furthermore, speakers across 100+ TE Academy events have shared invaluable insights in key sub-domains of token engineering such as tokenomics, mechanism design, risk management, and governance. Other contributions include lectures in cohort-based programs like Study Season 2024 and participation in the TE Academy Fellowship Prize, which recognizes exceptional talent and research in token engineering.

Classes of Achievements in Token Engineering

Analyzing these different contributions, we find four distinct categories of achievements:

• Learning Achievements: Successful completion of learning programs or courses

• Establishing Foundational Knowledge: Research and curriculum development

• Sharing Knowledge: Content creation, lectures, and talks

• Community Building: Peer-to-peer study groups, event organization, and support

Securing Legitimacy

Any reputation-based system requires tight verification to ensure that achievement proofs cannot be falsely minted. TE Academy has implemented various security measures to ensure that the system cannot be gamed:

TE Academy issues NFT certificates on Optimism mainnet (L2 chain), following the ERC-1155 standard. Only TE Academy's trusted wallet is authorized to mint new NFTs, and this is done exclusively based on meeting the rigorous passing requirements of the various courses. To address potential security issues, TE Academy has established policies for re-minting NFTs in case private keys are lost and for burning NFTs if they are maliciously used.

The NFTs are non-transferable, not tradeable, and not exchangeable. They are tied to the original recipient, and only the rightful owner of the achievement can control its visibility and gain utility, such as access or voting weight.

These features make the system highly sybil-resistant and thus, underpin the legitimacy of the reputation acquired by TE Academy NFTs.

Moreover, holders have full control over their NFTs and, by extension, their reputation. Unlike Web2 systems where users often have no visibility into or control over their reputation (or rating), TE Academy enables users to claim, reveal, or even burn their NFTs at will.

This approach aligns with the core principles of Web3, where individuals are the ultimate owners and stewards of their data.

TE Fundamentals Modules – A Deep Dive into the TE Academy Learning Journey

TE Academy NFTs were first minted in October 2022. Since then, achievements within the community have grown rapidly. The public record of proofs allows everyone to track them, and it enables gaining very granular insights into the evolution of the field, and the learning journey of students.

The TE Fundamentals (TEF) course is the cornerstone of TE Academy’s educational program and is structured across five learning modules. These modules are designed to equip students with the knowledge and skills needed to master the token engineering process. Students prove successful completion by passing online exams, that require a 96% score to pass.

Module Overview

• Module 1: Introduction to Token Engineering
  In This module serves as the foundation of the course, providing students with an introduction to the fundamental concepts and definitions of token engineering. It highlights five articles and papers that have shaped the field. Each paper is paired with an introductory video where the authors discuss their motivations and reflections on the concepts. This module requires no prior knowledge, making it an essential onboarding for every student.

• Module 2: The Discovery Phase of Token Engineering
  In this module, we focus on the discovery phase of the token engineering design process. Students learn how to explore and understand the system they aim to create, forming the "problem entity" that will guide the design and modeling of the token system.

• Module 3: Design Phase – Mathematical Modeling and Design
  Here, students develop the mathematical models for the systems introduced in the previous module. Using the illustrative example of Uniswap, we walk students through the token engineering process and provide context for how a system might evolve in response to real-world events.

• Module 4: Deployment Phase – Programming the Digital Twin
  This Module introduces the final phase of the token engineering process—model programming. Students learn how to code a model that replicates the intended system, known as the "digital twin." Once this model is validated and verified, it is ready to be tested and simulated.

• Module 5: Token-Based Governance
  In the final module, we explore the landscape of token-based governance. A multidisciplinary approach introduces key governance concepts, offering students the context and framework necessary to analyze, explore, and initiate Web3 governance systems and processes.

Evolution of Achievements

In the following section, we examine the evolution of achievements and the conclusions we can draw regarding the current state of the token engineering domain.

Growth of Passed Exams Across All Modules

This first chart illustrates the total growth of passed exams across all five TEF modules, from the program’s launch in October 2022 to October 2024. The data shows a clear upward trend, indicating a growing interest and participation in token engineering. This aligns with the expansion of our community on social media, as well as the continued increase in visibility through media coverage and at conferences. In total, 5,027 students had enrolled at TE Academy, and 1,338 exams were successfully passed across all modules by the end of October 2024.

NFTs Minted by Module

The second chart presents the total number of NFTs minted by module. As expected, Module 1 is the most popular overall, given that it serves as the starting point for all students. This module sets the foundation for the rest of the course, which explains its high completion rate.

However, Modules 4 and 5 exhibit a surprising trend—both have nearly identical numbers of NFTs minted, with Module 5 even slightly surpassing Module 4. This raises interesting questions about how students are approaching their learning journey: Are they following a sequential path through the modules? Or are they selecting modules based on their specific interests, background knowledge, or what they assume is most useful for their career in token engineering?

Evolution of Passed Exams by Quarter and Module

The third chart presents the distribution of passed exams across modules over time. We expected a stable cohort of students steadily advancing from Module 1 to Module 5, which would result in a decrease in Module 1 exams over time. Instead, the data shows that the share of Module 1 passed exams remains high over time. Meanwhile, a relatively consistent share of students complete Modules 2–5. This suggests that the TE Fundamentals has managed to attract a continuous inflow of new students throughout its lifetime. Moreover, TEF attracts a diverse range of learners, many of whom may not be progressing linearly through all five modules but rather engaging with the content in a way that aligns with their interests and goals.

Connections between NFTs earned for different TEF Modules

We continue exploring diversity by analyzing the connection between NFTs earned for different TEF modules. This data reveals interesting patterns in how students engage with the TEF curriculum.

First, examining the linear progression from Module 1 to Module 5, we observe that only a fraction of students successfully acquire NFTs for subsequent modules. This is expected in a program where content builds progressively, requiring increasing levels of commitment and skill at each stage.

Vice versa, a vast majority of students holding NFTs for any given module have also earned NFTs for prior modules. For example:

• Among Module 2 NFT holders, 93.3% also hold a Module 1 NFT.

• Approximately 95% of Module 3 NFT holders also possess Module 1 or 2 NFTs.

However, Module 5 stands out as a notable exception:

• Only 91.2% of Module 5 NFT holders also acquired a Module 3 NFT.

• Even fewer, 82.5%, of Module 5 NFT holders earned a Module 4 NFT.

To provide context, let’s revisit the content of the final three TEF modules:

• Module 3: Translates system requirements into mathematical models.

• Module 4: Verifies mechanism designs through a Python digital twin.

• Module 5: Introduces token-based governance principles.

The findings suggest a divide in the student body’s focus. While many students complete the foundational modules (1-3), there appears to be a divergence in paths for the final two modules:

• Some students bypass (or fail to pass) Module 3 and Module 4 in favor of governance topics in Module 5.

• Conversely, a significant portion of Module 4 graduates does proceed to Module 5, likely motivated by the proximity to earning full TE Fundamentals graduate status.

Analyzing student and NFT data allows us to draw meaningful conclusions about how students approach studying token engineering. Most notably, the data highlights diverse learning journeys within the community, with students selecting modules based on their interests and career goals.

At the same time, these dynamic learning patterns reflect the adaptability of the TE Academy’s curriculum, which successfully caters to a variety of learners at different stages of their token engineering journey.

In the next section, we explore the implications of this learning diversity further focusing on different levels of experience.

Levels of Expertise: Mapping the Community’s State

For monitoring the community’s state, we can categorize NFTs by levels of expertise:

• Pioneers: NFTs acknowledging achievements in mapping out the discipline with fundamental research and scientific work

• Experts: NFTs for token engineering practitioners and course authors who established curriculums building on the pioneer’s work and practical experience.

• Graduates: NFTs representing those who have successfully completed all five modules in TE Fundamentals

• Students: NFTs for successfully studying token engineering or attending TE Academy events

Naturally, in the early stage of the discipline, the system shows an imbalance, with a large number of students compared to a smaller number of experts. As of June 19, 2024, 871 student NFTs represent individuals who have not yet completed the TE Fundamentals course. Meanwhile, 550 graduate NFTs have proven the completion of all TE Fundamentals modules, and only 45 NFTs have been issued to experts—for contributions such as offering classes or creating learning materials. Note that by the date of this analysis, none of the pioneer NFTs had been minted.

In the following section, we explore new applications enabled by TE Academy’s reputation system.

Outlook: Building Applications on Top of a Reputation System

The NFT system established by TE Academy serves as a foundational public record of achievements in token engineering, enabling several key applications:

• Tracking individual achievements and progress over time

• Observing collective progress and the field’s evolution

• Rewarding contributions to foundational research, knowledge creation, and dissemination

• Assigning decision-making rights and weight to knowledge and achievements

This ability to measure the growth of knowledge and achievements within our ecosystem provides a solid foundation for building a meritocratic system. However, to fully realize its potential—particularly in empowering decision-making for those with outstanding achievements—we must address a critical challenge: in the early stages of an ecosystem, experts are scarce. As highlighted in the last chart, a 1-token-1-vote decision-making system risks the sheer number of student-held NFTs outvoting the smaller group of seasoned experts. For decisions where expertise is crucial, we need more advanced mechanisms to ensure that expert opinions are appropriately weighted

In the next article of this series, we introduce Dynamic Network-weight Scaling (DNS), a weighting mechanism built on TE Academy’s NFT reputation system. DNS assigns different voting weights to NFTs based on the holder’s achievements and dynamically adapts to the evolving expertise accumulated within the system. We first developed and applied this mechanism to select the winner of TE Academy’s inaugural $10K Fellowship Prize in July 2024.

Resources

• TE Academy’s research initiative on reputation systems is led by Angela Kreitenweis, Octopus (8arms9brains), and Vitor Marthendal. We kicked off developing Dynamic Network-weight Scaling with TE Academy Students during the 2024 Study Season program, where we went from defining the design goals of the mechanism all the way to its implementation and the actual voting. The session recordings of this program are available here.

• TE Academy NFT Smart Contracts (OP Mainnet, July 2024, note that while TE Academy started with a third-party NFT solution, we replaced all of our existing NFTs with our smart contract in June 2024)

• TE Academy NFT Registry on OpenSea

Acknowledgements

Big thanks to all the students in the 2024 Study Season Reputation-based Weighted Voting track—especially @joanbp, @FtheDev, @jade, @jonas, and @OneLV—for exceptional contributions to Reputation-based Voting.

Shoutout to OneLV, Kaidlyne Goepfrich, and Chima Apka for the helpful feedback on this article!

Read more:

• Part I: Hello, cadCAD GPT! Requirements and conceptual design of LLMs to support token system simulations

• Part II: This is me, cadCAD GPT! A deep-dive into cadCAD GPT’s powerful, customizable components

cadCAD GPT is a new LLM agent framework to support token systems simulations, built by Token Engineering Academy’s Rohan Mehta and Angela Kreitenweis. cadCAD GPT can be used on top of any Python model following the cadCAD/radCAD model structure. It is made for token engineers, and their project stakeholders. By offering a natural language interface, it makes it easy to iterate and establish simulation routines, freeing up bandwidth for reasoning about the system design.

Following the core design requirements introduced in Part I of this series, we demonstrate how cadCAD GPT can support typical token engineering tasks. For demonstration purposes, we go over a token sales use case first and showcase the same types of queries for a Lotka–Volterra predator–prey model (system dynamics model).

Accessing Memory

As outlined in Part II, Memory is the capability of agents to access sources of knowledge relevant to a simulation task. Examples in token engineering include on-chain transaction data, token fundraising data, or the documentation of a cadCAD model. To learn more about the concept of Memory in cadCAD GPT, check out Part II.

Accessing tabular, numerical data

In a token sales use case, the goal is to design an attractive offer to token investors. In this context, it's crucial to consider current market conditions and examine the accomplishments of comparable projects within the same sector and sales rounds. cadCAD GPT can process such benchmarking data and create token sales offers.

In our example, we search for data in an external database. We ask cadCAD GPT to find benchmarking data (pink).

cadCAD GPT provides the following result:

Here’s what happens behind the scenes: The cadCAD GPT chatbot receives the user query and triggers the Planner Agent, who submits the plan, including the tool to use (long_term_memory). Then, the Executor Agent reasons about this plan, runs long_term_memory, and includes the SQL query to filter the database according to the user query.

Accessing Memory with cadCAD GPT is easy. Search for data, run API calls, and access SQL tables or CSV files hosted anywhere. All you have to do is ask cadCAD GPT in natural language.

Moreover, token engineers can customize cadCAD GPT and add tools according to their needs. It only requires a Python function, the data source, and a well-formulated docstring. cadCAD GPT agents use these docstrings to figure out when to use these tools and how to use them. Learn more about adding tools and memory to cadCAD GPT in Part II of this series.

Accessing semantically searchable data

One of the first steps in a simulation project is familiarizing yourself with the token system model in use, its purpose, the metrics it can observe, or any other information included in the model documentation. In the following example, we explore a predator-prey model. We ask cadCAD GPT for the assumptions in the model.

cadCAD provides the following answer:

Here’s what happens behind the scenes: cadCAD GPT has access to a Markdown file in the model Github. The tool model_documentation makes the document available to agents and creates a semantically searchable vector database to allow LLMs to search for relevant information based on the user question. cadCAD GPT can process any text source, like PDF files or webpages. We equipped cadCAD GPT with a Retrieval Augmented Generation framework for semantic searching and enable token engineers to specify data sources for a maximum level of control over cadCAD GPT’s knowledge. Learn more about the concept of Memory in Part II.

In a final example, we ask for the parameters and metrics in the model. The screenshot below shows the full flow of user input and cadCAD GPT output:

Fetching simulation output

In a simulation project, token engineers typically iterate on a design and create a range of variants to find the best solution. This often requires numerous simulation runs with different parameter settings and simulation outputs.In the context of a token sale, designing a token supply scheme with total supply and supply unlocked over time is a typical token engineering task. cadCAD GPT makes it easy to keep track of current values by simply querying it in human language.

Equivalently, we can ask for the current values in the predator-prey model.

Behind the scenes, cadCAD GPT’s Planner agent picks the model_info tool to access the current parameter settings and simulation output (available via the cadCAD model objects). Today, cadCAD GPT allows access to current values only – which are updated in every simulation run. In future versions, we plan to include tools for version control and comparing simulation results easily. Learn more in Part II.

Analyzing and changing parameters

While the examples above demonstrate cadCAD GPT’s capabilities in simple queries, let’s now look at more complex tasks that demand a sequence of intermediate steps to be executed.

Improving a token sales offer

We start with the case of a token sale again. In Part I, we introduce the iterative process of designing a token sale offer, searching for a solution that is considered fair to any investor group, like participants in a private or public sale, and the startup team. Part of this offer is vesting terms, which are usually time-based to unlock tokens for particular stakeholders at a certain point in time. In negotiations, investors might ask for different distribution terms, and token engineers must analyze the consequences, e.g., the need for a larger total supply. cadCAD GPT can support this process. Let’s see if cadCAD GPT can find the following values for us:

In this case, the Planner Agent creates a task list with multiple ordered steps, the tools to pick from cadCAD GPT’s Toolkit, and the arguments to calculate the results - per step.

The Executor Agent takes these instructions and reruns the simulation. It loops over Thoughts, Actions, and Observations and runs data analysis via the analysis_agent, utilizing Python Pandas methods. The analysis agent is available in cadCAD GPT’s open-source Toolkit.

Providing simulation output as charts

On top of the features presented above, cadCAD GPT can plot charts and visualize simulation outcomes. Let’s assume we need a diagram showing the evolution of the prey population based on a new reproduction rate.

We ask cadCAD GPT:

cadCAD GPT provides the result in a graphical form by picking the right tool - plotter - without further instructions. The Planner Agent’s task list and the Executor agent’s messages are shown below:

Summary

cadCAD GPT is able to support a wide array of use cases in a token engineering context. Its intelligent agents are not limited to answer questions but reason about a system design, seek out the best parameter settings, conduct analyses, and visualize results. Remarkably, without the need for explicit instructions, they handle non-deterministic task sequencing.

Via Toolkits, cadCAD GPT can be equipped with the most cutting-edge data analysis, diagramming, and machine-learning Python libraries available today. Toolkits are modular and can be customized and expanded to serve any simulation project’s needs. Similarly, cadCAD GPT can access information and data in diverse formats stored in Memory, offering full control over which sources are incorporated in the solution of a given task.

Open research questions

This first release of cadCAD GPT marks an important milestone in developing AI-powered token engineering tools. When navigating the early stage of LLMs applied to the needs of our discipline, many intriguing questions emerge.

Open research questions include:

• Incorporating novel workflows: Data science workflows sometimes involve re-prioritizing tasks based on observations from the tool outputs. The Executor agent currently does not change its tracks once the Planner Agent makes the initial plan list. To solve this, we could get the Planner agent to flag certain tasks as required while leaving others to the discretion of the Executor to re-prioritize based on answers received by the tools. We would like to know if this can be done in a generalized way without introducing failure modes.

• Adding short-term memory to the Planner Agent: The Planner agent currently does not remember previous conversations; we need to add a dynamic message history to the Planner Agent for it to remember important details of earlier conversations in the same session. Since we will never have an infinite context window with any LLM, we must delete message history. What framework can we use to decide on the processing of conversation history? Ideas from cognitive architecture research should aid us in this aspect.

• Retrieval Augmented Generation to improve few-shot prompts: As tools increase in number and complexity, it will become increasingly likely that the plans developed by the Planner Agent may miss some steps or choose the wrong tools. Few-shot prompting is the best way to solve it, but there are only so many examples we can give before we run out of context length. We wonder if building dynamic system prompts with a retrieval augmented generation over a dataset of correct examples will be the right approach for the future.

cadCAD GPT Demo

cadCAD GPT will be available on Thursday, Nov 30, 3:00pm UTC
Sign up for the demo and be the first to get access!

Acknowlegements

cadCAD GPT was kickstarted by funding received from Token Engineering Commons. We thank the TE Commons community, and Gideon Rosenblatt in particular, who encouraged us to embark on this exciting journey. Big thank you to our advisors Roderick McKinley, Richard Blythman, and Robert Koschig for ongoing support and feedback. Shoutout to Dr. Achim Struve, Dimitrios Chatzianagnostou, Stephanie Tramicheck, Ivan Bermejo, Rohan Sundar, and Lukasz Szymanski for the most valuable alpha user feedback and insights, and Kaidlyne Neukam for her tireless support in publishing this work.

Links

The token sales spreadsheet model that informed our token sales experiments is available online, along with a comprehensive online course by Roderick McKinley.

TE Academy is the home for the token engineering community. Learn how to design token systems with rigor and responsibility! Sign up for our newsletter to receive the latest token engineering trends, tools, job offers and ecosystem news.

For further reading:

• Part I: Hello, cadCAD GPT! Requirements and conceptual design of LLMs to support token system simulations

• Part III: Let’s chat! Experiments and further development of cadCAD GPT

The Key Components of cadCAD GPT

Token systems are complex dynamical systems with emerging properties. Evaluating, stress-testing, and enhancing these systems' designs requires running extensive system simulations. Token engineers have to iterate on experiments, parameter settings, and a multitude of intermediary results to consider before arriving at a definitive conclusion. cadCAD GPT supports and structures this process to allow token engineers to concentrate their efforts on formulating the essential questions and making informed decisions. Moreover, a natural language interface enables stakeholders without a token engineering background to interact with system models. cadCAD GPT transcends today’s boundaries, empowering a wide spectrum of stakeholder groups to utilize simulations in their decision-making processes.

In this article, we introduce the cadCAD GPT components and how they function. We demonstrate how cadCAD GPT agents are constructed, how to connect cadCAD GPT with any radCAD or cadCAD model, and how the framework can be further expanded and customized.

At its core, cadCAD GPT takes three steps to get to a simulation result:

• The cadcad_gpt chatbot takes in a user query and passes it to the Planner Agent.

• The Planner Agent returns a task list and the tools and information to use in the correct order to answer the user query.

• The Executor Agent loops through each item of the task list one by one and has access to tools and memory. It reasons what inputs the tools would need based on the user query and context and remembers the results to get to the next task.

This orchestration core of cadCAD GPT is equipped with modular, customizable toolkits and memory. It grants cadCAD GPT access to today’s most powerful data analysis and machine learning libraries and external data to simulate token systems.

The cadCAD GPT Chatbot (cadcad_gpt.py)

cadCAD_GPT receives the user input and prints the user output. It initiates the setup of all agents, tools, and memory items. Moreover, it orchestrates all the agents, tools, and memories. While the Planner and Executor Agents process the steps toward the outcome, cadCAD_GPT collects all communication between agents and makes it available to the user.

To initiate the chatbot, we pass an Open AI API key and the cadCAD or radCAD model objects model, simulation, and experiment. Additionally, any information useful to run the simulations can be added as docs (see section Memory below).

In the current version, with no graphical UI available yet, any question to cadCAD GPT is prompted as a string to cadCAD GPT.

Generally, cadCAD GPT user output collects all communication between the Planner and Executor agents. In the current version’s default mode, the user output includes the Planner Agent’s task list and the Executor Agent’s thoughts, actions, and observations. However, cadCAD GPT user outputs can be customized to include or exclude any messages between agents. Our framework allows developers to include all messages to easily verify the results – or display only selected messages for better UX.

Planner Agent (agents.py)

After any user input, cadCAD GPT triggers the Planner Agent to process the user question. The Planner agent breaks down the user question into the low-level steps required to achieve the goal. It can reason about the user query and plan the steps needed to accomplish the task. It then describes each step with the tool to use and the context to pass to the tool, to finally provide the task list as an output in a parseable format.What’s unique about the Planner Agent is that it does not follow predefined workflows since it uses an LLM to make contextual decisions at each step, which enables it to solve non-deterministic workflows.

The Planner Agent runs on OpenAI’s gpt-3.5-turbo with a system prompt that includes instructions about how to reason about a user query. The prompt is inspired by ideas from Chain-of-Thought Reasoning and instructs the LLM to generate intermediate reasoning steps to improve contextual reasoning abilities. Additionally, the Planner Agent can access a dynamic list of tool names and descriptions available (see Toolkit below). Finally, we fine-tuned the Planner Agent’s behavior with few-shot examples to optimize the task list creation and further processing by the Executor agent. This part is a key lever to optimize agents for particular use cases and further tweak how they respond to user prompts. It should only be touched with a sufficient level of experience in AI agent design.

Once the Planner Agent finishes creating the task list, cadcad_gpt parses the plan into a Python list and passes it to the Executor Agent one task at a time.

Executor Agent (agents.py)

The Executor Agent specializes in working with tools and memory. When receiving a task, it uses an LLM to figure out the tool to use and the correct arguments to pass to complete the task.

The Executor Agent’s role is to have a Thought and reason about the task to accomplish. Second, it selects an Action. This step makes a call to OpenAI and generates a JSON, which includes the name of the function and the arguments to pass to the function. Then, the Executor Agent observes the result. It executes the function with the given arguments on a Python shell and reasons about the results. These Thought-Action-Observation loops are inspired by the ReAct framework. Along with Program-Aided Language Modelling techniques, these approaches show remarkably good results in diverse language reasoning, symbolic reasoning and decision-making tasks.

Finally, the Executor Agent saves the chat history in its short-term memory to aid in contextual decision-making for subsequent steps in the task list.

The example above shows a complete set of cadCAD GPT messages from user input and to user output (default mode) going over a multi-step task list. cadCAD GPT prints first the Planner Agent’s task list. Then, it displays Thought, Action, and Observation loops created by the Executor Agent. Finally, since in our example, the user asked for a plot, the Executor Agent provides a diagram using the plotter tool (see Toolkit below).

The Executor Agent uses OpenAI’s gpt-3.5-turbo-0613, equipped to run Python functions while solving a task called Function Calling (see Toolkit below). An initial system prompt includes basic instructions and information about the parameters of the simulation model.

Toolkit (toolkit.py)

One of cadCAD GPT’s most powerful features are Toolkits*,* made accessible to agents via Function Calling (see below). cadCAD GPT agents can select and run Python tools and, thus, enable natural language access to powerful data analysis and machine learning libraries. cadCAD GPT agents can interact with cadCAD/radCAD models to run experiments, analyze results, visualize plots, access APIs, and more. cadCAD GPT collects all tools (and memory accessible via tools) in a Toolkit class. The Planner Agent reviews the toolkit descriptions to find suitable tools for the tasklist and then hands them to the Executor Agent to parse arguments and execute.

Function Calling

To unlock OpenAI’s function calling capabilities, all tools in cadCAD GPT have to include a description marked as docstrings with """triple double quotes""", following the PEP 257 – Docstring Conventions.

With these descriptions included, cadCAD_GPT automatically generates a function calling schema for all tools in the toolkit and makes it readily available to both the Planner and the Executor Agent.

Modular Toolkits

We aim to add new tools and memory access via tools to cadCAD GPT continuously. cadCAD GPT makes it easy to expand toolkits and add information and data sources. All Python functions are available to the cadCAD GPT agents as soon as they are added to the toolkit class - provided they contain the necessary elements (see “Function Calling” above). This flexibility makes cadCAD GPT a powerful framework for any project’s specific simulation needs. At publishing date, the cadCAD GPT Toolkit includes the following tools:

• model_info() : returns current values of the radCAD/cadCAD model objects’ parameters

• change_param() : changes the parameter of the cadCAD simulation and runs the simulation to update the dataframe

• analysis_agent : A specialized agent stored as a tool. Builds and executes Python pandas queries to analyze the dataframe.

• model_documentation() : Allows natural language Question Answering for the model documentation using a Retrieval Augmented Generation approach. This is an example of how long-term memory is made accessible via the respective tool.

• plotter() : Plots any column of the dataframe.

Memory (memories.py)

Memory is information available to cadCAD GPT. An LLM off the shelf, like OpenAI’s GPT models, is not trained on particular token engineering domain knowledge, or this information might not be available publicly. Thus, we have to make this knowledge accessible via Memory. This approach enables cadCAD GPT to access knowledge. Moreover, token engineers can instruct cadGPT to include or exclude particular data and information. This enables factual consistency, improves the reliability of the generated responses, and helps to mitigate the problem of LLM Hallucinations.

Memory examples include data stored on external servers, knowledge about the industry in a digital book PDF or the context of a task stored in cadCAD GPT’s message history.In our framework, we conceptualize memories into two categories: Short-term memory and long-term memory.

Long-term Memory

Long-term memory is information available to cadCAD GPT at any point in time. Long-term memory does not have to be included in a user input. Both the Planner and Executor agent can access long-term memory in a controllable, verifiable way. See below how different types of long-term information can be made accessible to LLM agents.

Long-term memory via a semantically searchable vector database

Not all user questions to cadCAD GPT might require running a simulation. Users can ask cadCAD GPT about the purpose of a model or the definition of certain terms in an output received.

In the example below, a user asks cadCAD GPT questions about the radCAD/cadCAD model itself, which is made available in the documentation file (cadcad_gpt = CadCAD_GPT(openai_key, model, simulation, experiment, docs). Best practices for building such a model doc file recommend including an introduction to the model, the assumptions of the model, the set of parameters and metrics to observe, and an explanation of the state update logic.

We take this documentation file and split it up into chunks to embed them in a vector database. This allows cadCAD GPT to semantically search the file and answer questions via a Retrieval Augmented Generation setup. Similarly, any text-based data can be made available to agents via semantically searchable vector databases. When executing the task list, the Executor agents utilize the tool model_documentation to fetch the information needed for the user output.

Long-term memory as numerical, tabular data

Another type of long-term memory is the output of simulation runs, blockchain transaction data, or any type of table-format, numerical or text data. The example below shows how the current parameter value “prey death rate” can be extracted from a predator and prey simulation by prompting cadCAD GPT in natural language. The Planner Agent selects the tool “model_info” to fetch the parameter value, and the Executor Agents return the results.

Short-term memory

Short-term memory is information cadCAD GPT updates dynamically before making it available to agents. In default mode, short-term memory is available to agents only while processing the current user input.

cadCAD GPT stores the message history between the user, the Planner Agent, and the Executor Agent to enable optimal, contextual decisions on task list execution in short-term memory. In cadCAD GPT’s default mode, we delete this message history periodically (see Executor Agent above). This solution optimizes agents’ attention to the task and fits OpenAI's context window limitations in OpenAI gpt-3.5-turbo-0613.

cadCAD GPT Further Expansions

Due to its highly modular design, cadCAD GPT can be customized and expanded according to any project’s individual simulation needs. We aim to develop the following expansions with cadCAD GPT alpha users and contributors:

• Adding new tools:

  • Integrate Python parameter sweeps and A/B testing tools

  • Auto-create a standard cadCAD/radCAD model documentation to allow users to ask questions about the model

• Optimizing short-term memory

  • Build better logic for wiping and updating short-term memory to improve contextual decision-making for both Planner and Executor Agents

• Converting short-term memory to long-term memory

  • Allow agents to remember important aspects of a conversation beyond a single user prompt

  • Version control of model parameter settings and experiments

  • Enable to undo tasks

• Adding Retrieval Augmented Generation to tune Planner Agent’s planning abilities further

  • Build a repository of user inputs and their expected task lists in a token engineering context, which can be dynamically fetched into the system prompt, provides the planner agent with better few-shot examples (see Planner Agent above).

  • Fine-tune the Planner Agent’s model with a big enough token engineering user input and task list repository

• Newer/alternative LLMs

  • Update cadCAD GPT to use OpenAI GPT-4 Turbo  with new features to call models and tools, a 128K context window, and more

  • Test alternative open-source LLMs

If you are interested in contributing to cadCAD GPT’s further development, sign up for the demo below or drop us a line at contact@tokenengineering.net.

Summary

cadCAD GPT is designed to harness the immense potential of Large Language Models (LLMs) for supporting token system simulations. This article introduces the inner workings of cadCAD GPT, and its ability to interact with Python models following to the cadCAD/radCAD model structure. Through the concept of Toolkits, cadCAD GPT provides access to cutting-edge data analysis and machine learning libraries, enabling the utilization of data in diverse formats stored in Memory. A notable highlight is cadCAD GPT's capability to handle non-deterministic task sequencing, while empowering its human collaborators to oversee and track the workflow of AI agents, ensuring the generation of verifiable and reproducible results.

With cadCAD GPT, token engineers gain the tools needed to explore and optimize the design of complex systems with the support of LLM agents, ultimately contributing to the evolution of token engineering practices.

cadCAD GPT Demo

cadCAD GPT will be available on Thursday, Nov 30, 3:00pm UTC
Sign up for the demo and be the first to get access!

Acknowledgements

cadCAD GPT was kickstarted by funding received from Token Engineering Commons. We thank the TE Commons community, and Gideon Rosenblatt in particular, who encouraged us to embark on this exciting journey. Big thank you to our advisors Roderick McKinley, Richard Blythman, and Robert Koschig for ongoing support and feedback. Shoutout to Dr. Achim Struve, Dimitrios Chatzianagnostou, Stephanie Tramicheck, Ivan Bermejo, Rohan Sundar, and Lukasz Szymanski for the most valuable alpha user feedback and insights, and Kaidlyne Neukam for her tireless support in publishing this work.

Links

The token sales spreadsheet model that informed our token sales experiments is available online, along with a comprehensive online course by Roderick McKinley.

TE Academy is the home for the token engineering community. Learn how to design token systems with rigor and responsibility! Sign up for our newsletter to receive the latest token engineering trends, tools, job offers and ecosystem news.

Read more:

• Part II: This is me, cadCAD GPT! A deep-dive into cadCAD GPT’s powerful, customizable components

• Part III: Let’s chat! Experiments and further development of cadCAD GPT

Introduction

Large Language Models (LLMs), like OpenAI GPT-3 and its successors, have demonstrated remarkable potential in supporting humans in solving complex tasks. Token engineers invest substantial time and effort in constructing models and running system simulations to find an optimal token design. Imagine, token engineers could access token system simulations just by prompting a Python model with a natural language request.

And imagine that AI agents could not only answer questions but change parameter settings, analyze, and visualize results.

In fact, AI agents can do this - and much more!

cadCAD GPT is an open-source LLM agents framework to support token systems simulations. cadCAD GPT can be used on top of any Python model following the cadCAD/radCAD model structure. Via Python functions, it can integrate today’s most powerful data analysis and machine learning libraries (Tools) and can access data in various formats (Memory). Finally, cadCAD GPT allows users to control and track the agents’ workflow for verifiable and reproducible results.

In this article, we derive requirements for LLM agents in a token engineering context and present the generalized framework. Part II provides a detailed overview of the open-source cadCAD GPT components and shows how to apply and further develop cadCAD GPT to make it cater to any complex system modeling task. In Part III, we present a range of experiments demonstrating interactions with cadCAD GPT in two simulation use cases.

The Token Engineering Design Process

Token engineering is the design, verification, and optimization of token-based crypto-economic systems. The token engineering design process involves several steps, from discovering and collecting system requirements to designing incentive mechanisms to implementing the algorithms in smart contracts. In the design phase, token engineers test if the desired properties of the system hold under a certain parameter setting, stress test the system, or run parameter sweeps to find the optimal setting. Here, a Python model of the token system serves as a digital twin to run these experiments.

The modeling and simulations phase requires a lot of experience and includes the following steps:

a) Collect the questions to ask and prioritize;b) Derive the metrics to answer these questions and build a model of the token system; c) Verify the model ("did we build the model right?") and validate the model ("did we build the right model to represent reality to answer our questions?") Only then are we ready to d) start running experiments, collect insights, draw conclusions, and e) take decisions.

With cadCAD GPT, we focus on step d) to help token engineers and their teams iterate on the design, become faster in deriving conclusions, and, ultimately, gain confidence in making decisions.

Use Case Example: Building up a Token Sale Proposal

Designing a token sale is a challenge that every token engineer is tasked with at some point. The main goal of a token sale is to raise the funding needed to develop a protocol or application by selling tokens to investors. Tokens are typically sold in several sales rounds to different stakeholder groups. They are usually not immediately distributed when sold, though. Instead, a vesting schedule defines unlock dates and rates for investor groups.

In a token sale round, token issuers and investors aim to reach equilibrium via several iterations of one-to-one negotiations. A token price is typically a function of the amount a startup aims to raise and the token supply, or the token amount sold, respectively. As such, it is a simple calculation. However, any investor group seeks to achieve a fair deal compared to any other investor group. Thus, the number of tokens and conditions sold in prior and potential future rounds impact any token sales offer.

The fact that a token sales offer is influenced a) by the parameters supply and price in a particular round and b) by the same parameters across all other rounds, and c) by comparing other token sale events in the same sector, makes the optimization of a token sale offer a multidimensional optimization problem. We see such multidimensional optimization problems most frequently in token engineering. Designing a token sales offer requires a comparably simple model since most values (such as the token price or supply) are deterministic.

Token engineers build more complex models simulating the interplay between various token mechanisms and uncontrollable external factors such as token market price or human behavior. We hypothesize that LLM agents can provide massive value in solving this class of problems and support simulating complex, emerging, dynamical systems. In Part III, we’ll demonstrate cadCAD GPT’s results in a token sales use case and showcase its application to a typical dynamical systems model. For now, let’s look first at the requirements for LLMs in a token engineering context.

Requirements

LLM agents have not been built to support systems simulations initially. They are incapable of performing reliable calculations and lack intrinsic mathematical skills. Moreover, there are plenty of experiments showing their tendency to generate false information or hallucinations.

To provide value to token engineers, we have to satisfy the following requirements:

• cadCAD GPT agents take user input in natural language, change parameters accordingly, run simulations, analyze results, and give back answers in natural language.

• cadCAD GPT agents access and integrate data and information in the simulation, such as transaction data or benchmarking information, and a user should be able to control what information is included or excluded in running a simulation.

• cadCAD GPT agents access, modify, and run a Python token system model. They provide further analysis of simulation outputs and apply typical statistical methods, testing and optimization approaches (such as parameter sweeps and machine learning) to find the best parameter setting.

• cadCAD GPT agents run these methods; they should be able to determine when to use a certain method best. In case several are needed, LLM agents must determine the ordering of methods – all based on a user’s question in natural language.

• Finally, cadCAD GPT outputs have to satisfy criteria of credibility, whether the model’s results are ‘correct’ for each question addressed. Any cadCAD GPT output has to be correct, verifiable, and reproducible.

Introducing the cadCAD GPT framework

The following section introduces the conceptual design of cadCAD GPT.

cadCAD GPT can run simulations on any Python system model adhering to the cadCAD/radCAD system model structure. Through access to the model’s objects experiment, model, and simulation, cadCAD GPT can interact with the model to run simulations.

Users can communicate with agents in natural language (User Input) and receive the results of a simulation experiment in natural language (User Output) provided through the cadCAD GPT chatbot.

cadCAD GPT agents are equipped with Toolkits. Tools are typically Python functions to run experiments and execute parameter changes, A/B tests, or Monte Carlo simulations. Via libraries like Pandas, token engineers can gear agents with the ability to read simulations output data, clean, reformat and analyze output data, and visualize results. In Part III, we show examples and how they are made available to the agents. The modular setup of toolkits allows token engineers to customize and expand the toolkit according to the needs of their particular project. Additionally, we enable maximum control and verifiability of the simulation's outcome.

Memory is any information relevant to a simulation task. Examples include

• The documentation of the Python system model, the model parameters, or metrics to observe

• Protocol information, such as developer docs or whitepapers

• Market data about fundraising or token unlock events

• Blockchain transaction data

• Research papers, and much more.

cadCAD GPT can process a wide variety of data formats, from HTML, XML, and Markdown text to tabular data via API calls. In Part II, we provide a detailed introduction to integrating information in cadCAD GPT Memory.

The core innovation of cadCAD GPT is its orchestration components.To provide value in the simulation process, agents must make sense of user input and decide what tools and/or memory to use and in what sequence.This is where they take over the role of a token engineering copilot. Here, we apply the concepts from projects like BabyAGI and AI-powered task management. Its core idea is to separate the planning and execution steps to allow agents to prioritize steps and improve task completion autonomously.

cadCAD GPT provides a Planner Agent that creates a plan and task list based on the user’s natural language prompt. This task list points to the toolkits, tools, and information needed in the simulation.

This task list is then handed over to the Executor Agent, who goes over the tasklist, reasons about the steps needed, selects and executes actions, and observes the results toward the final output.

The actions of both the Planner Agent and Executor Agent are collected and made printable to allow users to monitor and verify cadCAD GPT’s reasoning process and results delivered.

Finally, the cadCAD GPT chatbot is where all components are weaved together. It provides the UI for natural language inputs and outputs. It imports the radCAD/cadCAD system model objects. Most importantly, it constructs the Planner and Executor Agents and triggers the interplay between agents and the tools and memory available.

Summary

cadCAD GPT equips token engineers with powerful tools to execute simulation tasks via natural language interfaces. With this article, we embark on a journey to establish the essential requirements for LLM agents within complex systems engineering. By deriving these requirements, we pave the way for a deeper understanding of cadCAD GPT’s technical implementation and practical applications. Read on in Part II, where we show how cadCAD GPT components are constructed in detail. We demonstrate how to connect cadCAD GPT with any radCAD or cadCAD model and how additional tools and memory can be added and further customized. Part III of this series showcases cadCAD GPT’s results in typical simulation use cases.

cadCAD GPT Demo

cadCAD GPT will be available on Thursday, Nov 30, 3:00pm UTCSign up for the demo and be the first to get access!

Acknowledgements

cadCAD GPT was kickstarted by funding received from Token Engineering Commons. We thank the TE Commons community, and Gideon Rosenblatt in particular, who encouraged us to embark on this exciting journey. Big thank you to our advisors Roderick McKinley, Richard Blythman, and Robert Koschig for ongoing support and feedback. Shoutout to Dr. Achim Struve, Dimitrios Chatzianagnostou, Stephanie Tramicheck, Ivan Bermejo, Rohan Sundar, and Lukasz Szymanski for the most valuable alpha user feedback and insights, and Kaidlyne Neukam for her tireless support in publishing this work.

Links

The token sales spreadsheet model that informed our token sales experiments is available online, along with a comprehensive online course by Roderick McKinley.

TE Academy is the home for the token engineering community. Learn how to design token systems with rigor and responsibility! Sign up for our newsletter to receive the latest token engineering trends, tools, job offers and ecosystem news.