# My RAG bot scored 26/26. A trace showed me what the score was hiding. By [The BlogChain Newsletter](https://paragraph.com/@kazani) ยท 2026-10-03 ai, rag, llm, ai-engineering, observability, langgraph, langfuse, building-in-public --- The trace said the LLM call took 0.08 seconds. That's too fast. When I later timed a fresh call to gpt-4o-mini, it took 1.30 seconds. Nothing in my terminal looked wrong. My eval had just scored 26 out of 26. Every answer was correct and every citation pointed to the right post. By any score I had, the project was done. The trace was showing me something the score couldn't. A passing eval tells you the answers were right. Only a trace tells you what it took to get them. What I built ------------ Ask BlogChain answers questions from this newsletter's archive. You ask something like "Where did my trading bot finish?" and it finds the right passages, writes a short answer, and cites the exact posts it used. If the archive doesn't have the answer, it says so instead of guessing. I built it to learn the tools AI engineers use for this kind of work, and to learn them properly. So I gave myself a rule: one new skill per stage, and measure each stage before starting the next. Before I wrote any search code, I wrote 26 test questions by hand. Twenty-four have a known answer in a specific post. Two have no answer in the archive at all, to check that the bot can say "I don't know." I wanted to know why it worked, not only that it worked. Each stage showed a gap between what the output said and what the measurement showed. There were four stages, so there are four gaps. The last one is the 0.08 seconds. Why a passing eval isn't enough ------------------------------- LLM apps look finished long before you know whether they are. The answer reads well, the format is right, and the demo works. The usual advice is to write evals, and that advice is good. I wrote one before I wrote anything else. But an eval only checks the questions you thought to ask. Mine checked whether each answer was correct and whether it cited the right post. It never checked how long each step took, how many tokens it used, how often it retried, or whether the reply came from the model at all. Those things were never wrong in a way the score could see. So you need two kinds of looking. A score answers "is it right?" A trace answers "what did it take?" Most of what surprised me came from the second kind. Stage 1: search that reads your words, not your meaning ------------------------------------------------------- The first step in any RAG app (retrieval-augmented generation, meaning search first, then answer) is finding the right passages. I started with BM25, a classic keyword ranking. It scores a passage by how many of your words it contains, and rare words count for more. On questions that reused the post's own words, BM25 put the right post first every time, 10 out of 10. On questions that asked the same thing in different words, it got 64%. A reader doesn't know my phrasing, so that second number is the one that matters. Embeddings fix this. They turn text into numbers that capture meaning, so "Where did my bot finish?" lands near "I finished around 48th." Embeddings raised the reworded questions to 79%. But they lost something keywords were good at. A question about "USDC on Chelsea's shirts" dropped to fourth place, because names and rare words are exactly where keyword matching wins. So I combined them. Hybrid search runs both and merges the two rankings. It put the right post first 88% of the time, compared with 79% for keywords alone. If your test questions reuse your documents' wording, your search will look better than it is. Write some of them the way a stranger would ask. Stage 2: a schema guarantees the shape, not the truth ----------------------------------------------------- Search was working, but the answers were free text. I couldn't check them automatically, so I made the model return structured data instead. Each answer is a JSON object with a `found` flag and a list of claims, and every claim lists the sources it came from. There are two layers. The API's strict mode guarantees the JSON has the right shape. Then Pydantic, a Python validation library, checks the rules a shape can't express: every source number must exist, every claim needs a source, and `found` must agree with the claims. If a check fails, the error goes back to the model and it tries again, up to three times. The results were 25 out of 26 correct and 98% citation precision. The retry loop never fired once in a live run. Strict mode was so reliable that my safety net never got used. That sounds like good news, but it also meant the retry code had never run. Code that has never run is code you don't know works. So I tested it on purpose, with a fake model that returns broken JSON and invalid replies, and checked that the loop recovered. The one miss wasn't the model's fault. The right post wasn't in the top five search results, so the model said "not found" instead of making something up. That's the behavior I wanted, but the miss still needed fixing. A safety net that never fires is one you haven't tested yet. Stage 3: an agent that checks its own work, and what that costs --------------------------------------------------------------- The Stage 2 miss happened because the pipeline searched once and trusted whatever came back. So I turned it into a small agent using LangGraph, a library for building LLM steps as a graph with loops. The agent retrieves passages, then grades them by asking the model which passages actually answer the question. If none do, or if the answer step says "not found," it rewrites the question in different words and searches again, at most twice. This pattern is called corrective RAG. It worked: 26 out of 26, with every citation correct. The question that Stage 2 missed is the interesting one. The grader made a mistake and kept a passage that was off topic. The answer step looked at it and said "not found." That sent the agent back to rewrite, and the new wording found the right post. The loop caught an error made by another part of the same system. It also cost 46% more tokens. Most of that went to the two unanswerable questions, which now try three different searches before giving up. For a demo, that trade is worth it. At a million questions a day, I'd probably allow only one rewrite. Before you add a loop, put its cost per question next to the questions it rescues. Stage 4: what the trace saw --------------------------- At 26 out of 26, the project looked finished. That's when I added tracing with Langfuse. A trace is a tree of everything one request did: each step, nested inside the step that called it, with its time, tokens and cost. One question produced 20 nested steps and cost $0.000928. A log gives you a list of lines. A trace shows you which step ran inside which, and where the time and money went. When I looked through the traces, some LLM calls finished in 0.08 seconds. So I tested it. I sent a fresh request and timed it at 1.30 seconds. Then I sent the identical request again and it came back in 0.08 seconds. The gateway I use to reach the models caches identical requests. If it has seen the exact same request before, it returns the saved reply without calling the model. That changes how I read my own results. My eval sends the same 26 questions every time. After the first run, most replies come from the cache. The answers are still correct, so the score is honest. The timings are not, because a repeat eval run looks much faster than real use would be. I still don't know whether the gateway charges full price for a cached reply. Langfuse calculates cost from token counts, so its cost figure may not match my bill. The score could never have shown me this. It was right about the answers and silent about everything else. A smaller surprise came earlier. When I first tried to read a trace back through Langfuse's API, I got an HTTP 410 error, which means "gone." The old endpoint had been retired for new accounts, and the error message named the replacement. Reading the whole error, instead of the status code, made it a quick fix. Try it on your own app ---------------------- You don't need my project to find your own 0.08 seconds. This is the experiment I'd run on any LLM app. 1. Add tracing to one request path in your app. Langfuse's `@observe` decorator is what I used, but any tracing tool that records time and tokens per step will work. 2. Run your eval twice in a row without changing anything. 3. Put the two runs side by side and compare the time and tokens for each step. 4. Look for anything that differs when it shouldn't, or matches when it shouldn't. Steps that are too fast, retries you didn't expect, and token counts that jump between runs are all worth a look. 5. Write down the one number you can't explain. Then test it directly, the way I timed one fresh request against the same request repeated. One rule helps a lot here: read the trace back through the API instead of trusting the dashboard link. A link proves you sent something. Reading it back proves what actually arrived. What the score couldn't say --------------------------- The 0.08-second call wasn't a bug in my code. Every answer was still right. But it meant one of my measurements was flattering me, and nothing in the score would ever have told me. Each stage of this project taught the same thing in a different way. Keyword search looked perfect until the questions were reworded. The retry loop looked solid because it had never run. The agent looked free until I counted its tokens. The eval looked complete until a trace showed what it took to pass. If you take one step from this post, trace one request in your own app this week and read it back. The code -------- Ask BlogChain is public at [github.com/kazani-351/ask-blogchain](https://github.com/kazani-351/ask-blogchain). It includes the code, the 26-question eval set, and a learning record for each stage in LESSONS.md. Run `show_trace.py` to see a full trace in your terminal. If you add tracing to something you've built, reply and tell me the strangest number you found. I want to know what other people's 0.08 seconds look like. --- *Originally published on [The BlogChain Newsletter](https://paragraph.com/@kazani/my-rag-bot-scored-2626-a-trace-showed-me-what-the-score-was-hiding)*