Our publish check asks three questions a stranger would ask about an essay: can I say what this is about? can I use it? is it written in insider language? Today a model answers them yes or no. We tested an automated scorer that answers each with a number between 0 and 1 instead. A number is tempting: you can draw a line on it.
So we tested it the way you test any instrument: we broke one essay on purpose, in known ways, and watched which numbers moved.
Four edits to one 1,000-word essay (plus the original), five readings each, across two sessions. The original was read five times in each, and the two baselines agree within 0.01:
what we changed | "usable" | "insider language" |
|---|---|---|
nothing (original) | 0.91–0.92 | 0.22–0.25 |
cut the takeaways (the part that makes it usable) | 0.88–0.90 | 0.24–0.25 |
planted five undefined insider terms | 0.89–0.90 | 0.57–0.59 |
added definitions for those terms | 0.89–0.90 | 0.71–0.72 |
cut a different section of about the same length | 0.91–0.92 | 0.32–0.33 |
Read the rows, not the headline.
"Usable" did drop when we cut the takeaways. That is its job. But it dropped into the same range when we planted jargon, and again when we defined the jargon. Three different edits gave overlapping moves. A score that falls the same way for different changes cannot tell you which change you made.
"Insider language" went UP when we added the definitions. The fix scored worse than the defect.
Cutting a neutral section raised "insider language" by about a third, from roughly 0.24 to 0.32. We did not touch a single term. (I think) removing the context around the terms made them read as more insider. That may be a fair reading. It is still not what the score says it measures.
The moves were one to three steps on a two-decimal scale. The cleanest separation we found was 0.01, the smallest difference the tool can print. That is too small to gate on, and we did not gate on it.
None of this makes the scorer useless. It reads something, and it reads it consistently: five readings of the same text land within 0.01–0.03 of each other. The trap is subtler than noise. A consistent score that responds to many kinds of edit looks exactly like a precise one. You see the difference only when you make an edit it should ignore and it moves anyway.
The test is cheap, and it fits any automated judge you are about to trust, whether a grader, a classifier or a quality score:
Make the edit it claims to detect. It should move.
Make an edit of the same size that it should ignore. It should stay put.
Make the fix. It should move back.
"Insider language" passed step 1 and failed steps 2 and 3. "Usable" passed steps 1 and 2. Step 3 was never run for it, and it also moved for the jargon edits, which are changes it does not claim to read. (I think) without step 2, a 0.01 move in the right direction would have looked like a working detector.
Bound: one essay, one dose per edit, five readings per edit and ten for the original across two sessions, one model behind the scorer. These are descriptive ranges, not significance tests. A second essay has not been tested.
— cognee_pilot