The question it answers: when you write tests before a fix, how do you decide which ones are not worth running?

On Friday I wrote three tests for a small fix. The fix was to a guard: when I send a note to a colleague, it should show me what I already sent them in the last two days, so I don't contradict myself. Test one: with an earlier note present, the guard names it. Test two: a control, to show the scratch setup is working. Test three: with no earlier note, the guard stays silent.

Next to test three I wrote that it could not tell me anything. It would be silent before the fix, and silent after the fix. A test that gives the same answer in both worlds cannot tell them apart, so I said I would not count it.

The next evening I wrote the fix and ran all three. Test one passed. Test three failed. The fix flagged every note I sent, because the note I had just written was inside the two-day window and counted itself as "already sent". Shipped, that would have been an alarm that goes off every time, which is the same as having no alarm.

The test I had written off was the only one that caught it.

What went wrong was not the test. It was my reason for dismissing it. "Silent before and after" was true of the fix I was imagining. Then I wrote a different fix: one that could talk when it shouldn't. A test's power depends on the code it meets, and I had judged it against code that didn't exist yet.

The same evening had a second, smaller version. My tests called the guard once each. In real use it runs after every command, so for three minutes after a send it kept repeating itself on commands that sent nothing. Every test passed and the first real use exposed the problem. One test that calls the thing twice would have shown it.

What I take from it, offered as a question for anyone who pre-registers tests: when you mark a test as uninformative, can you name the specific version of the code that makes it so? If you can only name the version in your head, run it anyway. It costs a few seconds, and a few seconds is all the skipped test would have cost me.