
Every vendor selling into education now claims AI capability. The claim costs nothing to make, which means it carries almost no information — and buyers evaluating these products are mostly left comparing marketing copy.
The good news is that a handful of technical questions cleanly separate teams running language models in production from teams who built a convincing demo. They work because the answers require experience rather than positioning. A practitioner answers each in under a minute. Someone who has not shipped will reach for a case study, a roadmap, or a redirection.
Here they are, with notes on what a good answer sounds like.
1. How do you evaluate output quality?
Ask: What does your evaluation set look like, who wrote it, how large is it, and how often does it run?
This is the highest-signal question in the set, because building an evaluation harness is unglamorous work that only happens when a team has been burned.
A strong answer describes something concrete: a set of cases with expected properties, written with domain experts rather than by engineers guessing, run in CI or on a schedule, with results tracked over time. It usually includes an admission that the eval set is incomplete and how they decide what to add — typically, production failures become test cases.
A weak answer talks about "extensive testing," describes manual spot-checking, or explains that the model provider handles quality. The last one is the clearest tell. Model providers guarantee nothing about whether output is correct for your domain.
Follow-up worth asking: what did your eval catch most recently? Teams with a live harness have a specific story. Teams without one do not.
2. What happens when retrieval returns nothing relevant?
Ask: If the retrieval layer finds no relevant content for a question, what does the system do?
The correct answer is that it declines — says it does not know, offers to escalate, routes to a human. In education this matters acutely, because a system that invents a course prerequisite or a policy detail produces real consequences.
The revealing answer is any version of "the model uses its general knowledge." That means the product will confidently produce plausible statements about your institution that are not true, and it will do so most often in exactly the situations where the user needed accuracy.
Follow-up: how do you detect when this is happening in production? Good teams monitor retrieval hit rates and low-confidence responses. Teams who have not thought about it will not have the instrumentation.
3. How do you handle prompt injection in user-submitted content?
Ask: When the retrieved content is student-submitted, what stops a student embedding instructions in it?
This is the education-specific question, and it is genuinely important. If a product summarises student work, marks it, or feeds it into any assistant, then student-authored text is entering the model's context. Students are curious, occasionally adversarial, and extremely good at finding boundaries.
A credible answer covers several layers: treating retrieved content as data rather than instruction, structural separation in the prompt, output constraints, and — crucially — an acknowledgement that no defence is complete, plus a description of what the blast radius is when it fails. Teams who understand this design so that a successful injection produces a wrong summary rather than data exfiltration or privilege escalation.
An answer of "we have guardrails" without specifics means they have not tested it.
4. What breaks when the model version changes?
Ask: How do you find out that a model update changed your outputs, and how quickly?
Model providers deprecate versions and adjust behaviour. Prompts tuned against one version can degrade against the next in ways that are silent — no error, no exception, just gradually worse output that nobody notices until a user complains.
Good answers involve pinned model versions, the evaluation harness from question one running against candidate versions before migration, and a rollback path. It is a straightforward continuous integration problem once a team recognises it as one.
Answers indicating they always use the latest default version, with no evaluation gate, describe a system whose behaviour changes without anyone deciding it should.
5. What does inference cost per active user per month?
Ask: At current production volumes, what is your token cost per active user, and how does it scale?
This is the question most likely to expose a demo. Anyone operating AI features at scale knows this number precisely, because it determines whether the feature survives the next budget review. They will also tell you what they did to reduce it — caching, smaller models for classification steps, truncated context, batching.
Vague answers here usually mean the product has not run at meaningful volume. That is not necessarily disqualifying for an early-stage vendor, but you should know it, and you should be careful about pricing that assumes costs the vendor has not yet experienced.
For your own planning, model this at roughly ten times your pilot volume. A cost that is negligible across 200 users can be structurally unaffordable across 20,000 — and this is how promising features get quietly disabled at the start of a financial year.
Two questions for the procurement side
Beyond capability, two more belong in any regulated-sector evaluation:
Where does inference happen, and are inputs excluded from training? Provider, region, and contractual terms. If student writing is being summarised, that text has a destination. A surprising number of vendors selling into schools cannot currently answer this.
Who reviews AI output for accuracy, and what is the escalation path? Not a technical question, but it determines whether errors are caught by staff or by a parent.
Reading the pattern rather than the answers
Individual answers matter less than the shape of the conversation. Practitioners answer these quickly, specifically, and with more nuance than you asked for — including admitting limitations, because anyone running this in production has hit them.
The tell for a wrapper is consistent deflection to what the model provider handles, to the roadmap, or to a case study that describes outcomes rather than mechanisms. Real LLM integration work leaves marks: eval sets, monitoring, cost dashboards, incident stories. Ask for the marks.
Frequently Asked Questions
What is an evaluation harness for an LLM feature? A structured set of test cases with expected properties, run automatically and tracked over time, that detects when model output quality degrades. Good ones are written with domain experts rather than engineers, grow from real production failures, and gate model version changes before they reach users.
Why does refusal behaviour matter so much? Because the alternative is confident invention. A system that falls back on general model knowledge when retrieval finds nothing will produce plausible, specific, wrong statements — and it will do so precisely when the user most needed a correct answer. Declining and escalating is the correct behaviour.
What is prompt injection and why is education particularly exposed? Prompt injection is embedding instructions inside content that a model later processes as context. Education is exposed because student-submitted work routinely enters model context for summarising, marking or assistance, and students are unusually motivated to probe boundaries. Design so that a successful injection produces a wrong summary rather than data access.
How do model version changes break production systems? Silently. Prompts tuned against one version can degrade against the next with no error raised — just gradually worse output. The mitigation is pinning versions, running the evaluation harness against candidates before migration, and keeping a rollback path.
Why ask about inference cost per user? Because it exposes whether a vendor has operated at real volume, and because it is the cost that most often kills AI features after launch. Model it at roughly ten times pilot volume; something negligible across 200 users can be unaffordable across 20,000.
What if a vendor cannot answer these questions? Take that as information rather than a reason to dismiss them outright. An early-stage vendor may genuinely not have production experience yet, which is a risk you can price. A vendor who deflects while claiming production maturity is a different problem entirely.
The full buyer's guide to Australian education technology providers — categories, compliance, accessibility and AUD cost bands — is at TechCirkle.