Self-healing selectors and autonomous exploratory agents get most of the attention in AI-driven QA, and for good reason — they solved a real, expensive problem: locator maintenance. But they solved the easy half of test automation. Generating inputs and navigating an app was never the hard part. Knowing whether the result you got back is actually correct — the test oracle problem — is, and it's where the interesting work is happening now.
What a Test Oracle Actually Is
An oracle is whatever mechanism decides pass or fail. In classic Playwright testing, that's an explicit assertion: expect(heading).toHaveText('Order Confirmed'). Simple, deterministic, and useless the moment the correct output isn't a fixed string — a generated summary, a chatbot reply, a recommendation ranking, a visually-but-not-textually correct layout.
Research on LLM-based oracles categorizes them into a few forms:
| Oracle type | What it checks | Example use |
|---|---|---|
| Assertions | Exact expected values | Traditional unit/E2E tests |
| Expected output | Known correct result | Deterministic business logic |
| Metamorphic relations | Consistency between related inputs | "Same query, reordered filters, should return same result set" |
| Properties / invariants | Rules that must always hold | "Total price never negative" |
| Exception oracles | Expected failure modes | Input validation, error boundaries |
| LLM-as-judge | Semantic correctness, no fixed string | Chatbot responses, generated content, ambiguous business logic |
The last row is the one classic automation can't touch, and it's exactly where most AI-generated features actually live now — a support bot reply, a summarization feature, an AI-assisted search result. There's no string to assert against, only a judgment call about whether the output is good.
Why Generation Was the Easy Half
Generating test inputs — form data, API payloads, exploratory navigation paths — is a well-bounded problem: enumerate the space, mutate it, fuzz it. Generating a correct oracle for a non-deterministic output requires actually understanding intended program behavior, not just coverage. That's a fundamentally harder ask, and it's why "AI writes your tests" tools have gotten good at the input side while mostly punting on the assertion side — falling back to snapshot testing, which just encodes whatever the output happened to be on day one as "correct," bugs included.
Multi-Agent Judge Panels
The current state of the art doesn't use a single LLM call as a judge — a lone judge model inherits the same blind spots and biases as whatever it's grading. Two architectures worth knowing:
- Panel-consensus oracle generation — several judge agents independently evaluate an output and reach consensus through structured deliberation, rather than one model rubber-stamping a pass/fail.
- Deliberation-validation-refinement — a judge proposes an oracle, a second agent validates it against the actual execution trace (not just the stated spec), and a third refines it if the validator flags a mismatch — closing the gap between what a spec says should happen and what the code actually does.
Roughly, a panel-based verdict looks like this in shape:
{
"output_under_test": "AI support reply to a refund request",
"judges": [
{ "agent": "correctness", "verdict": "pass", "reasoning": "Refund policy correctly cited" },
{ "agent": "tone", "verdict": "pass", "reasoning": "Matches brand voice guidelines" },
{ "agent": "safety", "verdict": "fail", "reasoning": "Leaked internal order ID format in example" }
],
"consensus": "fail",
"resolution": "Escalate to human review — safety judge dissent blocks auto-pass"
}The value isn't any single judge's opinion — it's that disagreement between judges is itself signal. A unanimous pass is trustworthy in a way a single model's pass never is.
Where This Actually Belongs in Your Pipeline
This is not a replacement for deterministic assertions, and treating it as one is the mistake to avoid. The same layered thinking that applies to autonomous exploratory agents applies here: use LLM-as-judge oracles specifically for the outputs that have no fixed correct string — generated text, ranked results, chatbot behavior, subjective UX quality — and keep hard assertions on anything with a deterministic expected value. Payment totals, auth state, and data integrity checks should never be graded by a model that can hallucinate a passing verdict.
Two practical constraints to build around:
- Judge panels are expensive and slow. Reserve them for the outputs that genuinely lack a deterministic oracle, not as a blanket replacement for
expect(). - Judges need their own audit trail. If a panel passes something a human later flags as wrong, you need the per-judge reasoning logged, not just the consensus verdict — otherwise you've just moved the "why did this pass" problem from Playwright flakiness to opaque model reasoning, which is worse, not better.
The oracle problem was always the real bottleneck in AI-driven testing — automation could always click the right button, it just couldn't reliably tell you whether what happened next was correct. Multi-agent judge consensus is the first approach that treats that as the hard problem it actually is, instead of quietly falling back to snapshot diffs and hoping nothing changed.
Sources: