Test oracle (expected result): meaning and examples

A test oracle is whatever supplies the expected result of a test: the thing an actual result is compared against. For a Boolean rule, the oracle is the intended rule, and a truth table of it gives the expected outcome for every input.

Every test has two halves, an input and an expected result, and the oracle is the source of the second. It may be a specification, a reference implementation, a human judgement recorded once, or an older version of the same program. The oracle problem is that an oracle can be wrong, incomplete or expensive; when the oracle is a formula, as it is for Boolean rules, it is at least complete and cheap, and comparing it with the implementation on every row turns testing into an equivalence check.

The artefact on this page sets the oracle intended = (member OR invited) AND NOT banned beside the implementation implemented = member OR (invited AND NOT banned), a misplaced parenthesis of the kind that passes casual testing. The two columns differ on rows 5 and 7, and the equivalence check is violated with the counterexample member = TRUE, invited = FALSE, banned = TRUE: the oracle says FALSE, a banned member is refused, while the implementation says TRUE. Row 7, where the member is also invited, is the other case; on the remaining six rows the two agree.

Oracles are what make a test meaningful; a test with no oracle is a smoke test that only checks nothing crashed. In rule review, writing the intended rule down separately from the code, in plain words or as a formula, is the step that creates an oracle. The term is confused with the test itself, with a golden master (an oracle made of recorded outputs), and with the database product of the same name.

Example

intended := (member OR invited) AND NOT banned · outputimplemented := member OR (invited AND NOT banned) · output

Exact checks

Always true (tautology)? No Counterexample: member = F, invited = F, banned = F · Exhaustively checked and solver cross-checked · 1 rows

Always false (contradiction)? No Counterexample: member = F, invited = T, banned = F · Exhaustively checked and solver cross-checked · 3 rows

Can be true? Yes Example assignment: member = F, invited = T, banned = F · Exhaustively checked and solver cross-checked · 3 rows

Are intended and implemented equivalent? No Counterexample: member = T, invited = F, banned = T · Exhaustively checked and solver cross-checked · 6 rows

Truth table (8 rows)

Test oracle (expected result): meaning and examples
#	member	invited	banned	intended	implemented
0	FALSE	FALSE	FALSE	FALSE	FALSE
1	FALSE	FALSE	TRUE	FALSE	FALSE
2	FALSE	TRUE	FALSE	TRUE	TRUE
3	FALSE	TRUE	TRUE	FALSE	FALSE
4	TRUE	FALSE	FALSE	TRUE	TRUE
5	TRUE	FALSE	TRUE	FALSE	TRUE
6	TRUE	TRUE	FALSE	TRUE	TRUE
7	TRUE	TRUE	TRUE	FALSE	TRUE

Plain-language reading

Beginnerintended is true exactly when both of the following hold:
1. at least one of member or invited is true (both may be true)
2. banned is false
In every other case, intended is false.
About the operators:
- AND is true only when all of its parts are true.
- OR is inclusive: it is true when at least one part is true, including when several are.
- NOT reverses a value: true becomes false and false becomes true.

Intermediateintended is true exactly when both of the following hold:
1. at least one of member or invited is true (both may be true)
2. banned is false
In every other case, intended is false.

Advancedintended is true exactly when both of the following hold [conjunction ∧]:
1. at least one of member or invited is true (both may be true) [inclusive disjunction ∨]
2. banned is false [negation ¬]
In every other case, intended is false.
Formal notation: (member ∨ invited) ∧ ¬banned

The common slip

Using the implementation as its own oracle. Recording the outputs of implemented and testing against them would pass on rows 5 and 7 and enshrine the bug. The oracle must come from somewhere other than the code under test, as intended does here.

Questions

What if the oracle is wrong?

Then the tests enforce the wrong rule, which is why the oracle should be reviewed by someone who knows the domain rather than the code. The engine's plain-language reading helps: it renders the formula as a sentence that a non-programmer can confirm or reject.

Glossary

Generated from the Tangibool engine, semantics tangibool-semantics/1, engine 1.0.0, build 2026-11-07. Every table, map, simplification and verdict on these pages was computed by the Tangibool engine when the site was built, and every claim was re-checked by the same exact checker the app uses. Limits are stated where they apply. © 2026 Foreranger LTD.