Lesson 12: Designing tests that find bugs (advanced, 15 min)

Tests that find bugs need expected results taken from the requirement, scenarios that show each input matters, and a way to check whether the suite would notice a mistake. Three steps, advanced level, about fifteen minutes.

Level: advanced15 min7 claims in this lesson were confirmed by the exact checker at build time

Write expected results from the requirement, show each input’s influence with paired scenarios, and use mutation testing to see whether your tests would notice a bug.

The first step separates the oracle from the code: an expected result must come from the requirement (“banned people never enter”), not from running the formula, because a formula with a slip produces slipped answers with perfect confidence. The second shows each input's influence with a pair of scenarios that differ in that input alone and give different outputs; for the entry rule, four scenarios cover all three inputs. The third introduces mutation testing: small deliberate bugs such as (member AND invited) AND NOT banned, a mutant that swaps OR for AND, are run against the suite, and a mutant is killed when some test fails on it.

The lesson's mutant comparison is a verified claim that the mutant differs from the original. The exercises ask for an assignment that kills a mutant with a dropped NOT, one that kills a mutant with the outer AND replaced by OR, a judgement on a mutant that only reordered operands (equivalent, so no test can kill it), and an expected result written from the requirement rather than from a formula.

The testing section expands each step into a page: independence pairs for MC/DC, mutation testing for Boolean rules, and the pitfall that coverage is not correctness. The app's mutation report classifies each mutant as killed, surviving or equivalent, and offers to turn a surviving mutant's witness into a test.

Do this lesson interactively in the app

Steps

1. Expected results come from the requirement

A test pairs an input scenario with an expected result. That expectation must come from the requirement (“banned people never enter”), not from running the formula.

This formula has a bug: member OR invited AND NOT banned lets a banned member in. A test whose expected value was copied from the formula would also say TRUE and pass, so it proves nothing. Tangibool labels copied outputs as exploratory, not as evidence of intent.

allow_entry := member OR invited AND NOT banned · output

member: the person is a member; invited: the person has an invitation; banned: the person is banned; allow_entry: the person may enter;

Expected results come from the requirement
#	member	invited	banned	allow_entry
0	FALSE	FALSE	FALSE	FALSE
1	FALSE	FALSE	TRUE	FALSE
2	FALSE	TRUE	FALSE	TRUE
3	FALSE	TRUE	TRUE	FALSE
4	TRUE	FALSE	FALSE	TRUE
5	TRUE	FALSE	TRUE	TRUE
6	TRUE	TRUE	FALSE	TRUE
7	TRUE	TRUE	TRUE	TRUE

Add a test member=TRUE, invited=FALSE, banned=TRUE with the expected value FALSE from the requirement, and run it.

2. Show that every input matters

Good tests demonstrate each input’s influence with a pair of scenarios that differ in that input alone and give different outputs. For the entry rule, four scenarios are enough:

- member=FALSE, invited=FALSE, banned=FALSE gives FALSE; - changing only member gives TRUE; - changing only invited gives TRUE; - member=TRUE, invited=FALSE, banned=TRUE gives FALSE (it pairs with the member scenario).

Coverage measures what your tests exercised; it never replaces independent expected results.

allow_entry := (member OR invited) AND NOT banned · output

member: the person is a member; invited: the person has an invitation; banned: the person is banned; allow_entry: the person may enter;

Show that every input matters
#	member	invited	banned	allow_entry
0	FALSE	FALSE	FALSE	FALSE
1	FALSE	FALSE	TRUE	FALSE
2	FALSE	TRUE	FALSE	TRUE
3	FALSE	TRUE	TRUE	FALSE
4	TRUE	FALSE	FALSE	TRUE
5	TRUE	FALSE	TRUE	FALSE
6	TRUE	TRUE	FALSE	TRUE
7	TRUE	TRUE	TRUE	FALSE

Save the four scenarios as tests and open the coverage view.

3. Mutation testing: would your tests notice?

Mutation testing makes small, deliberate bugs (mutants), such as swapping OR for AND or dropping a NOT, and reruns your suite. A mutant is killed when some test fails on it.

(member AND invited) AND NOT banned is killed by the test member=TRUE, invited=FALSE, banned=FALSE: expected TRUE, mutant FALSE. A mutant that behaves exactly like the original is equivalent; Tangibool identifies those with an exact check, so they never count against your score.

original := (member OR invited) AND NOT banned · outputmutant := (member AND invited) AND NOT banned · output

member: the person is a member; invited: the person has an invitation; banned: the person is banned;

Mutation testing: would your tests notice?
#	member	invited	banned	original	mutant
0	FALSE	FALSE	FALSE	FALSE	FALSE
1	FALSE	FALSE	TRUE	FALSE	FALSE
2	FALSE	TRUE	FALSE	TRUE	FALSE
3	FALSE	TRUE	TRUE	FALSE	FALSE
4	TRUE	FALSE	FALSE	TRUE	FALSE
5	TRUE	FALSE	TRUE	FALSE	FALSE
6	TRUE	TRUE	FALSE	TRUE	TRUE
7	TRUE	TRUE	TRUE	FALSE	FALSE

Compare original with mutant and save the counterexample as a test.

not-equivalent original / mutant · holds

Exercises

This mutant dropped the NOT: (member OR invited) AND banned. Find a test scenario that kills it (the two outputs differ). (find-assignment)
This mutant swapped the outer AND for OR: (member OR invited) OR NOT banned. Find a scenario that kills it. (find-assignment)
A mutant only reordered the operands: NOT banned AND (invited OR member). Classify ((member OR invited) AND NOT banned) <-> (NOT banned AND (invited OR member)). (classify)
Write the expected result from the requirement, not from a formula: a member who is also invited, but banned (member=TRUE, invited=TRUE, banned=TRUE). Should allow_entry be TRUE or FALSE? (predict-output)

Do this lesson interactively in the app

Questions

How many tests does the entry rule need?

Four scenarios give an independence pair for each of its three inputs, the second step's result. That is the MC/DC minimum of n + 1 for n conditions; the independence-pair page shows the four rows.

What is an equivalent mutant?

A mutant that computes the same function as the original, such as a reordering of operands. No test can kill it, and the engine reports it as equivalent rather than counting it as a surviving bug.

Tools

Truth table generator: any Boolean expression, up to 12 variables, with intermediate columns

Lessons

Generated from the Tangibool engine, semantics tangibool-semantics/1, engine 1.0.0, build 2026-11-07. Every table, map, simplification and verdict on these pages was computed by the Tangibool engine when the site was built, and every claim was re-checked by the same exact checker the app uses. Limits are stated where they apply. © 2026 Foreranger LTD.