The mechanics of testing a thunk are straightforward — mark some work items as test work items, write assertions, re-run them. The hard part is deciding which work items and which assertions. A suite of thirty tests over near-identical inputs gives you a comforting pass rate and almost no information.
This article is about that decision. For the controls themselves, see Evals / Automated Tests and Testing your custom tools.
Two laws
Everything below follows from two facts that hold without exception:
Low coverage produces a low-quality thunk. What you never test, you never find out about until a user does.
Low-quality test data produces a low-quality thunk. A test whose expected answer is wrong actively trains you to accept wrong behavior.
Choosing test data
Use real inputs, not invented ones
The single most common failure of a test suite is that its inputs are cleaner than production. Someone writes six tidy example documents, everything passes, and the thunk falls over on the first real one — which arrives as a scanned PDF, with the fields in a different order, in a language nobody mentioned.
Take your inputs from the actual source: the mailbox, the drive, the system that will feed the thunk. If you cannot use real data for privacy reasons, redact it rather than rewriting it — the messiness is the part you are trying to keep.
Cover the shapes, not the volume
Coverage is about distinct kinds of input, not counts. Twenty invoices from the same supplier in the same format are one test. Aim to have at least one case for each:
Typical case per meaningfully different source, format, or region.
Edge of the range — the largest document, the shortest message, the zero amount, the date at a boundary.
Legitimately missing data — an optional field that is absent, a document with no attachment. What should the thunk do here? Decide, then assert it.
Bad input — the malformed file, the wrong document type, the empty submission. A workflow's behavior on garbage is part of its specification.
Each branch your workflow can take. If a step routes on a condition, you need an input on each side of it, or that branch is untested.
Do not forget the negative cases
Most suites test that the thunk does the right thing. Fewer test that it does not do the wrong thing — approves what it should escalate, sends what it should hold, fills in a value it should have left empty. These are usually the failures that matter most, and they need their own inputs.
Turn every reported problem into a test
This is the highest-return habit in the whole practice. When someone reports that the thunk got something wrong, the case they are complaining about is a perfect test input, already known to be interesting. Capture it as a test work item, write the assertion that would have caught it, then fix the problem.
Do it the same day. A suite grown this way converges on exactly the cases your workflow finds hard.
Choosing assertions
Assert what matters, not everything
The temptation — especially when seeding assertions automatically from a good run — is to assert every property. That produces a brittle suite that fails whenever anything changes, including harmless things, and trains everyone to ignore failures.
For each work item, ask what would actually constitute a wrong answer, and assert that. Three sharp assertions beat fifteen incidental ones.
Prefer a rule to an AI judgment
If an expectation can be written as a condition on a field, write it as a rule. Rules are free, instant, and deterministic — the same input gives the same verdict forever. AI-graded assertions cost a model call and carry judgment, which is exactly what you want for generated prose and exactly what you do not want for an amount.
A good rule of thumb: rules for structure and constraints (this field is not empty, this value is one of these three, this number is under the limit); AI assertions for content and meaning (the summary mentions the renewal date; the reply does not promise a delivery date).
Make the expected answer match what you told the thunk to produce
Before you save an expected value, check it against the instructions. If the step is configured to write status values from a fixed list, the assertion must use those exact values — not a synonym you happen to prefer. A test that demands something the thunk was never asked to do is not a failing thunk; it is a failing test, and it will burn hours before anyone notices.
Avoid asserting things that change on their own
Do not build assertions on live schedules, external web pages, or anything whose correct answer moves over time. Those tests go stale and start failing for reasons that have nothing to do with your change, which is worse than having no test — it is a test that teaches people to ignore red.
Where a case genuinely depends on the date, control the clock instead: set a Simulated current time in Test Settings so the case is evaluated against a fixed "now".
Making a suite you can actually re-run
A regression suite has to be runnable on demand, repeatedly, without consequences. That means dealing with side effects up front:
Email — set the test email policy to suppress, or redirect to one address you own.
Steps with external effects — a step that files a ticket or updates a record should be marked to be skipped for test work items.
Tools that write somewhere — replay captured tool outputs instead of calling them live, or point them at a sandbox.
If running the suite is scary, it will not be run — and a suite that is not run is not a suite.
Deciding when to ship
Before a thunk carries production work, be able to state three things:
Coverage — which of the expected kinds of input have a test, and which do not. Name the gaps explicitly; an unnamed gap is a surprise waiting to happen.
Pass rate — how the suite currently scores, from an actual run report rather than memory.
Risk and reward — whether the value of shipping now exceeds the cost of the gaps you just named.
Make that call with the people who own the business outcome, not alone. The run report is downloadable for exactly this reason.
Who does what
Testing slips when nobody owns it. Three roles need to be filled, and they are rarely the same person:
Business owners supply the cases and confirm that the expected answers are genuinely correct. They are the only people who can say what "right" means.
Thunk builders make the thunk pass those cases, and investigate every failure rather than adjusting the assertion until it goes green.
Whoever operates the thunk runs the suite on a regular cadence and circulates the results — including when nothing changed.
Name all three at the start of the project. A gap in any one of them shows up as declining quality a few weeks later.
For the broader engineering practices around shipping a thunk to production, see Shipping Enterprise Thunks: Testing and Quality Principles.