A custom tool is the part of a thunk that behaves most like traditional software: it takes defined inputs, does something specific, and returns a defined output. That makes it the easiest part to test properly — and the most valuable, because a tool that quietly returns the wrong shape will make every step that calls it look unreliable.

Tool tests run the tool directly, once per case, and check what came back. They are much faster and much cheaper than re-running a whole workflow, so they are the right place to catch a tool problem.

For testing the workflow around the tool, see Evals / Automated Tests.

Open Custom-built tools in the left nav and select a tool. Its detail page has a Tests tab.

The tools list itself carries a Tests column, so you can see at a glance which tools have cases and how the last run went — a check, a cross, or a warning, with the passed-out-of-total count. A tool showing a dash has no tests at all.

A case names an input and states what must be true of the run that input produces. Each case has:

A case can be marked as skipped when you want to keep it on file without running it.

Running a case executes the tool for real. If the tool writes to an external system, its test cases will too — so write cases against safe inputs, or against a sandbox target.

Tool assertions use exactly the same two kinds as work item assertions.

Rules (deterministic) are conditions on the tool's output fields — graded without an AI model, so they are free and give the same answer every time. Because the tool declares its output schema, the rule builder knows each field's type: an enumerated field gets a dropdown of its real values, a numeric one gets numeric operators.

Three extra fields describe the invocation rather than a value in the output, and are always available:

AI-graded assertions are plain-English statements, for expectations a rule cannot express — whether generated prose says the right thing, whether a produced document is laid out correctly. They cost a model call each.

For an AI-graded case you can also nominate output fields under Show the grader these fields' content. Where a field holds a URL or a file, its content is fetched and shown to the grader — so an assertion can judge the screenshot or the PDF the tool produced, not just the URL string. This applies to the case's AI-graded assertions; rules ignore it.

Add test on the Tests tab opens the case editor: give it a description, paste the input JSON, and add rules and AI-graded assertions.

Add tests with AI hands the job to the Tool Builder AI. It reads the tool — what it does, its inputs, its outputs — and writes a set of cases covering the normal path and the edges.

You can steer it before it starts: the dialog takes optional guidance such as cover currency USD, EUR, and JPY; include an empty-input case, or specific output fields you want asserted on. Leave it blank to let the agent decide.

This is the fastest way to go from no coverage to a reasonable suite. Review what it writes — it knows what the tool does, which is not always the same as what it should do.

The Try It tab runs the tool with arguments you supply, as a form or as raw JSON. When the result looks right, Add to tests turns that exact invocation into a case, with the input already filled in.

This is the natural loop while building a tool: try it, look at the output, keep the good one as a test.

The Call history tab lists the tool's past invocations with their inputs, outputs, and status. Each one has Add to tests as well.

This is the highest-value source of cases you have. The inputs in call history are the inputs your workflow genuinely produces — which are rarely the tidy examples you would have invented. When a tool misbehaves in production, the failing call is sitting right there, one click from becoming a permanent regression test.

Run tests on the Tests tab runs the whole suite. The Tool Builder AI carries it out — you can watch it work in the builder chat beside the tool — executing the tool once per case and grading each result against that case's assertions. Cases run in the order you wrote them.

A test run never changes the tool. Asking for tests, or running them, will not rename the tool, rewrite its instructions, or alter its schema or implementation. If the Tool Builder AI notices something wrong with the tool while testing it, it tells you rather than changing it behind your back.

The banner at the top of the tab carries the outcome of the last run: passed, failed, and inconclusive counts, the total number of cases, and when it ran. View details opens the per-case breakdown — the input, the expected behavior, what the tool actually returned, and for each assertion the verdict and the reason behind it.

Each case row also has its own Run, which runs just that case. Use it while iterating on a single case: it does not touch the recorded suite result, so your last full-run status stays meaningful.

As with work item assertions, a result is passed, failed, or inconclusive. Inconclusive means the case could not be judged — the tool errored ambiguously, or the assertion is undecidable for that input. It is a signal to fix the test, not a soft pass.

Export CSV writes every case to a file — id, description, skip flag, input, and assertions — and Import CSV reads them back. Imported cases are appended, never overwriting what is there, and any clashing ids are renumbered.

This is how you hand a suite to someone else, keep a copy outside the product, or let a subject-matter expert draft cases in a spreadsheet without giving them the tool editor.

A tool suite earns its keep by covering the inputs your workflow actually produces, not the inputs you had in mind when you wrote it. In practice that means:

See Building a test plan for how to choose test data more generally, and Custom Tools for building the tools themselves.