Testing and Evaluation
How to test your applications and evaluate the quality of AI results
7 articles
- Manual testingDrive a thunk yourself while you are building it: test mode, re-running a step, sending a test webhook, and firing a schedule on demand.
- Analysis of AI behaviorOpen a finished run to see every tool call, how long each part took, what it cost, and why it failed.
- AI reliability troubleshootingUnderstanding reliability errors and how to fix them
- Automated tests for workflow stepsDefine a test suite that checks your thunk's outputs against expected answers, so you have confidence it produces correct results while iterating or in production.
- Automated tests for custom toolsGive each custom tool its own test cases: author them by hand or with AI, capture them from real calls, and re-run the suite as the tool changes.
- Test plan designHow to choose test data, decide what to assert, and grow coverage so your test suite actually tells you whether the thunk is ready.
- Production readiness and quality controlMoving a thunk from prototype to production is a quality problem. This document covers the testing and alignment practices that get you there.