Move a thunk to a new AI model
Sooner or later a thunk changes AI models — a newer model is faster or cheaper, or the one it uses is being retired. A different model can produce different results from the same instructions, so switch the way you would ship any other change: try the new model on your test work items first, compare, and only then switch the thunk over.
For the models you can choose from and what happens when one is retired, see Supported AI models and what happens when they change.
Before you start
Have a test suite. You need a set of test work items with assertions that cover the thunk's important cases. If you don't have one yet, see Automated tests for workflow steps.
Check the model is available. The models you can pick depend on the environment the thunk runs in. If the model you want isn't offered, ask your administrator to add it to that environment — see Orgs, Roles, and Environments.
Know where the thunk's model is set. The thunk-wide model is in Settings → AI Settings, under LLM model to use for execution AI agents in this thunk. A step can override it: open the step, then AI Settings → Preferred model. Custom AI tools can also have a preferred model of their own.
1. Record a baseline
Run your test suite on the current model so you have something to compare against:
Open Tests → Test Results and Re-run the test work items.
When the run finishes, note which assertions passed and failed. To keep a copy, use Re-grade test assertions, which produces a downloadable report.
Also note the current figures on Monitor → Token Costs and Monitor → Time Analysis, or simply keep the window wide enough to cover both runs.
2. Run the tests on the new model
Open Tests → Test Settings.
Under Test run, set Model to the model you want to try. This forces that model for your test work items' ordinary steps, without changing the thunk itself. (Planning steps keep their own model.)
Go back to Test Results and Re-run the test work items.
Only test work items use the forced model. Real work keeps running on the current model the whole time.
3. Compare
Results. Compare the new results with the baseline: how many assertions pass, and which ones changed. Open any work item whose result changed and read what the AI agent did differently.
Cost. On Monitor → Token Costs, set Colour dots by to LLM model. The two runs appear side by side in the chart, so you can see whether the new model costs more or less per work item.
Time. Do the same on Monitor → Time Analysis to compare how long work items take.
If the new model fails cases the old one passed, you can often fix that with clearer step instructions — see How to Write Effective Directions. Re-run the tests after each change.
4. Switch the thunk
When you are satisfied:
Open Settings → AI Settings and change LLM model to use for execution AI agents in this thunk to the new model.
Check steps and custom AI tools that set their own Preferred model. Update the ones that should move too; a step's preferred model wins over the thunk-wide setting.
Back in Tests → Test Settings, set Model back to no override.
Re-run the tests once more. This run uses exactly the models the thunk will use for real work.
5. Promote and watch
If the thunk moves through Prototype, Testing, and Production phases, promote the change as usual from the Deploy pane — see Lifecycle of a Thunk. For the first few days, keep an eye on Errors, Token Costs, and Time Analysis in Monitor. If results get worse, switch the model back while you investigate.
When a model is retired
If you do nothing, a thunk whose model is retired moves to a supported replacement from the same family automatically, and a message in the thunk's chat tells you. Switching yourself before the retirement date — using the steps above — means you choose the replacement and see its results before your real work depends on it.