A thunk that holds a conversation, such as a chat app or a step that emails a customer, waits for a person to answer. That makes it hard to test on its own: a test run would stop at the first question and sit there.
A simulated user fills that gap. You attach one to a test work item and describe the person the agent is talking to. When the agent sends a message, the simulated user answers it, so the whole conversation plays out without anyone typing. Add assertions to the same work item and you have a repeatable test of the conversation: it runs, the workflow finishes, and the assertions say whether it ended the way you expected.
When to use it
You have a chat app, or any step that asks a person something and waits, and you want to re-run it after every change to the instructions.
A real conversation went wrong and you want to replay it against the fixed thunk. See Start from a real conversation.
You want to check a path that is hard to trigger by hand, such as a customer who gives a vague answer or asks for something the thunk should refuse.
A simulated user only plays the person in the conversation. The agent, its instructions, its tools and its connections all run for real, so a test costs what a real run costs. Outbound email follows the thunk's Test email setting. See Manual testing.
Add a simulated user to a test work item
The work item must be a test work item. See Creating a test work item.
Open the test work item and go to its Tests tab.
Find the Simulated user section for the conversation you want to play. A thunk with several conversations has one section for each.
Fill in the fields below. All of them are optional, but a simulated user with no direction has little to go on.
Re-run the work item, which restarts it from its first step.
Field | What it does |
Simulated person's name | The name shown on their messages. Test only. |
User id (optional) | Attributes their messages to a particular user. Leave it blank to show only the name. |
Max scripted replies per replay | The most replies they will send in one run. The budget starts again each time you re-run. A run stops at the limit, so set it a little higher than the replies you expect. |
Initial message | The opening line of a chat conversation. It replaces the first user message when you re-run the first step, so every run starts the same way. Chat conversations only, and only on a work item that already has a conversation, such as a copy of a real one. A brand-new work item has no conversation yet, so the run waits for a first message that never comes. |
Simulated behavior | A scenario in plain words, or an example script of what they say. They may stay silent once the script or scenario is complete. |
When to end conversation | When they should end the conversation, in plain words. They always send a last message when they do. Ending the conversation also ends the workflow. |
Insert existing conversation fills Simulated behavior with the conversation the work item already has. Use it when you want to start from a real exchange and edit it.
Write direction the agent can follow
The simulated user is itself an AI, so the same rules as for step instructions apply: be specific.
Say exactly what you need. "After the agent answers, reply with exactly: I AM DONE" is more reliable than "finish up when you're satisfied".
Describe the person, not only the lines. "A customer who does not know their order number and gets impatient" tests more than a fixed script does.
Say when to stop. Without When to end conversation, the simulated user keeps going until it runs out of replies or the script.
Keep it to one scenario per work item. Make a second test work item for the second scenario.
Add assertions that check the outcome
A conversation test is only useful if something can fail. Give the work item at least one assertion on what the conversation should leave behind:
Rules on properties the steps fill in, such as a "Reason for contact" the agent records or a flag it sets when the customer is done. Rules are graded without an AI model, so they are fast and give the same answer every time.
AI-evaluated assertions for what the agent said, such as "The agent apologised once and did not promise a refund." The grader reads property values as text, so assert on a property that holds the part of the conversation you care about.
Assertions are graded when the workflow finishes, so the conversation has to reach the end: either through the steps' own finish conditions or because the simulated user ends it. If an assertion never gets a result, the run most likely stopped early. See Troubleshoot a run that stops early.
A simulated user ends the conversation when its script is complete, when it reaches Max scripted replies per replay, or when When to end conversation applies. That ends the workflow, and the work item can be graded before the agent has finished handling the simulated user's last line. If an assertion on something the agent records in answer to that last line fails, wait for the run to settle and then re-grade.
Add one assertion that should fail while you are writing the test, such as a rule for a value the run can never produce, and check that it shows as Failed. Remove it once you have seen it fail. That confirms the test can tell a good run from a bad one.
Run it and read the result
Re-run the test work item. Re-running restarts the work item from its first step, clears the conversation and, for a chat, posts the initial message. A plain run does not post an initial message.
Open the work item while it runs to watch the conversation: the agent's messages, then the simulated user's replies shown under the name you set. When the work item finishes, each assertion shows Passed or Failed. If one failed, open the agent's run to see what it said and which tools it called. See Automated tests for workflow steps.
Re-grading does not replay the conversation. It grades the output the work item already holds, so use it when the work item was graded before the run settled. To test a change to the instructions, re-run.
Start from a real conversation
If a real conversation already shows the case you want, copy it instead of writing the script.
Copy as test row (on the work item's Tests tab, where your account has it) makes a new test work item from the original's starting state. Each conversation gets a simulated user whose script is the original conversation, and the new work item starts in Draft with its own conversation threads. The original is not changed. Run the copy when you are ready.
Insert existing conversation adds a work item's current conversation to the simulated user's behavior so you can edit it.
Copy the conversation first and add the assertions after, because a copy begins with none. A copy replays what was said, so its assertions should describe what you want the thunk to do now, which is not always what it did then.
Troubleshoot a run that stops early
The conversation stops after the first question. The work item is probably not a test work item, or the simulated user has no direction. Check the Tests tab.
The simulated user stops answering. It reached Max scripted replies per replay, or its script says the scenario is over. Raise the limit or extend the script.
An assertion fails although the work item ended up with the right value. The work item was graded when the simulated user ended the conversation, before the agent recorded its last answer. Re-grade it once the run has settled.
The assertions never get a result. The workflow did not finish. Check that the simulated user ends the conversation with the words the step waits for.
The chat does not start with your initial message. You started a plain run. Re-run the work item.
Two runs differ. The simulated user and the agent are both AI, so wording varies between runs. Assert on outcomes, such as a property value, and not on exact phrasing.
Do it from an AI coding assistant
If you build thunks from Claude Code or Codex, your assistant can set this up for you. See Build thunks from Claude Code or Codex and Building thunks with the Builder MCP. Ask it to create a test work item for the conversation with a simulated user and assertions on how it should end, run it, and tell you which assertions passed.