The benefit of AI agents is the enormous efficiency that can be gained by the intelligent automation of work. The desired characteristics of AI agent systems are:
Autonomy -- the freedom to decide what to do
Agency -- the freedom to act without seeking approval
In a business environment, there is an additional desired characteristic. Process workflows needs reliable behavior -- predictable, consistent, compliant, and transparent. We know however that AI agents are based on AI large language models ((LLMs), that these models are probabilistic, and can make mistakes. This makes it challenging to apply AI agents to business workflows.
There are inherent tradeoffs between reliable behavior and the degree of AI agency/autonomy. With most AI agent platforms and implementations, the "reliability boundary" excludes most high-value business workflows.
In this article, we describe how the Thunk.AI platform expands the reliability boundary through the concepts of Controlled Agency and Controlled Autonomy.
Defining AI Reliability
A business workflow has to run repeatedly in similar but somewhat different contexts and with somewhat different inputs. There are four expectations of a reliable workflow in such a context:
Correctness: It achieves a desired outcome in each instance.
Process conformance: It follows a prescribed process in each instance.
Consistency: It acts identically or similarly on identical or similar inputs.
Scope: To the extent the environments and inputs vary and to the extent the prescribed process doesn't specify exactly what to do, it makes intelligent decisions as appropriate. This can include invoking a human-in-the-loop engagement.
Sources of unreliability
There are many root causes of AI automation unreliability. They may be broadly classified as follows:
Design errors
Incomplete or inconsistent instructions
Inadequate or inconsistent context or tools
Data errors
Errors (wrong or missing information) in context or tool results
Scope errors
Inputs do not follow expected behavior
Tool results do not follow expected behavior
Execution errors
LLM errors
AI automation platform errors
Tool invocation errors
There is no silver bullet solution for AI agent reliability. Instead, a set of core principles need to be applied, taking a "defense in depth" approach to ensure that relatively few errors occur. When errors do occur, they should be detected and corrected.
Design Principles during the Planning Phase
At the level of entire AI workflow/thunk, there is an explicit planning phase that precedes the execution of the workflow.
The design and testing environment of Thunk.AI identifies design errors and helps mitigate them.
The purpose of planning is to capture appropriate intent (steering and control) -- both at the coarse level of of the whole workflow and at the finer level of individual granular tasks that the AI agent may be asked to run.
Before any work is done, its intent and plan are explicitly articulated --- either by a human or by AI or by a combination thereof. For work that should be repetitive, the intent and plan are persisted and reused for consistency.
Work is divided into hierarchical units of smaller granularity, reducing the scope of agency and autonomy needed for each unit of work.
Each granular unit of work is individually configured for an appropriate level of autonomy and agency.
Planning is a joint activity between the thunk designer (human) and a design-time AI agent. In this phase, the platform provides many exploratory options (greater autonomy and agency in a prototype period) and the thunk designer can make choices that achieve the right balance between control and flexibility. The thunk designer can choose to define workflows with a lot of control (over agency and autonomy), or workflows with very little control.
Controlled Autonomy
Principle of static planning: when work is explicitly planned and a plan ( a sequence of steps) is articulated, it provides a process guideline for repeated consistent and conformant execution. More detailed process guidelines lead to more consistent and conformant results.
Principle of minimal granularity: the broader the instructions given to the LLM and the broader the context it has to interpret, the more variability there will be in the results. Therefore, to achieve reliability, the platform gives the thunk designer the ability to specify the "tightest" (most granular) instructions and context.
Controlled Agency
Principle of maximal constraints: the broader the possible set of responses from an LLM in a particular granular context, the greater the variability of those responses. Therefore, to achieve reliability, the platform gives the thunk designer the ability to independently restrict the LLM to the "tightest" (most limited) set of allowed responses in each granular work context.
Principle of minimal capability: LLMs interact with the business environment through "tools" to read or update content. These tools are provided by the Thunk.AI agent platform. To achieve reliability, the platform gives the thunk designer the ability to independently specify the "tightest" set of tools for each granular work context.
The workflow plan in a thunk has many granular components to it: it has a sequence of workflow steps and it defines schematized state that the workflow should maintain. Every step of work is granular and the degree of granularity is in the control of the thunk designer. The actual choice of workflow step granularity depends on the thunk designer reflecting the needs of the particular business process. Tools can be defined in a granular fashion, including intelligent AI tools that can wrap and invoke other tools. Workflow schematized state can be defined in a granular fashion, with individual schematized properties specifying their own detailed descriptions, types, and constraints.
Finer granularity leads to more specific process. Coarser granularity offers more flexibility in dealing with dynamic environments.
Each granular step of the workflow includes detailed AI agent instructions. These instructions have four elements -- steering (what it should do, includes examples), grounding (what input information is available to it), agency (what tools it is allowed to use), and validation. The limits on grounding information and agency tools reflect the Principle of Maximal Constraints and the Principle of Minimal Capability.
Design Principles during the Execution Phase
AI workflow execution is orchestrated by the Thunk.AI platform. This execution phase is primarily an automated phase of execution where the orchestration engine and the AI agents can choose to involve human-in-the-loop engagement if needed. The reliability of AI agents during this execution phase is largely based on the choices made during the planning phase.
The Thunk.AI platform implementation detects and minimizes the impact of scope errors, LLM errors, and tool errors.
At runtime, the AI agent engages with the LLM in an iterative conversational loop, but only allows it to respond by invoking one of the tools provided. Free text responses (one of the greatest sources of randomization) are explicitly disallowed.
All autonomous decisions are verified before being committed.
All agentic actions are verified before being executed.
All non-deterministic agentic actions are verified after being executed.
Controlled Autonomy
Principle of dynamic planning: most individual AI agent tasks require multiple iterations and tool invocations. Dynamic micro-planning of individual AI agent tasks increases the reliability of agent execution.
Principle of explanation: The platform requires the LLM to provide reasoning for its responses that is consistent with the plan and goals of the workflow. This explicit explanation forces greater consistency with the intended workflow process.
Controlled Agency
Principle of checkpointing: The platform requires the LLM to update schematized state with its partial progress or results. This improves alignment with the desired outcomes, increases reliability, and makes the work transparent to users.
Principle of verification: The platform checks every LLM response for validity. This creates an opportunity to correct and refine results. There can be a variety of checks, including deterministic checks (eg: for schema conformance), checks implemented by LLM calls (eg: for semantic conformance to constraints), and human-in-the-loop verification.
Since every individual task execution involves (a) potentially multiple iterations with the LLM, (b) multiple tool calls, (c) variable environments and inputs, the Thunk.AI platform always starts with "micro-planning" the task. This reflects the Principle of Dynamic Planning. The dynamic micro-plan is itself constrained by the available tools and by the data bindings specified during the initial planning phase, so it creates a further level of detail for subsequent execution. By explicitly requiring articulating of the micro-plan, the AI agent platform steers subsequent stages of the iteration in a consistent direction.
Every response from the LLM is a tool call with arguments and importantly, an explanation. This reflects the Principle of Explanation. There are three benefits of these explanations. One important benefit is that the explanation increases the alignment of the LLM's immediate response with the desired goal and plan. In effect, the requirement to provide a rational explanation acts as a constraint on the response of the LLM. A second benefit is that the explanation reinforces alignment of subsequent LLM responses with the plan. Finally, the explanations are useful for human validation.
The platform steers the AI agent to checkpoint its work and update the workflow state as work progresses. Since the workflow state is schematized and structured, this imposes constraints on the output of the LLM. This reflects the Principle of Checkpointing. Just like the principle of explanation, this increases alignment of the LLM's responses with the desired outcomes.
Finally, every response of the LLM, every tool result, and every workflow step is checked for consistency. This reflects the Principle of Verification. If the verification identifies inconsistencies or inaccuracy, these are fed back to the LLM for correction. There are many kinds of verification. Conformance with schema and structure are the most obvious and deterministic. More subjective verification is very valuable also -- for example, whether an LLM response conforms to policy, or whether an LLM response satisfies the descriptions of tool arguments or of workflow state property descriptions.
End-to-end workflow reliability
In practice, the end-to-end reliability of AI agent automation depends on a combination of four factors:
The nature of the workflow process --- how specific the process is and how much "intelligent" decision-making is expected from AI agents to handle variability of inputs and contexts.
The consistency and thoroughness of instruction details provided by the thunk designer during the planning phase
The inherent reliability of the AI agent platform in following the plan, adhering to provided instructions, and controlling the LLM's responses towards the desired outcomes
The actual degree of variability of the runtime workload and the quality of the data and context provided at runtime
The Thunk.AI platform provides the design mechanisms for the thunk designer to provide intentional granular instructions in the workflow definition. The design environment helps the thunk designer define reliable instructions. The execution environment ensures that AI automation conforms to the instructions, escalating to human-in-the-loop approval when required. Finally, the platform also provides mechanisms to measure and audit outcomes, and improve reliability based on ongoing feedback.
One quantifiable measure of the AI reliability achieved by the Thunk.AI platform is published in the HiFi Reliability Benchmark.