Evals are automated tests that simulate conversations with your agent and assert on the results. You write a conversation script with expected behaviors, and the eval runner plays it against your running agent and reports pass/fail.
Create an eval
Create .eval.ts files in the evals/ directory at your agent root:
import { Eval } from "@botpress/runtime"
export default new Eval({
name: "greeting",
description: "Agent should greet the user",
conversation: [
{
user: "Hello",
assert: {
response: [
{ llm_judge: "The response is a friendly greeting" },
],
},
},
],
})
Eval instances as the default export or named exports.
Conversation turns
An eval’s conversation is a sequence of turns. Each turn can send a user message, fire an event, or assert the agent stays silent, plus optionally assert on what happens.
Send user messages:
conversation: [
{
user: "What's the weather in Paris?",
assert: {
response: [{ contains: "Paris" }],
tools: [{ called: "getWeather", params: { city: { equals: "Paris" } } }],
},
},
{
user: "And in London?",
assert: {
response: [{ contains: "London" }],
},
},
]
{
event: {
type: "order.placed",
payload: { orderId: "ord-123", total: 49.99 },
},
assert: {
response: [{ contains: "order" }],
},
}
{
user: "ok thanks",
expectSilence: true,
}
Assertions
Each turn’s assert block can hold any combination of the assertion types below.
Response
Assert on the text of the agent’s reply:
| Assertion | Description |
|---|
{ contains: "text" } | Response includes this string |
{ not_contains: "text" } | Response does not include this string |
{ matches: "regex" } | Response matches this regex pattern |
{ similar_to: "text" } | Response is semantically similar to this text |
{ llm_judge: "criteria" } | An LLM evaluates whether the response meets the criteria (see LLM judge) |
| Assertion | Description |
|---|
{ called: "toolName" } | Tool was called |
{ called: "toolName", params: { key: { equals: "value" } } } | Tool was called with matching parameters |
{ not_called: "toolName" } | Tool was not called |
{ call_order: ["tool1", "tool2"] } | Tools were called in this order |
params to match tool arguments:
| Operator | Example |
|---|
{ equals: value } | Exact match |
{ contains: "text" } | String contains |
{ not_contains: "text" } | String does not contain |
{ matches: "regex" } | Regex match |
{ in: [1, 2, 3] } | Value is in array |
{ exists: true } | Field exists |
{ gte: 10 } | Greater than or equal |
{ lte: 100 } | Less than or equal |
State
Assert on state changes after a turn:
assert: {
state: [
{ path: "messageCount", changed: true },
{ path: "topic", equals: "weather" },
],
}
Tables
Assert on table data:
assert: {
tables: [
{ table: "OrderTable", row_exists: { userId: { equals: "user123" } } },
{ table: "OrderTable", row_count: { gte: 1 }, where: { status: { equals: "pending" } } },
],
}
Workflows
Assert on workflow state:
assert: {
workflow: [
{ name: "processOrder", entered: true },
{ name: "processOrder", completed: true },
],
}
Timing
Assert on response time:
assert: {
timing: [
{ response_time: { lte: 5000 } },
],
}
LLM judge
Use llm_judge when the right answer isn’t a fixed string. You describe what a good response looks like in plain English, and an LLM scores the agent’s reply against it. It’s the right choice for subjective checks like tone, intent, or whether the agent understood the user:
assert: {
response: [
{ llm_judge: "The response is polite and professional" },
{ llm_judge: "The response correctly identifies the user's issue" },
],
}
Setup and outcome
setup runs before the first turn. outcome runs after the last.
Setup
Seed state or trigger a workflow before the conversation starts:
new Eval({
name: "returning-user",
setup: {
state: {
user: { name: "Alice", visitCount: 5 },
conversation: { topic: "billing" },
},
},
conversation: [
{
user: "Hi",
assert: {
response: [{ contains: "Alice" }],
},
},
],
})
setup: {
workflow: {
trigger: "onboarding",
input: { userId: "user123" },
},
},
Outcome
Assert on state, tables, or workflows after the entire conversation completes. Outcome assertions use the same shapes as turn assertions:
new Eval({
name: "full-flow",
conversation: [
{ user: "Create a ticket for VPN issues" },
{ user: "Set priority to high" },
],
outcome: {
tables: [
{ table: "TicketTable", row_exists: { priority: { equals: "high" } } },
],
workflow: [
{ name: "createTicket", completed: true },
],
},
})
agent.config.ts:
evals: {
judgeModel: "openai:gpt-4o",
judgePassThreshold: 3,
idleTimeout: 15000,
},
| Option | Type | Description |
|---|
judgeModel | string | Model for llm_judge assertions. Defaults to "fast" |
judgePassThreshold | number (1–5) | Minimum score for llm_judge to pass. Defaults to 3 |
idleTimeout | number (ms) | How long to wait for the agent to respond. Defaults to 15000 |
judgePassThreshold or idleTimeout on a specific eval:
new Eval({
name: "quality-check",
options: {
judgePassThreshold: 4,
idleTimeout: 30000,
},
conversation: [/* ... */],
})
Organize evals
Evals have two organizing fields that pair with CLI filters: tags and type.
Tags are free-form labels. Use them to group evals you want to run together (smoke tests, critical paths, slow suites):
new Eval({
name: "greeting",
tags: ["smoke", "core"],
conversation: [/* ... */],
})
adk evals --tag smoke # Run only evals tagged "smoke"
capability: tests that a feature works as designedregression: reproduces a past bug so it doesn’t come back
new Eval({
name: "fix-for-ticket-priority-bug",
type: "regression",
conversation: [/* ... */],
})
adk evals --type regression # Run only regression evals
Run evals
From the CLI:
adk evals # Run all evals
adk evals greeting # Run a specific eval by name
adk evals --tag smoke # Run evals with a specific tag
adk evals --type regression # Run only regression evals
adk evals -v # Show full details, not just failures
adk evals runs # List recent runs
adk evals runs --latest # Show the latest run
adk evals runs <run-id> # Show a specific run
Last modified on April 24, 2026
