Skip to main content
When a developer asks a coding agent to build with your product, the agent has to find your docs, install your package, set it up and make it work. Agent Experience runs those tasks for you and shows where agents get stuck, so you know what to fix in your product and its docs.

​
How it works

You write experience evaluations. Each one is a realistic task for a coding agent plus a checklist of what a correct result looks like:
  • Task: what a developer would hand the agent. State every requirement the result could fail on.
  • Criteria: each one something a reviewer can answer yes or no by reading the finished work, in the order they would check them.
Each run gives one agent with one model the task alone, in a starter repository. The agent never sees your criteria.

​
Add an evaluation

Only an Admin can add, edit, run or delete evaluations.
1

Open the evaluation list

Open Agent Experience in the sidebar and scroll to Evaluations.
2

Start a new evaluation

Click Add evaluation. The Add experience evaluation panel opens.
3

Write the task

Fill in Task the way a developer would ask a coding agent.
4

Group it

Choose a Topic, and any Tags. See Topics and tags.
5

Add criteria

The first criterion is ready to fill in. Click Add criterion for each one after it. Use the arrows to reorder them, and the bin to remove one.
6

Choose how it runs

Pick the Repository runs start from. It starts on the default starter repository. Tick the Agents and models to run it with. See Executions.
7

Save

Click Add evaluation.
Write criteria a person could check without asking the agent anything. “The signup form sends a welcome email” is checkable; “the code is good” is not.
An evaluation without a topic, or without any criteria, is flagged as needing attention and cannot run until you fix it. In the list, No criteria marks one with none. To find them all, turn on Needs attention above the list. To change or remove an evaluation, open the actions menu on its row and choose Edit or Delete. Deleting removes the evaluation from every list, along with its criteria and tag assignments. Editing changes future runs only.

​
Run an evaluation

Open an evaluation’s actions menu and choose Run, or open the evaluation and click Run. Agentrail starts one run for each agent and model it has. Evaluations also run on their own, on the schedule set in Run settings. See Executions.

​
Read the results

The top of the Agent Experience page sums up runs in the chosen view. Choose a period (Last 30 days by default), an agent and model (All agents), a tag (All tags), or a topic tab. Each figure is a percentage with the runs behind it; No measurable results means nothing could be counted.
Five steps with your product: Read the docs, Installed, Configured, Integrated (the code uses your product) and Checked (the agent’s own check, such as its tests or build, passed after its last change). Each step counts out of every run, with the share of the step before that reached it.
How often agents integrated your product and checked their work, per day, week or month depending on the period.
What the agent said it did: Claimed done, Claimed partly done, Said not done or Stopped to ask. Beside it, Check passed, of those that claimed done, and Claimed done without a passing check: agents that said they were done without proving it.
Which product the code used: Your product, A competitor or Neither.
Steps the agent said a person must take before the feature works: Add an API key, Create an account, Verify a domain or sender, Install an app or integration, Configure the dashboard and Run a command. Open the quotes to read the agent’s words.
Failed steps sorted into Problems in your product (API or SDK errors, documentation problems, package install failures, version or dependency conflicts, missing credentials or account) and Everything else. A run counts once per kind. Problems, run by run quotes each one, linked to its step in the run’s log.
Per agent and model: Runs, Average time, Average tokens and Average cost.
Results count runs that Finished, Failed or Timed out. Runs left out are listed under Not counted with the reason, the same way as on Agent Preference.

​
An evaluation’s page

Click an evaluation’s task in the list to open it. It shows the task, its numbered criteria, its settings, Results for this evaluation over the last 30 days across every agent, and every run of it under Executions.

​
Limits

​
Who can do what

See People and roles.

Agent Preference

See which products agents recommend and choose.

Executions

Start runs, set the schedule and read each run.