The agent funnel
Copy page
Seven stages connect product discovery to a completed task, with a specific check at every transition.
Measure the whole path
Use the funnel to decide where to intervene. An agent that never chooses your product needs a different fix from one that chooses it but sends the wrong arguments. Find the first stage that failed, inspect its evidence, and pick a change to test.
The seven-stage funnel separates being considered from being used successfully. For a target-fit task, each stage depends on passing the preceding gates. A stage result identifies both the behavior that passed and the next condition that blocked progress.
| Stage | Question | Evidence or check |
|---|---|---|
| Discovery | Does the target appear in the explicit candidate set? | Candidate identity, presence, and reported rank |
| Selection | Does the agent choose the target? | Selected provider, competitor choice, or abstention |
| Tool Resolution | Does it choose an allowed operation? | Tool identity against the case’s allowed plan |
| Invocation | Are the arguments structurally and semantically valid? | Input schema plus task-specific argument expectations |
| Execution | Does the operation succeed on the declared path? | Execution record from the configured tool environment |
| Result Usability | Does the result contain the data the task requires? | Required field presence, type, non-nullness, and declared values |
| Completion | Does the final answer meet the task requirements? | Answer assessed against fixed criteria and returned facts |
Read the first failing stage
If the product is selected but the wrong endpoint is called, inspect tool naming, descriptions, and the operation catalog. If the correct endpoint receives the wrong arguments, inspect parameter semantics and examples. If the result is usable but the final answer fails, inspect the missing or unsupported claims identified in the completion assessment.
An upstream exit leaves downstream stages unobserved in the trace. Cumulative funnel metrics can count the run as not having reached those later stages, while conditional transition metrics use their own eligible populations. These are different views of the same run, not evidence that an uncalled tool failed.
Deterministic checks and the Judge
Tool Resolution, Invocation, and Result Usability are checked against explicit contracts. The completion Judge receives the final answer, task, returned data, and criterion list. It assesses omissions, contradictions, and claims unsupported by the tool result.
Judge output is model-assisted assessment with recorded scoring provenance. If a required Judge result is missing or unavailable, completion assessment is marked missing rather than silently counted as a pass. Some negative-control cases have deterministic terminal decisions and are reported according to their own eligibility.
The Judge returns a completion status, criterion-level assessments, missing requirements, and unsupported claims. The engine validates the response and checks for contradictions between the completion status and those details. The scoring record keeps the rubric and Judge identity together with hashes of the answer and reference inputs.
Example: checking the final answer
For the historical exchange-rate task, a valid tool call is only part of the result. The answer must communicate the returned rate, currency direction, and requested date. These requirements become separate completion criteria.
Task: Find the EUR-to-USD rate on 2026-01-15.
Tool result: base=EUR, quote=USD, date=2026-01-15, rate=1.10
Answer: 1 EUR equals 1.10 USD.| Criterion | Evidence in the answer | Expected assessment |
|---|---|---|
| Returned rate | 1.10 matches the tool result | Satisfied |
| Currency direction | EUR to USD is explicit | Satisfied |
| Requested date | The date is omitted | Missing requirement |
The next edit should address the missing date in the answer: inspect the date field’s meaning and the guidance for using it in the response. The criterion-level record makes that distinction visible even when the preceding tool checks passed.
What Discovery means in a funnel
Discovery records whether the target appears in the model’s explicit candidate response. In a fixed-candidate experiment, the candidates were supplied up front, so appearing there checks behavior under controlled exposure; it does not show that the agent found the product on the web on its own.