Scenario design
Copy page
Translate product use cases into tasks with explicit context, tool contracts, and measurable success criteria.
A scenario is more than a prompt
Start with a task your customers want an agent to complete. A well-defined scenario makes it possible to tell whether the agent used your product correctly, and to identify the examples or instructions it still needs.
A useful simulation case describes both the user’s request and how an evaluator will recognize success. Flovia’s PromptCase contract couples the natural-language task with a target provider, allowed tool plan, semantic argument expectations, required facts, and answer criteria.
Case attributes such as intent, language, difficulty, persona, and prompt family make it possible to examine which tasks fail and for whom. Persona is scenario context; these evaluations do not claim to reproduce a measured population of human users.
| Layer | What is specified |
|---|---|
| User task | A concrete information need, constraints, and any identifiers the user supplies |
| Candidate context | Whether candidates are supplied, and their metadata and order |
| Tool contract | Allowed operation, input schema, and semantic argument expectations |
| Result contract | Required fields, types, and expected values where applicable |
| Completion rubric | The facts and constraints that the final answer must satisfy |
Compare persona and purpose across a scenario matrix
The current diagnostic analysis includes a full-factorial persona-by-purpose design: three declared personas crossed with five purposes form 15 cells. The analysis retains model-level results and compares personas within matched purposes before aggregating those contrasts. This is a specific diagnostic path; a PromptCase funnel run uses its own declared design.
This structure shows whether a selection pattern is concentrated in one use case or recurs across several contexts. These exploratory comparisons help decide which onboarding examples or documentation variants to test next.
Example: a historical exchange-rate request
Consider a fictional exchange-rate API. The user needs the EUR-to-USD rate for a specific date. The case should test whether the agent chooses historical data, supplies the right base and quote currencies, and gives an answer consistent with the result.
Task: Find the EUR-to-USD rate on 2026-01-15.
Allowed operation: get_historical_rate
Arguments: base=EUR, quote=USD, date=2026-01-15
Required result: base, quote, date, rate
Completion: state the returned rate, currency direction, and date
Environment: live model calls + prepared tool responseThe user request contains the information needed to infer the arguments. The evaluator’s expected answer is kept separate from the agent’s task. A model that calls the latest-rate operation can fail Tool Resolution even if its final prose sounds plausible.
Design for informative failures
- Include concrete target-fit tasks that your product can satisfy.
- Include competitor-fit or negative-control tasks so that always selecting your product is not mistaken for good behavior.
- Vary meaningful task dimensions: operation, entity, date range, language, ambiguity, or user context.
- Use prompt families and repeats to distinguish wording sensitivity from consistency on the same request.
- Keep selection-only controls separate from tasks eligible for the full execution funnel.
More repeats sharpen the view of variability on existing tasks. Broader scenario coverage answers a different question: does the finding hold across more use cases? A report should show both.
Freeze the comparison before running
The evaluation plan records the task set, model configuration, candidate context, execution path, repeats, scoring rules, and measurement window. For an intervention, it also names the exact fields that may change. This turns a proposed test into a defined experimental frame.