Simulation engine
Copy page
A controlled environment for observing model decisions, tool invocation, and task completion.
From a task to an observable agent journey
Find out whether an agent's difficulty comes from choosing your product, calling the right operation, or using the data it gets back. The engine records each step, so your team can trace a failure to the documentation or interface that needs to change.
Flovia’s simulation engine turns an evaluation plan into identifiable runs. Each run binds a task to a model, candidate context, execution path, repeat, and input versions. The staged evaluation uses PromptCase: a task definition that includes the allowed operation, argument expectations, required result fields, and completion criteria.
The engine makes the agent journey observable by separating decisions from validation. Model calls produce candidate choices, tool calls, and answers. Deterministic checks validate tool identity, arguments, and result structure. A separately configured model-assisted Judge assesses the final answer against the task rubric.
- 01ModelDiscover + select
- 02ContractResolve + validate
- 03EnvironmentExecute + inspect
- 04AssessmentAnswer + score
The model-call sequence
| Stage | Input | Observed output |
|---|---|---|
| Discovery and selection | User task and the declared candidate context | Structured candidate list, rank, and selected provider |
| Tool invocation | Task, tool descriptions, and input schemas | A function-call name and structured arguments |
| Answer generation | Task and normalized tool result | A final answer grounded in the returned data |
| Completion assessment, when configured | Answer, tool result, and fixed completion criteria | Criterion-level verdicts, missing facts, and unsupported claims |
The engine checks the model’s decisions at each stage against the conditions for proceeding. If a condition is not met, the run stops and records the failure and its supporting evidence.
Model calls and tool execution are separate
In the PromptCase Live LLM / Mock Tool path, the specified model endpoint is called for real, while the target tool runs against prepared, versioned responses called fixtures. Holding the response environment stable lets an experiment focus on how agent decisions, invocation, and answer grounding change.
An operation binding maps the selected tool and its arguments to a request contract. The fixture adapter builds the request representation locally, matches it against the tool and argument conditions, and returns the corresponding response. It does not send that request to the target provider. This makes invocation errors and answer-grounding failures inspectable in a stable tool environment.
| Execution path | What it establishes |
|---|---|
| Live model call | Behavior observed from the requested model endpoint in the measurement window |
| Fixture-backed tool execution | Behavior against the declared response contract and prepared result |
| Offline test with prepared model responses | Pipeline and artifact checks; not an observation of live model behavior |
Reproducibility through versioned inputs
Prompt cases, tool catalogs, operation bindings, fixtures, and scoring prompts are versioned or hash-bound. The engine checks their relationships before execution: expected tools must exist in the catalog, semantic argument expectations must resolve, and every required result fact must have a defined contract.
Runs retain their identity, stage records, errors, and scoring provenance. Bounded concurrency, checkpoints, and resume support let the engine execute a declared run set while preserving individual observations. Reproducibility here means traceable conditions and repeatable procedures; a later call to a model endpoint may still produce a different answer.
Design coverage and comparison conditions
Flovia treats user purpose, persona, model, and candidate presentation as experimental factors. Evaluation cases are built around the behavior being studied, with explicit choices about which conditions change and which stay fixed.
- Scenario matrices: cross purposes with personas in diagnostic studies and compare results by model and purpose. Identify the conditions where the product is less likely to be selected, then choose the descriptions or usage examples to test next.
- Candidate-position balancing: supported comparison studies rotate candidates using a cyclic Latin square. Each candidate appears in each position, and baseline and variant are compared within the same order.
- Stable tool responses: versioned fixtures hold the returned data constant for a task. This controls response variation while the evaluation examines how changes to descriptions or tool information affect decisions, invocation, and answers.