# What is Flovia?

Understand how agents discover, choose, and use your product. Turn the point of failure into your next improvement.

Source: https://flovia.dev/docs/

## An evaluation loop for agent experience

Flovia is an Agent Experience Optimizer for teams that build APIs and MCP tools. It evaluates whether an AI agent can choose your product, use it correctly, and complete the user’s task. Each evaluation connects a point of failure to a concrete fix and a retest.

An agent can mention your product and still call the wrong endpoint or supply the wrong arguments. Flovia follows the journey past visibility to find the API, tool-description, or documentation change that would unblock successful use.

AX Simulation is Flovia’s evaluation methodology. It turns agent behavior into a controlled experiment. It combines task-based scenarios, native function calling, contract validation, and stage-level scoring to locate where the agent gets stuck between choosing a product and completing a task. Paired experiments then compare a targeted change under matched conditions. Your private dashboard connects the evidence to a fix and its retest.

To use Flovia, request an evaluation and agree on its scope with the team. Results, diagnosis, recommended changes, and retests are delivered through a private dashboard.

- **Define:** Tasks + success criteria
- **Simulate:** Models + tool environment
- **Diagnose:** Stages + evidence
- **Improve:** Change + controlled retest

## Example: improving API guidance

For a blockchain data API, we compared AI answers before and after a documentation update. Before the update, answers either failed to identify how to retrieve transaction IDs only or suggested a capability intended for another purpose. After the update, they explained the correct method.

For real-time streaming, answers after the update also correctly explained that getting connection details does not start the stream: the developer needs to run the connection code.

- [Explore the documentation improvement](https://flovia.dev/docs/improvement-loop/): See the before-and-after comparison for each capability.

## Review your results in the dashboard

Review the evaluation summary, prioritized changes, supporting run evidence, and implementation prompts. Understand why a change is recommended before applying it.

- [Explore the evaluation dashboard](https://flovia.dev/docs/improvement-loop/#dashboard-example): Follow one example from overview to finding and evidence.

## Technical methods that inform product decisions

The engine combines experimental design, executable task contracts, and traceable measurement. Each evaluation uses the methods suited to its question and execution path.

| Method | What it makes observable | Product decision |
| --- | --- | --- |
| Paired comparisons + Latin-square candidate balancing | Behavior before and after a description change, with task and candidate position controlled in supported studies | Which description deserves a retest across more use cases? |
| Schema + semantic argument validation | Whether the chosen operation and its arguments satisfy the task contract | Should the tool name, parameter guidance, or operation boundary change? |
| Result checks + criterion-based answer assessment | Where returned data or the final answer falls short of the task | Which fields or interpretation guidance would unblock completion? |
| Cluster bootstrap + versioned evidence | Uncertainty across task groups where supported, and the inputs behind each finding | How much evidence supports the result, and what should be tested next? |

## From evaluation request to improvement

Your first evaluation is free. Start with a 30-minute call to choose a task an agent should complete using your API or MCP. Flovia delivers the initial results in your private dashboard.

1. Prepare your context. Share your product and documentation URLs, a representative task, and any points of confusion you already know about.
2. Scope the evaluation in a 30-minute call. Your team and Flovia agree on the task, models, environment, and success criteria.
3. Review the results. Flovia runs the evaluation and shows, in your dashboard, where agents struggled, the evidence behind each finding, and the recommended changes.
4. Apply a change. Your team reviews the recommendation and implements it. Enterprise customers can also use a GitHub-connected, semi-automated workflow while retaining review and deployment control.
5. Check the improvement. Flovia retests under comparable conditions. Review the before-and-after evidence together and decide what to improve next. Retesting follows the scope of your plan.

## What to prepare

| Input | Why it matters |
| --- | --- |
| Product documentation and interface descriptions | Define what the agent can learn and which operations are available. |
| Representative user requests | Anchor the evaluation in tasks that matter to your product. |
| Expected behavior and required facts | Make successful completion testable. |
| Relevant alternatives and constraints | Make selection meaningful, including cases where another tool is a better fit. |
| A proposed change, if you have one | Define a baseline and a variant for a controlled comparison. |

Request an evaluation through the website. Flovia runs the engine and delivers results, failure analysis, remediation guidance, and retest findings in a private dashboard. Your team reviews and applies the change in its own environment.

Before the run, agree on the model endpoints and the completion Judge configuration. Separately, decide who prepares the tool-response data, which tasks and repeats to run, and when to review the results. Recording these in the evaluation plan makes preparation responsibilities and the decision the result will support clear to everyone.

## Prepare your evaluation request

Evaluation brief · copy and fill in

```text
Product / documentation URL:
Task an agent should complete:
Expected result and required facts:
Relevant alternatives:
Known failure or proposed change:
Constraints and questions for Flovia:
```

- [Prepare a brief with your agent](https://flovia.dev/skills/prepare-flovia-evaluation/SKILL.md): A downloadable skill for reading the docs and drafting a request. It does not run a simulation.
- [Plans and pricing](https://flovia.dev/pricing): See the current plans and what they include.
- [Discuss an evaluation](https://flovia.dev/contact): Share your product and task with the team. Confirm scope and data-handling needs before sharing private material.

## Explore the methodology

- [Design a scenario](https://flovia.dev/docs/scenario-design/): Turn one product decision into a task with testable success criteria.
- [Inside the simulation engine](https://flovia.dev/docs/simulation-engine/): Follow the model calls, tool contracts, and scoring path.
- [The seven-stage agent funnel](https://flovia.dev/docs/agent-funnel/): See what each stage measures and how failures are located.
- [Controlled experiments](https://flovia.dev/docs/controlled-experiments/): Understand pairing, candidate rotation, and comparison design.
- [Metrics and evidence](https://flovia.dev/docs/metrics-and-evidence/): Read denominators, uncertainty, and the evidence behind a result.

---

# Simulation engine

A controlled environment for observing model decisions, tool invocation, and task completion.

Source: https://flovia.dev/docs/simulation-engine/

## From a task to an observable agent journey

Find out whether an agent's difficulty comes from choosing your product, calling the right operation, or using the data it gets back. The engine records each step, so your team can trace a failure to the documentation or interface that needs to change.

Flovia’s simulation engine turns an evaluation plan into identifiable runs. Each run binds a task to a model, candidate context, execution path, repeat, and input versions. The staged evaluation uses PromptCase: a task definition that includes the allowed operation, argument expectations, required result fields, and completion criteria.

The engine makes the agent journey observable by separating decisions from validation. Model calls produce candidate choices, tool calls, and answers. Deterministic checks validate tool identity, arguments, and result structure. A separately configured model-assisted Judge assesses the final answer against the task rubric.

- **Model:** Discover + select
- **Contract:** Resolve + validate
- **Environment:** Execute + inspect
- **Assessment:** Answer + score

## The model-call sequence

| Stage | Input | Observed output |
| --- | --- | --- |
| Discovery and selection | User task and the declared candidate context | Structured candidate list, rank, and selected provider |
| Tool invocation | Task, tool descriptions, and input schemas | A function-call name and structured arguments |
| Answer generation | Task and normalized tool result | A final answer grounded in the returned data |
| Completion assessment, when configured | Answer, tool result, and fixed completion criteria | Criterion-level verdicts, missing facts, and unsupported claims |

The engine checks the model’s decisions at each stage against the conditions for proceeding. If a condition is not met, the run stops and records the failure and its supporting evidence.

## Model calls and tool execution are separate

In the PromptCase Live LLM / Mock Tool path, the specified model endpoint is called for real, while the target tool runs against prepared, versioned responses called fixtures. Holding the response environment stable lets an experiment focus on how agent decisions, invocation, and answer grounding change.

An operation binding maps the selected tool and its arguments to a request contract. The fixture adapter builds the request representation locally, matches it against the tool and argument conditions, and returns the corresponding response. It does not send that request to the target provider. This makes invocation errors and answer-grounding failures inspectable in a stable tool environment.

| Execution path | What it establishes |
| --- | --- |
| Live model call | Behavior observed from the requested model endpoint in the measurement window |
| Fixture-backed tool execution | Behavior against the declared response contract and prepared result |
| Offline test with prepared model responses | Pipeline and artifact checks; not an observation of live model behavior |

> **Reading an execution result.** A successful fixture-backed execution establishes success on the simulated path. It does not measure a provider’s live API availability, authentication, latency, or production reliability.

## Reproducibility through versioned inputs

Prompt cases, tool catalogs, operation bindings, fixtures, and scoring prompts are versioned or hash-bound. The engine checks their relationships before execution: expected tools must exist in the catalog, semantic argument expectations must resolve, and every required result fact must have a defined contract.

Runs retain their identity, stage records, errors, and scoring provenance. Bounded concurrency, checkpoints, and resume support let the engine execute a declared run set while preserving individual observations. Reproducibility here means traceable conditions and repeatable procedures; a later call to a model endpoint may still produce a different answer.

## Design coverage and comparison conditions

Flovia treats user purpose, persona, model, and candidate presentation as experimental factors. Evaluation cases are built around the behavior being studied, with explicit choices about which conditions change and which stay fixed.

- Scenario matrices: cross purposes with personas in diagnostic studies and compare results by model and purpose. Identify the conditions where the product is less likely to be selected, then choose the descriptions or usage examples to test next.
- Candidate-position balancing: supported comparison studies rotate candidates using a cyclic Latin square. Each candidate appears in each position, and baseline and variant are compared within the same order.
- Stable tool responses: versioned fixtures hold the returned data constant for a task. This controls response variation while the evaluation examines how changes to descriptions or tool information affect decisions, invocation, and answers.

---

# Scenario design

Translate product use cases into tasks with explicit context, tool contracts, and measurable success criteria.

Source: https://flovia.dev/docs/scenario-design/

## A scenario is more than a prompt

Start with a task your customers want an agent to complete. A well-defined scenario makes it possible to tell whether the agent used your product correctly, and to identify the examples or instructions it still needs.

A useful simulation case describes both the user’s request and how an evaluator will recognize success. Flovia’s PromptCase contract couples the natural-language task with a target provider, allowed tool plan, semantic argument expectations, required facts, and answer criteria.

Case attributes such as intent, language, difficulty, persona, and prompt family make it possible to examine which tasks fail and for whom. Persona is scenario context; these evaluations do not claim to reproduce a measured population of human users.

| Layer | What is specified |
| --- | --- |
| User task | A concrete information need, constraints, and any identifiers the user supplies |
| Candidate context | Whether candidates are supplied, and their metadata and order |
| Tool contract | Allowed operation, input schema, and semantic argument expectations |
| Result contract | Required fields, types, and expected values where applicable |
| Completion rubric | The facts and constraints that the final answer must satisfy |

## Compare persona and purpose across a scenario matrix

The current diagnostic analysis includes a full-factorial persona-by-purpose design: three declared personas crossed with five purposes form 15 cells. The analysis retains model-level results and compares personas within matched purposes before aggregating those contrasts. This is a specific diagnostic path; a PromptCase funnel run uses its own declared design.

This structure shows whether a selection pattern is concentrated in one use case or recurs across several contexts. These exploratory comparisons help decide which onboarding examples or documentation variants to test next.

## Example: a historical exchange-rate request

Consider a fictional exchange-rate API. The user needs the EUR-to-USD rate for a specific date. The case should test whether the agent chooses historical data, supplies the right base and quote currencies, and gives an answer consistent with the result.

Illustrative evaluation brief · not a runnable configuration

```text
Task: Find the EUR-to-USD rate on 2026-01-15.
Allowed operation: get_historical_rate
Arguments: base=EUR, quote=USD, date=2026-01-15
Required result: base, quote, date, rate
Completion: state the returned rate, currency direction, and date
Environment: live model calls + prepared tool response
```

The user request contains the information needed to infer the arguments. The evaluator’s expected answer is kept separate from the agent’s task. A model that calls the latest-rate operation can fail Tool Resolution even if its final prose sounds plausible.

## Design for informative failures

- Include concrete target-fit tasks that your product can satisfy.
- Include competitor-fit or negative-control tasks so that always selecting your product is not mistaken for good behavior.
- Vary meaningful task dimensions: operation, entity, date range, language, ambiguity, or user context.
- Use prompt families and repeats to distinguish wording sensitivity from consistency on the same request.
- Keep selection-only controls separate from tasks eligible for the full execution funnel.

More repeats sharpen the view of variability on existing tasks. Broader scenario coverage answers a different question: does the finding hold across more use cases? A report should show both.

## Freeze the comparison before running

The evaluation plan records the task set, model configuration, candidate context, execution path, repeats, scoring rules, and measurement window. For an intervention, it also names the exact fields that may change. This turns a proposed test into a defined experimental frame.

- [Design a controlled comparison](https://flovia.dev/docs/controlled-experiments/): Pair baseline and variant while holding the surrounding context fixed.

---

# The agent funnel

Seven stages connect product discovery to a completed task, with a specific check at every transition.

Source: https://flovia.dev/docs/agent-funnel/

## Measure the whole path

Use the funnel to decide where to intervene. An agent that never chooses your product needs a different fix from one that chooses it but sends the wrong arguments. Find the first stage that failed, inspect its evidence, and pick a change to test.

The seven-stage funnel separates being considered from being used successfully. For a target-fit task, each stage depends on passing the preceding gates. A stage result identifies both the behavior that passed and the next condition that blocked progress.

| Stage | Question | Evidence or check |
| --- | --- | --- |
| Discovery | Does the target appear in the explicit candidate set? | Candidate identity, presence, and reported rank |
| Selection | Does the agent choose the target? | Selected provider, competitor choice, or abstention |
| Tool Resolution | Does it choose an allowed operation? | Tool identity against the case’s allowed plan |
| Invocation | Are the arguments structurally and semantically valid? | Input schema plus task-specific argument expectations |
| Execution | Does the operation succeed on the declared path? | Execution record from the configured tool environment |
| Result Usability | Does the result contain the data the task requires? | Required field presence, type, non-nullness, and declared values |
| Completion | Does the final answer meet the task requirements? | Answer assessed against fixed criteria and returned facts |

## Read the first failing stage

If the product is selected but the wrong endpoint is called, inspect tool naming, descriptions, and the operation catalog. If the correct endpoint receives the wrong arguments, inspect parameter semantics and examples. If the result is usable but the final answer fails, inspect the missing or unsupported claims identified in the completion assessment.

An upstream exit leaves downstream stages unobserved in the trace. Cumulative funnel metrics can count the run as not having reached those later stages, while conditional transition metrics use their own eligible populations. These are different views of the same run, not evidence that an uncalled tool failed.

## Deterministic checks and the Judge

Tool Resolution, Invocation, and Result Usability are checked against explicit contracts. The completion Judge receives the final answer, task, returned data, and criterion list. It assesses omissions, contradictions, and claims unsupported by the tool result.

Judge output is model-assisted assessment with recorded scoring provenance. If a required Judge result is missing or unavailable, completion assessment is marked missing rather than silently counted as a pass. Some negative-control cases have deterministic terminal decisions and are reported according to their own eligibility.

The Judge returns a completion status, criterion-level assessments, missing requirements, and unsupported claims. The engine validates the response and checks for contradictions between the completion status and those details. The scoring record keeps the rubric and Judge identity together with hashes of the answer and reference inputs.

## Example: checking the final answer

For the historical exchange-rate task, a valid tool call is only part of the result. The answer must communicate the returned rate, currency direction, and requested date. These requirements become separate completion criteria.

Illustrative answer check · fictional exchange-rate API

```text
Task: Find the EUR-to-USD rate on 2026-01-15.
Tool result: base=EUR, quote=USD, date=2026-01-15, rate=1.10
Answer: 1 EUR equals 1.10 USD.
```

| Criterion | Evidence in the answer | Expected assessment |
| --- | --- | --- |
| Returned rate | 1.10 matches the tool result | Satisfied |
| Currency direction | EUR to USD is explicit | Satisfied |
| Requested date | The date is omitted | Missing requirement |

The next edit should address the missing date in the answer: inspect the date field’s meaning and the guidance for using it in the response. The criterion-level record makes that distinction visible even when the preceding tool checks passed.

- [Define the task and its criteria](https://flovia.dev/docs/scenario-design/): Start with the historical exchange-rate example and its required facts.
- [Choose the next change](https://flovia.dev/docs/improvement-loop/): Connect the first failing stage to a concrete edit and retest.

## What Discovery means in a funnel

Discovery records whether the target appears in the model’s explicit candidate response. In a fixed-candidate experiment, the candidates were supplied up front, so appearing there checks behavior under controlled exposure; it does not show that the agent found the product on the web on its own.

- [Read the metrics correctly](https://flovia.dev/docs/metrics-and-evidence/): Understand cumulative rates, conditional transitions, and missing observations.

---

# Controlled experiments

Isolate a product-surface change and compare agent behavior under a declared experimental design.

Source: https://flovia.dev/docs/controlled-experiments/

## Choose the question before the test

Use a controlled comparison to decide whether a change to your documentation or tool descriptions helps agents choose and use your product. Define the question first, then change only the fields needed to answer it.

| Design | Question it answers |
| --- | --- |
| No supplied candidate list (separate design) | Which providers does the model propose on its own? The current PromptCase discovery prompt can ask for candidates without a supplied lineup. This measures model-reported candidates, not web retrieval; a search-based evaluation needs a separately configured retrieval path and its execution evidence. |
| Fixed-candidate comparison | Which provider does the agent choose among the specified alternatives? |
| Paired metadata A/B | How does a declared description change affect behavior within the same candidate environment? |

These designs answer different questions. A description experiment supplies controlled information to test its effect on decisions. A discovery evaluation examines appearance under its declared retrieval conditions. A fixed-candidate lift is not an organic-discovery result.

## Match the method to the evaluation

Flovia uses different analysis paths for diagnosing a task, comparing a description change across the funnel, and estimating a change in provider selection. Choose the path around the decision you need to make.

| Evaluation | What is compared | How results are summarized |
| --- | --- | --- |
| PromptCase funnel | Agent decisions and contract checks for each task | Stage rates and conditional transitions; supported rate metrics use prompt-cluster bootstrap intervals |
| PromptCase metadata A/B | Matched baseline and variant runs through the funnel; candidate rotations follow the study | Stage-by-stage differences, missing observations, and pairs that improved or regressed; candidate orders are pooled with equal weight |
| Selection-lift study | Provider selection under baseline and variant metadata | Complete eligible pairs, the study’s intent or paraphrase-group weighting, and paired cluster bootstrap intervals |

The funnel A/B summary and selection-lift estimator are separate analyses. A confirmation study also requires its own recorded sample-size decision and evidence checks. Agree on the analysis path, candidate rotations, interval method, and confirmation requirements when scoping the evaluation.

## Pair baseline and variant

In a metadata A/B study, the baseline and variant share the task, requested model, repeat, candidate order, and non-intervened inputs. Only the target fields named in the plan change. Depending on the study, these can be Discovery descriptions, Invocation descriptions, or a declared combination.

Conceptual comparison unit

```text
Shared: task × model × repeat × candidate order × execution path
Baseline: original target description
Variant: revised target description
Fixed: competitor metadata, tool schemas, fixtures, scoring rules
Compare: selection, invocation, completion, and failure stage
```

When both Discovery and Invocation descriptions change together, the result measures the joint intervention. Attributing the difference to one surface requires a design that isolates that surface.

## Control candidate-position effects

Candidate order is an experimental factor. Supported metadata studies use cyclic Latin-square rotations so each provider occupies each candidate position. Baseline and variant are paired within the same order, and summaries can pool with equal weight across the Latin orders.

Pairing holds the surrounding scenario constant; rotation balances the target’s exposure across positions. Together, they make a description comparison easier to interpret when the agent is sensitive to how alternatives are presented.

| Rotation | Position 1 | Position 2 | Position 3 |
| --- | --- | --- | --- |
| A | Target | Alternative 1 | Alternative 2 |
| B | Alternative 1 | Alternative 2 | Target |
| C | Alternative 2 | Target | Alternative 1 |

This example illustrates position balancing, not a customer experiment. Run order and arm-first balancing follow the selected study plan; they are not assumed for every evaluation.

## Interpret the contrast at the tested scope

A paired difference estimates how the tested behavior changed between baseline and variant in the stated tasks, models, candidate environment, and window. Read it alongside pair completeness, missing observations, model-level results, and the estimator actually used.

Descriptive funnel A/B findings do not by themselves establish statistical significance or a general causal effect. A confirmation study has its own pre-specified evidence and qualification requirements. Claims about production impact need evidence beyond a controlled simulated comparison.

- [Turn a comparison into an improvement](https://flovia.dev/docs/improvement-loop/): Use stage-level evidence to choose a fix and define the next retest.

---

# Metrics and evidence

Read every result with its population, uncertainty, execution conditions, and supporting trace.

Source: https://flovia.dev/docs/metrics-and-evidence/

## Start with the denominator

Use the results to decide which change to keep and which task to investigate next. Start with the affected tasks and models, compare the outcomes, then open the underlying runs to see what changed.

To read a rate, start with what was counted. Flovia’s metric records keep the estimate, the numerator and denominator where applicable, the eligible run count, the missing count, and the interval method when one is available. Cumulative stage rates and conditional transitions use different populations.

| View | How to read it |
| --- | --- |
| Cumulative stage rate | How many runs in the declared population reached this stage? Earlier drop-off affects later reach. |
| Conditional transition rate | Of the runs eligible at the preceding gate, how many passed the next one? |
| End-to-end completion | How many eligible tasks completed across the whole evaluated journey? |
| Paired lift | How did the outcome differ between matched baseline and variant observations? |

Illustrative arithmetic · not a customer result

```text
# Illustrative figures only — not measured customer results
100 eligible tasks → 60 selections → 45 valid invocations
Selection reach: 60 / 100 = 60%
Invocation reach: 45 / 100 = 45%
Selection → invocation: 45 / 60 = 75%

Baseline completion: 30%
Variant completion: 42%
Difference: +12 percentage points
```

Infrastructure errors, unobserved stages, and missing Judge results mean different things. The selected metric contract sets the exclusions and missing-data treatment. Some paired estimators use only complete eligible pairs; planned-population summaries can keep missing observations in the denominator. The policy declared in the report takes precedence over any generic formula.

## Account for repeated tasks

Repeated runs of the same prompt are related observations. Where the selected metric supports it, the engine uses prompt-cluster or intent-cluster bootstrap intervals: it resamples task clusters together to preserve that grouping. Metadata-lift estimators in the corresponding evaluation path support paired cluster bootstrap.

An interval is reported only when the selected method’s requirements are met. A missing interval does not imply zero uncertainty. Read model-level and task-family results alongside the aggregate, especially when models behave differently or the task set is small.

In evaluation paths that support confirmation studies, the engine checks whether coverage, sample size, precision, and traceability to source runs meet the predeclared requirements. This distinguishes a promising pilot from a result that meets confirmation criteria. Descriptive funnel A/B summaries use their own declared estimator.

## Trace a finding back to a run

- **Plan:** Scope + input versions
- **Run:** Decisions + results
- **Score:** Checks + rubric
- **Finding:** Metric + diagnosis

- Evaluation definition: tasks, prompts, models, candidate conditions, execution paths, and measurement window.
- Coverage: planned and completed runs, missing observations, exclusions, and failure reasons.
- Run evidence: candidate and selection outputs, tool calls, argument validation, execution result, and answer where reached.
- Scoring provenance: deterministic checks, Judge status, criteria, and the selected scoring version.
- Comparison evidence: baseline and variant identities, changed fields, pairing, estimator, and any applicable qualification.

## Scope: what a result can support

An evaluation is tied to its tasks, models, candidate conditions, execution environment, and measurement window. Reading that scope helps you decide whether to adopt a change or collect more evidence.

| Evidence | Decision it supports | What needs separate evidence |
| --- | --- | --- |
| Fixed-candidate comparison | Changes in selection and use among supplied alternatives | Discovery without supplied candidates or actual web retrieval |
| Fixture-backed execution | Invocation and answer correctness against a declared response contract | Live API availability, authentication, and latency |
| Controlled retest | The stage that improved and the failures that remain in the tested scope | Effects on production usage, conversion, or revenue |

The requested model identity is recorded; confirming which model actually served the request also needs provider-side evidence. Repeats and related-task evaluations can extend the evidence beyond an initial result.

---

# Improve and retest

Translate a failed stage into a concrete surface change, then test the same task again.

Source: https://flovia.dev/docs/improvement-loop/

## Let the failure determine the fix

Every evaluation should lead to a product decision: what to change, why it should help, and how to check it. Start with a finding in the dashboard, review the supporting evidence, apply a targeted fix, then compare the retest before choosing your next move.

Start with the observed failure and the product surface that can influence it. Flovia generates a remediation prompt your team can run with its coding agent. It ties the proposed edit to the task, the failed stage, and the behavior the retest should verify.

| Observed failure | Surface to inspect | Retest question |
| --- | --- | --- |
| Target absent or not selected | Product positioning, documentation, landing page, llms.txt, and candidate descriptions | Does the target appear or get selected under the declared discovery conditions? |
| Wrong operation | API or MCP tool names, descriptions, and operation boundaries | Does the agent resolve the allowed operation? |
| Invalid arguments | Parameter meanings, input constraints, and worked examples | Does the request pass schema and semantic validation? |
| Unusable result or incomplete answer | Response contract, required fields, units, and interpretation guidance | Can the returned data support every completion criterion? |

These are diagnostic starting points. A changed page is not guaranteed to be retrieved or used by a model. The retest must either keep how the agent receives that information the same or change it explicitly.

## From a finding to a change in your dashboard

Your private dashboard connects the evaluation summary, the supporting evidence, and implementation guidance. The excerpts below show the existing dashboard with identifying details replaced. The figures shown belong to those specific tests.

This example asks a coding agent to write transaction-history code and compares how the documentation reaches it. With a pointer in CLAUDE.md, selection of the target API changed from 0 of 6 runs to 6 of 6. This is the API-selection outcome observed for this task under the tested conditions.

Choose the next action — The overview lists recommended actions alongside their measured impact and the test behind each result. Use it to choose which finding to investigate first.

![Overview: recommended actions and the comparison for the selected item.](https://flovia.dev/docs/images/dashboard-overview.png)

Overview: recommended actions and the comparison for the selected item.

Read the finding — Open an action to see what the agents struggled with and which change is recommended. In this example, agents missed a native API and a signup path because those docs were not linked from the instructions they loaded at startup.

![Finding: missed entry points, a recommended change, and links to evidence and an implementation prompt.](https://flovia.dev/docs/images/dashboard-finding.png)

Finding: missed entry points, a recommended change, and links to evidence and an implementation prompt.

Inspect the evidence and prepare a fix — Open the supporting evidence to check the conditions and the recorded behavior. Copy the implementation prompt, review it with your team, and apply the change. After release, rerun the same test to decide whether to keep the change or revise it.

![Evidence: the documentation-entry conditions and observed outcomes before and after the change.](https://flovia.dev/docs/images/dashboard-evidence.png)

Evidence: the documentation-entry conditions and observed outcomes before and after the change.

## Example: a documentation improvement

A blockchain data API updated its documentation to clarify how to choose and use its capabilities. Comparing AI answers based on the earlier and updated documentation showed which explanations improved and where errors remained.

The comparison checked whether each answer picked an appropriate capability, identified the correct call target, and explained the work left on the developer’s side. Each item below was checked across five answers with the updated documentation.

| Capability checked | Before | After |
| --- | --- | --- |
| Retrieve transaction identifiers only | 0 of 5 gave correct guidance: answers either could not identify the method or suggested a capability meant for another purpose | 5 of 5 explained how to retrieve identifiers only through the history-retrieval capability |
| Choose between history-retrieval methods | Could not explain the difference between retrieving original transaction data and a human-readable, parsed history | 5 of 5 explained the difference; 4 still named the wrong API for retrieving original data |
| Monitor transactions through WebSocket | Unclear whether calling the connection-preparation tool would start monitoring | 5 of 5 explained that the tool returns connection details or sample code and that the developer must run the monitoring process themselves |
| Receive real-time streams on a server | Did not correctly distinguish looking up connection information from generating connection configuration | 5 of 5 chose the configuration-generation capability and explained that the developer must establish the connection |
| Estimate priority fees and retrieve connection information | Mapped capabilities to incorrect call targets | 5 of 5 identified the correct call targets |

Improvements showed up in guidance on transaction-history methods, real-time connection preparation, and call targets for capabilities such as priority-fee estimates.

For developers, the expected benefit is fewer wrong implementation choices and fewer misunderstandings about whether preparing a connection starts a live service. The comparison observed better AI guidance; it did not measure any reduction in implementation errors or support requests.

These counts describe the answers checked in this comparison, not live API execution rates or success rates across other tasks and models. The next retest should check the API name separately from the explanation of when to use each method.

## Example: make the historical-rate operation easier to choose

The following walkthrough uses the fictional exchange-rate task introduced in Scenario design. It illustrates how a team moves from a question to a decision; it does not report a customer result.

| Step | What to do |
| --- | --- |
| Question | Can an agent retrieve the EUR-to-USD rate for a specified date, rather than today's rate? |
| Evaluation | Run the task with a fixed tool catalog and a prepared response. Check the selected operation, the currency arguments, the date, and the final answer. |
| Finding | Suppose the agent selects the latest-rate operation. The first failure is operation selection, so inspect how the two operations are described. |
| Fix | Clarify when to use the historical operation and add a dated request example. Keep the tool schema and the response unchanged for this comparison. |
| Retest | Repeat the same task under comparable conditions. Check whether the agent chooses the historical operation and completes the answer correctly. |
| Decision | Keep the revised description if the tested behavior improves. If operation selection improves but the date argument is still wrong, make the date instructions the next test. |

## A useful remediation prompt is testable

Illustrative remediation · fictional exchange-rate API

```text
Observed failure: the agent chooses the latest-rate operation
for a request that specifies a historical date.

Revise the operation descriptions to distinguish latest and
historical rates. Document the date format and add one dated
EUR-to-USD example. Keep endpoint behavior and schemas unchanged.

Acceptance: for the same dated task, the agent selects
get_historical_rate and supplies the correct date and currencies.
Then verify that the final answer uses the returned rate and date.
```

Your team reviews and implements the change in its own environment. Flovia provides the diagnosis, remediation guidance, and verification, connecting the change to its retest results.

## Decide what to do after the retest

Read the changed stage together with the final outcome. An improvement can remove one blocker and expose the next. Use that result to choose the next edit, keeping the parts that already work.

| Retest finding | Next action |
| --- | --- |
| Selection improves, but the wrong operation is still chosen | Keep the clearer product description and inspect the operation names, descriptions, and boundaries |
| The correct operation is called with invalid arguments | Clarify the parameter meanings and add an example with the task’s actual constraints |
| The result is usable, but an answer requirement is missing | Inspect the completion criterion and the relevant output field; make its meaning and use in the answer explicit |
| Results vary by model or task, or observations are missing | Inspect those groups and failure records before choosing an edit; complete or extend the comparison where needed |

For the historical exchange-rate example, choosing the historical operation fixes the operation error. If the answer then omits the date, that becomes a separate completion issue with its own retest criterion.

- [Inspect a completion assessment](https://flovia.dev/docs/agent-funnel/): See how the rate, currency direction, and date are checked separately.

## Retest the same decision

1. Record the changed artifact and the intervention fields.
2. Keep the task, requested model, candidate conditions, execution path, and scoring comparable.
3. Confirm that the revised surface is actually supplied or retrieved the way the design intends.
4. Compare stage-level outcomes and completion, together with missingness and coverage.
5. Use the remaining failure to define the next intervention.

Model updates and measurement windows matter. If the surrounding conditions change, name them in the comparison. The result should state what improved in the tested stage and what is still unresolved.

---

# Technical FAQ

Direct answers about the simulation, evaluation methods, and current product interface.

Source: https://flovia.dev/docs/faq/

## What exactly does Flovia simulate?

Flovia simulates an agent’s task journey under declared conditions: candidate discovery, provider selection, tool resolution, valid invocation, execution on the configured path, result usability, and answer completion. Task definitions, tool contracts, and scoring criteria make the journey inspectable and comparable across runs.

## Does the simulation use real model calls?

Yes, in the Live LLM / Mock Tool staged path: it calls the specified model endpoints and evaluates tool execution against prepared responses. Offline pipeline tests can instead use prepared model responses. Read every result together with its model-call path and its tool-execution path; they are configured separately.

## Does a passing simulation prove the live API works?

No. When an evaluation uses prepared data in place of live API responses, it checks whether the AI can construct the correct call and use that data to answer appropriately. To confirm that the live API itself works correctly, you need to test it separately by connecting to the API.

- [Read the evidence scope](https://flovia.dev/docs/metrics-and-evidence/): Understand what comparisons, execution results, and retests support.

## What technical methods does Flovia use?

The evaluation engine combines task-based experimental design, versioned input contracts, staged function calling, deterministic schema and semantic validation, model-assisted completion assessment, controlled paired metadata experiments, and evidence provenance. Supported study designs use cyclic Latin-square candidate balancing. Applicable metrics use prompt- or intent-cluster bootstrap and paired cluster bootstrap. Each evaluation names the subset and configuration actually used.

## How are evaluation methods chosen?

Methods are combined around the question the evaluation needs to answer. Task-specific contract checks and stage-level assessment diagnose failures; matched pairs compare description changes. Studies that examine candidate-position effects use Latin-square balancing, and supported metrics use cluster bootstrap intervals. Each evaluation records the selected methods and their settings in its plan.

## How is this different from checking mentions?

A mention answers whether the product appeared. The funnel also asks whether the agent selected it, reached the right operation, supplied valid arguments, obtained usable data, and completed the task. This locates the change needed when visibility improves but successful use does not.

## Does metadata A/B measure organic discovery?

No. A fixed-candidate metadata A/B test measures behavior under controlled exposure to the declared candidates and does not establish organic discovery. An evaluation without supplied candidates answers a different question, and any claim about actual web retrieval needs retrieval evidence from that evaluation.

## How do we use Flovia? Is there a public API?

Your first evaluation is free. Start with a 30-minute call to choose a task an agent should complete using your API or MCP. Flovia delivers the initial results in your private dashboard.

Start through the evaluation request form on the Flovia website. Flovia works with your team on the evaluation scope and delivers results, failure analysis, remediation guidance, and retests through a private customer dashboard. There is currently no public Flovia API, SDK, or MCP server.

## Can the current profile evaluate a multi-step workflow?

PromptCase evaluates whether an agent selects the defined operation, supplies appropriate arguments, and answers using the returned data. An operation can group several underlying actions, but workflows where the agent assembles a sequence of independent calls are outside this profile. Agree on an execution path that fits the workflow when defining the evaluation scope.

- [Execution environment and scope](https://flovia.dev/docs/simulation-engine/): Read how model calls, tool responses, and completion checks fit together.

## Who applies the recommended changes?

Choose the implementation approach that fits your team's policies. Your team can review the recommended changes and apply them in its own environment. Enterprise customers can also use a GitHub-connected, semi-automated workflow, with their team retaining review and deployment control. After a change, Flovia reruns the evaluation so you can see what improved and what to work on next.