Technical FAQ

Direct answers about the simulation, evaluation methods, and current product interface.

What exactly does Flovia simulate?

Flovia simulates an agent’s task journey under declared conditions: candidate discovery, provider selection, tool resolution, valid invocation, execution on the configured path, result usability, and answer completion. Task definitions, tool contracts, and scoring criteria make the journey inspectable and comparable across runs.

Does the simulation use real model calls?

Yes, in the Live LLM / Mock Tool staged path: it calls the specified model endpoints and evaluates tool execution against prepared responses. Offline pipeline tests can instead use prepared model responses. Read every result together with its model-call path and its tool-execution path; they are configured separately.

Does a passing simulation prove the live API works?

No. When an evaluation uses prepared data in place of live API responses, it checks whether the AI can construct the correct call and use that data to answer appropriately. To confirm that the live API itself works correctly, you need to test it separately by connecting to the API.

Read the evidence scope Understand what comparisons, execution results, and retests support.

What technical methods does Flovia use?

The evaluation engine combines task-based experimental design, versioned input contracts, staged function calling, deterministic schema and semantic validation, model-assisted completion assessment, controlled paired metadata experiments, and evidence provenance. Supported study designs use cyclic Latin-square candidate balancing. Applicable metrics use prompt- or intent-cluster bootstrap and paired cluster bootstrap. Each evaluation names the subset and configuration actually used.

How are evaluation methods chosen?

Methods are combined around the question the evaluation needs to answer. Task-specific contract checks and stage-level assessment diagnose failures; matched pairs compare description changes. Studies that examine candidate-position effects use Latin-square balancing, and supported metrics use cluster bootstrap intervals. Each evaluation records the selected methods and their settings in its plan.

How is this different from checking mentions?

A mention answers whether the product appeared. The funnel also asks whether the agent selected it, reached the right operation, supplied valid arguments, obtained usable data, and completed the task. This locates the change needed when visibility improves but successful use does not.

Does metadata A/B measure organic discovery?

No. A fixed-candidate metadata A/B test measures behavior under controlled exposure to the declared candidates and does not establish organic discovery. An evaluation without supplied candidates answers a different question, and any claim about actual web retrieval needs retrieval evidence from that evaluation.

How do we use Flovia? Is there a public API?

Your first evaluation is free. Start with a 30-minute call to choose a task an agent should complete using your API or MCP. Flovia delivers the initial results in your private dashboard.

Start through the evaluation request form on the Flovia website. Flovia works with your team on the evaluation scope and delivers results, failure analysis, remediation guidance, and retests through a private customer dashboard. There is currently no public Flovia API, SDK, or MCP server.

Can the current profile evaluate a multi-step workflow?

PromptCase evaluates whether an agent selects the defined operation, supplies appropriate arguments, and answers using the returned data. An operation can group several underlying actions, but workflows where the agent assembles a sequence of independent calls are outside this profile. Agree on an execution path that fits the workflow when defining the evaluation scope.

Execution environment and scope Read how model calls, tool responses, and completion checks fit together.

Who applies the recommended changes?

Choose the implementation approach that fits your team's policies. Your team can review the recommended changes and apply them in its own environment. Enterprise customers can also use a GitHub-connected, semi-automated workflow, with their team retaining review and deployment control. After a change, Flovia reruns the evaluation so you can see what improved and what to work on next.