AI Quality Assurance

Quality assurance for AI systems

The most important question about any AI feature is not what it can do – but whether you can trust it. Our answer is a set of architecture patterns that don't promise trust, but enforce it.

The first pattern is the second opinion: an independent model cross-reads another system's confident results – including the seemingly perfect matches – and checks whether the meaning actually fits. The rule is warn-only: the reviewer may lower confidence, never change results. A doubted match loses its green in the confidence traffic light and carries a reasoning note – the human decides.

The second pattern is structured output with code having the last word: models answer typed and schema-validated, and deterministic application logic audits every proposal against the system's rules – with logged overrides, versioned prompts, and decisions that stay traceable back to the exact instruction that produced them.

And because trust also means cost control: a token ledger makes multi-model, multi-provider consumption accountable per business object – every run carries its itemised receipt.

AI Quality Assurance

Trust as architecture

Three patterns that make AI results auditable

The invisible second opinion

A review model cross-reads every automatic assignment the way a domain expert would. It may only lower confidence – with reasoning and an alternative suggestion. No AI badge, just an honest number.

Code has the last word

Models propose – deterministic logic audits every proposal against the system's rules and can override it. Every override is logged with the proposal, the effective action, and the reason.

The itemised cost receipt

A token ledger books consumption per source and model; prices only materialise at display time from a configured price table. A model without a rate is visibly flagged – instead of silently priced at zero.

QA patterns in practice

Three systems in which AI results are systematically distrusted – by design.

The 100% you can trust
The voice never grades the answers
What did that run cost?
Second Opinion

The 100% you can trust

In a regional film fund's budget import, an LLM reviews every assignment after deterministic auto-mapping – including the seemingly perfect string matches – the way a production accountant would: does the meaning of the line actually fit the account? Only labels and account paths reach the model, never amounts. A representative catch: a sound post-production line that name similarity had confidently mapped to a pre-production screen-tests account lost its green – with the correct account suggested.

Structured Output

The voice never grades the answers

In our voice interview platform, the conversational model never decides whether an answer was good enough: after each answer, a separate evaluator model returns a typed verdict – coverage per criterion, missing points, proposed next action. Deterministic code audits every proposal against the interview's rules and can override it – logged with the proposal, the effective action, and the reason. Prompts are versioned; every decision carries its prompt version and a hash, and thus even survives deletion of the transcript.

Cost Governance

What did that run cost?

Every run of the platform carries an itemised token receipt across multiple models and providers. The browser reports only token counts, never model identities – which price applies is always decided by server configuration; a manipulated client can't reprice anything. Money is never stored: the ledger accumulates tokens, pricing happens at display time – a rate correction repairs all history, and a model swap never retro-prices old runs.

AI skepticism as a feature

Why we build this way

Often, the visible product isn't an "AI feature" at all – it's an honest number, an auditable proposal, a traceable receipt. Bringing AI into business processes where the numbers have to be right takes systems that distrust their own results – and that's exactly what we build the architecture for: second opinions, code with the last word, end-to-end traceability, and costs that can be explained per business object.

Highlights

  • A second model checks what the first one claims – including the perfect matches
  • Warn-only: AI may lower confidence, never change results
  • Structured output: typed, schema-validated answers instead of free text
  • Prompt versioning: every decision is traceable back to its instruction
  • Token ledger: every run carries its itemised cost receipt

Software Development in Hamburg!

Start new project with us or upgrade an existing one to the next level
Request Project Join the team