Benchmarks and model qualification

Akno benchmarks complete user-visible paths, not just isolated model responses. A model can produce a valid schema while silently dropping the direct answer, making recall slower without changing order, or leaking an instruction-bearing source into its output. The gates are designed to catch those failures.

All committed benchmark corpora are invented. Live benchmark commands never open or send the configured knowledge base unless the command explicitly says it is measuring that installation.

akno bench --write
akno bench --retrieval-only
akno bench graph
akno bench entities --provider openai --model gpt-6-luna --reasoning none
akno bench answer --concurrency 2
akno bench auto-recall --concurrency 2
akno bench auto-recall-answer --concurrency 2

--split test --runs 5 selects a frozen held-out corpus for supported model-dependent gates. Development is one run by default so iteration remains cheap. Held-out artifacts are release evidence, never a tuning set.

A useful pass is more than an average score. Depending on the surface, Akno also requires:

• exact corpus fingerprint validation;
• successful independent review of every source and judgment;
• repeated-run decision stability;
• no schema fallback or typed degradation;
• complete retention of direct and supporting evidence;
• rejection of irrelevant and instruction-bearing candidates;
• correct empty and abstention behavior;
• source-locator integrity and hard output budgets;
• latency within the path’s declared ceiling; and
• a bound end-to-end run through production candidate generation and assembly.

Reports deliberately omit prompts, questions, evidence bodies, answers, source paths, endpoints, credentials, and provider error text. They retain stable invented ids, aggregate judgments, model configuration, latency, token receipts, fingerprints, checks, and blockers.

The guided OpenAI setup now uses gpt-6-luna for every generative role and prompted reranking, with text-embedding-3-small at 1,536 dimensions for embeddings. Existing installation settings are not rewritten. Run setup preflight with your configured endpoint before adopting the preset.

The following qualification measurements describe the historical configuration, not the current model. That configuration used two models through one provider endpoint:

• text-embedding-3-small at 1,536 dimensions for semantic candidates; and
• gpt-5.6-luna with no reasoning for generative roles and prompted reranking.

This is a single-endpoint preset, not a single-model preset. The historical configuration was selected for quality and cost only after the ranking and production-path gates passed. The reranking contract remains akno-judgment-map-v9 / tuple-judgment-map-v6, with a ten-candidate window.

The independently reviewed invented-ranking-v5 held-out release run covered 120 sources and 80 cases. Across five repeats, the selected configuration produced:

The separate bound production run embedded 120/120 chunks, retained every direct answer through the fusion pool, judgment window, and final assembly, reached 100% success at ranks 1 and 3, and recorded no degradation or fallback.

Primary evidence:

• held-out matrix
• latency receipt
• production-path receipt

These numbers qualify one frozen configuration against one invented corpus. They do not claim that every query, endpoint load, future model revision, or private knowledge base will behave identically.

Historical artifacts are not clutter to overwrite. They establish which prompt, schema, corpus, or selector failed and prevent the same regression from being disguised by a newer average.

When an observed held-out failure changes the prompt or grading contract, that corpus remains frozen. The new contract must be evaluated on fresh held-out content. Selector-only defects may be corrected from existing measurements when doing so requires no new provider call; the artifact must record that distinction.

Detailed corpus contracts and historical evidence live beside each benchmark:

• ranking
• automatic recall
• automatic recall plus host answer

pnpm bench:longitudinal is the opt-in first slice of #66: repeated English retention, source correction, retraction, source-folder timeline discovery and actual downstream use through built packages and a socket. It includes development/held-out episodes, explicitly seeded caller errors, unchanged cycles and restart/rebuild. Freeze inputs before live egress, then source-review all checkpoint outputs. Runtime verification is not ground truth, and an always-abstain pipeline fails the useful-coverage gates.

The longitudinal evaluation contract describes reproducible commands, review format, declared gates and coverage gaps. This slice has no positive observation/reflection control or model substitution and does not establish full #66 coverage.

Choose the narrowest surface affected by a change, then run the general suite before release. Examples:

akno bench graph
akno bench answer --split test --runs 5 --output /tmp/akno-answer.json
akno bench auto-recall --split test --runs 5 --output /tmp/akno-auto-recall.json
akno bench auto-recall-answer --split test --runs 5 \
  --output /tmp/akno-auto-recall-answer.json

Do not repeatedly run a frozen held-out corpus while tuning. Reproduce the failure shape in new development data, make the change there, obtain independent review for a fresh frozen corpus, and run the final gate once.