Documentation / Ship / Evaluation release contract

Release and prove the evaluation contract

VAANI 1.0.0rc32 is the published private release containing the LiveKit text evaluation contract merged in #109. Its authenticated publication, clean private-registry install, and credentialed synthetic LiveKit text attempt are recorded in rc32-evaluation-release.json. The deterministic local gate remains separate and does not certify those operational observations by itself.

vaani-evals remains a separately packaged consumer and VAANI has no dependency on it. The supported production-distribution seam is deliberately bounded to run_livekit_worker(..., evaluation_evidence=True), server-authored job metadata, the two text topics, and the finalized evidence wire record described in Results and evidence.

Authorization and correlation

Only a trusted dispatch backend may author job.metadata.vaani_evaluation_evidence. It binds an opaque correlation ID to the evaluator's consumer_identity. VAANI accepts text and end control only from that identity after it becomes the PrimaryParticipant, and publishes evidence only to the same identity. The evaluator should allocate exactly one disposable consumer per attempt: its first valid correlated record is terminal, and later replay is rejected.

Local and remote workers use the same topics and payloads:

• vaani.evaluation.text.v1 carries one participant utterance per LiveKit text stream;
• vaani.control.v1 accepts only the exact graceful-end document;
• vaani.evaluation.evidence.v1 returns chunk envelopes followed by a commit.

The receiver must verify the envelope schema, bounds, expiry, record digest, canonical CallResult digest, and expected correlation ID before accepting the record. Room data with a participant sender identity is not server-authored evidence and must be rejected.

Timeout, redaction, and failure semantics

The existing result_publication_seconds deadline bounds serialization and LiveKit delivery after Closure. Its absolute UTC expiry is repeated in every chunk and commit. A receiver accepts a commit only before that expiry; a timeout or missing commit leaves no accepted record and cannot change the sealed CallResult.

The record contains the configured transcript capture, normalized tool and lifecycle evidence, timing and usage ledgers, and the finalized Judge report. It never contains credentials or bot-private objects. Errors expose normalized, redacted provider diagnostics, not raw provider exceptions or payloads. Tool arguments/results and consumer Judge evidence remain subject to the bot's explicit audit and evidence allow-lists.

Treat a room connection, dispatch, authorization, expiry, packet, or evidence delivery problem as an infrastructure failure. Treat a successfully received finalized result that violates the scenario expectation as a product failure. Do not turn absence of evidence into a product verdict, and do not turn a failed scenario into an infrastructure incident merely because the base call completed.

All evaluation calls and tools must use synthetic, fake, or sandbox data and destinations. Production customer data and production side effects are outside this contract.

Deterministic clean-wheel gate

From a clean checkout, build rc32 and run the disposable consumer against the built wheel:

uv build --wheel --sdist --out-dir /tmp/vaani-rc32-dist .
python scripts/evaluation_release.py deterministic-proof \
  --wheel /tmp/vaani-rc32-dist/vaani-1.0.0rc32-py3-none-any.whl \
  --source-commit "$(git rev-parse HEAD)" \
  --output /tmp/vaani-rc32-deterministic-proof.json
python scripts/evaluation_release.py check \
  --evidence /tmp/vaani-rc32-deterministic-proof.json \
  --expected-version 1.0.0rc32

The gate installs the wheel bytes non-editably with no local-path dependency, removes PYTHONPATH, imports VAANI only from the isolated install target, and runs the real worker, Assistant, tool, lifecycle, finalization, and LiveKit text bridge. LiveKit service and provider boundaries are deterministic fakes. The proof therefore does not certify private publication, registry authentication, provider credentials, or a hosted LiveKit room. The wheel file is the artifact under test; it is not an editable or local-path dependency declared by the installed environment.

The deterministic scenario requires participant and agent transcript items, a returned synthetic lookup_ticket tool, a CLOSED lifecycle, timing events, observed usage entries, and the complete Judge report. The bundled basic profile does not configure a Judge, so its honest finalized status is NOT_CONFIGURED; an operational scenario may configure a Judge and must record the status it actually receives.

Credentialed operational gates

To reproduce or supersede the recorded operational proof, copy rc32-evaluation-release.template.json outside the source tree and replace every placeholder from direct observations:

1. Record the immutable source commit, uploaded wheel and sdist filenames, and SHA-256 digests. Download them again from the authenticated private registry and verify the bytes.
2. In a new environment, install the exact registry artifact without editable or local-path dependencies. Record installed distribution metadata and archive provenance.
3. Dispatch one synthetic credentialed LiveKit text attempt, receive the finalized evidence as the bound identity, and retain the full correlated record plus the Base64-encoded exact canonical CallResult bytes. Use only fake/sandbox tools.
4. Change all three operational gate states to passed, set credentialed_livekit_service to true, remove the template notice, and validate the completed file:

python scripts/evaluation_release.py check \
  --evidence /secure/evidence/rc32-evaluation-release.json \
  --expected-version 1.0.0rc32 \
  --require-operational

The checked-in rc32 record passes that command. Any future version or attempt must repeat the gates from direct evidence; never relabel the deterministic proof or a historical call as current credentialed evidence.