Documentation / Core concepts / Results

Results, evidence, cost, and failure

One CallResult answers: what VAANI can prove happened during this call attempt. Unknown provider facts stay unknown; the runtime does not manufacture precision.

Main evidence groups

Group	What it records
outcome, ending, lifecycle_history	terminal classification and progression
transcript	committed/interrupted participant and agent text under capture policy
provider_history	route, provider, model/voice, revision, attempt, disposition
language_history, update_history	configuration changes and decisions
tool_activity	normalized tool execution and external-completion status
timings	monotonic events and measurements
usage, cost	observed quantities, estimated and provider-reported money
failure	primary/secondary normalized errors
telephony	immutable identity plus transfer and recording evidence
judge	optional bounded evaluation that cannot change the base outcome
provenance	schema, runtime, adapters, presets, price book, fingerprint

Known, unknown, not applicable

ObservedText and ObservedDecimal distinguish three states:

• KNOWN: VAANI has an attributable value;
• UNKNOWN: the fact applies but reliable evidence is unavailable;
• NOT_APPLICABLE: the fact does not apply, with a reason.

This distinction flows into usage and cost completeness. A missing provider counter is not silently treated as zero.

Cost is evidence, not billing

The consumer supplies a revisioned PriceBook. VAANI can estimate known usage against those rates and retain separately observed provider costs. COMPLETE, PARTIAL, and UNAVAILABLE describe the evidence—not whether an invoice is correct.

Canonical bytes and digest

body = result.canonical_bytes()
digest = result.result_digest()

The bytes are deterministic call-result.v1 JSON and the digest is full SHA-256 over those exact bytes. Use both to build an append-only ResultPublicationRecord; do not serialize a separate mutable dictionary and call it the canonical result.

Failure model

• A setup problem before call ownership raises InvalidCallSetup and returns no result.
• A failed call after ownership still closes with one result and normalized FailureRecord.
• VoiceRuntimeError carries category, stable code, operation, phase, retry guidance, remediations, and selected action.
• Provider diagnostics are normalized and redacted; raw provider exceptions, payloads, and secrets do not enter the public record.
• Publisher failure happens after Closure and produces a DeliveryReport, not a mutated CallResult.

LiveKit evaluation evidence

The rc32 compatibility transport below publishes finalized canonical CallResult evidence. It does not by itself attest that a consumer's business providers are fake or sandboxed. Remote simulated-user suites that arrange fixtures or require deterministic branch facts must instead start an explicit fail-closed remote evaluation worker. That contract sends a signed readiness attestation before text intake and a signed final attestation correlated with this canonical result.

An authorized external evaluator can exercise a complete development call without importing the bot's Assistant, tools, or other Python internals. Run a dedicated development worker with run_livekit_worker(..., evaluation_evidence=True) and have the dispatch backend place this request in private LiveKit job metadata:

{
  "vaani_evaluation_evidence": {
    "correlation_id": "scenario-42-attempt-1",
    "consumer_identity": "evaluation-participant-42",
    "transport": "text"
  }
}

The evaluator may choose the opaque, 1–128-character UTF-8 correlation ID, but the trusted dispatch backend must bind consumer_identity to that evaluator's LiveKit identity. Non-UTF-8-encodable correlation or consumer identities are rejected before the Assistant starts. VAANI accepts the evidence request only when that identity becomes the call's PrimaryParticipant. The feature is rejected outside the development environment. These checks prevent another room participant from supplying text for the call or receiving its result.

The simulated participant must observe the agent participant attribute lk.agent.state=listening before it sends its first utterance. VAANI first enqueues primary_participant_ready, installs the evaluation stream handler, then makes that listening state visible. Headers received before readiness are therefore ignored by the LiveKit SDK and cannot allocate a VAANI reader or become a cascade turn. The participant sends each ready utterance as one LiveKit text stream on vaani.evaluation.text.v1; a stream is limited to 16 KiB and 1,024 yielded chunks. VAANI requests the next item so a reader with exactly 1,024 chunks can expose its trailer. A 1,025th yielded chunk proves that the work bound was exceeded. Every yielded chunk counts, including empty or non-text chunks and chunks encountered after that reader has already become invalid, so a stream cannot retain a slot with unbounded zero-length work. VAANI admits at most four open authorized evaluation streams. Identity is authorized before this budget is consumed, so held-open wrong-identity streams cannot deny an evaluator turn. Wrong-identity, malformed, empty, oversized, and authorized-overflow streams retain no further content while they are consumed through a trailer or the first excess chunk. An ordinary authorized-concurrency overflow does not close intake; after an admitted reader retires, a later turn can be accepted.

LiveKit 1.1.8 retains a receiver-side text reader until its trailer and exposes no supported close for one reader. Consequently, any accepted or rejected reader that yields a 1,025th chunk fails the whole text transport: VAANI unregisters both text handlers, exposes one immediate connection_failed, and calls the supported room disconnect() operation. Queued evaluation turns are discarded. This promptly stops the SDK room event pump so retained readers cannot receive later chunks; ordinary transport cleanup then releases the room. It never synthesizes a PrimaryParticipant end.

Completed readers do not make pending turns unbounded. Across them, VAANI retains at most four undelivered evaluation turns and at most four in-flight lk.agent.state=thinking updates. A valid stream that would exceed either cap fails the text transport by the same room-disconnect path, stops further stream intake, and contributes no turn or state-update task. Delivery retires a turn slot; task completion retires a state-update slot. Transport cleanup cancels and joins all remaining tracked work. The evaluator observes agent text through LiveKit's normal lk.transcription streams and ends the call by sending exactly {"kind":"end_call"} on vaani.control.v1. Control streams use the same authorization-first and 1,024-chunk rules, a separate limit of four open authorized readers, and a 256-byte payload limit. Wrong-identity and authorized-overflow control streams cannot request a PrimaryParticipant end.

Rejected stream draining is shared and bounded independently of both authorized budgets. VAANI drains four concurrent rejected readers. A fifth concurrent rejected reader synchronously unregisters both SDK stream handlers before scheduling failure propagation and room disconnect. A same-turn header burst therefore cannot allocate another SDK reader after that callback. The failure is then emitted and the room is disconnected while the fifth reader is also under bounded drain. If SDK unregistration is unavailable or raises, that fifth handler raises synchronously to abort dispatch instead of accepting more allocated readers; failure propagation and room disconnect remain scheduled. Successful unregistration retains the ordinary non-raising behavior. This bounds a never-ending header flood without attributing it to the PrimaryParticipant. The call then follows normal forced-failure teardown. Completed drain tasks retire, so ordinary sequential rejections do not accumulate. This is the LiveKit transport boundary: the evaluator does not call an Agent method or read bot state. Audio workers retain their normal participant-readiness behavior; this listening handshake is the explicit observable protocol for the development text evaluator.

After Closure, VAANI sends one logical evidence record to only consumer_identity as reliable data on vaani.evaluation.evidence.v1. The operation uses the profile's bounded result_publication_seconds deadline and cannot change the sealed result or delay job teardown beyond that deadline. The deadline includes complete-record and packet serialization. A process-wide publication admission admits at most two complete evidence delivery lifecycles, from before serializer admission until the publication task finishes after its final transport send or failure. Capacity exhaustion fails that publication closed before serialization or sending. A process-wide executor independently admits at most two evidence serialization work items and runs at most two serializer threads, with no additional CallResult queue. Each serializer slot is released when its executor work item finishes, before any subsequent transport send; caller cancellation does not release it while that work item is still running. Each publication slot is released only when its publication task actually finishes. Thus a timeout or cancellation-suppressing transport can delay publication-slot release without retaining an already-completed serializer slot, and cannot create an unbounded detached-publication set. Completion notification blocks without polling the async event loop.

The complete evidence record is at most 1,048,576 bytes. A byte-bounded canonical encoder writes the existing exact canonical CallResult form only up to the remaining record capacity after fixed framing. It preserves Unicode, escaping, exact Decimal numbers, canonical bytes, and both digests, but stops before a larger complete byte string or record can be materialized; the unbounded CallResult.canonical_bytes() path is not used for this transport. It traverses the public form directly instead of first copying the complete object tree, bounds normalized mapping-key storage by the output budget, preflights Decimal magnitude before fixed-point formatting, and rejects nesting beyond 128 levels. Ordinary bounded values remain byte-for-byte identical to CallResult.canonical_bytes(). VAANI validates the complete record and every serialized packet before publishing the first chunk, so an oversized record emits no partial record. A record uses 1–128 chunks, each carrying 1–8,192 decoded payload bytes; every serialized UTF-8 envelope packet is at most 12,288 bytes. On timeout VAANI closes the publication fence, cancels and detaches the transport operation, and logs a distinct delivery timeout. A transport operation that suppresses cancellation may finish later and route a packet that was already in flight, but the receiver-side attempt expiry keeps a late commit unaccepted; any partial chunks remain unaccepted. The wire record is chunked so it does not depend on one oversized LiveKit data packet:

{
  "schema": "vaani.livekit-evaluation-evidence-chunk.v1",
  "kind": "chunk",
  "record_id": "sha256-of-complete-record",
  "attempt_expires_at_unix_ns": "1786030200123456789",
  "index": 0,
  "count": 2,
  "payload": "base64-bytes"
}

The LiveKit room service routes these packets only to the verified consumer_identity. At the evaluator, the received packet is the normal receiver shape (data and topic) with server-sender semantics; it does not contain destination_identities, so the consumer must not try to authorize itself by inspecting the server's send request. The consumer accepts packets only from that server-authored room-data path on the evidence topic, buffers chunks by record_id, accepts a final kind: "commit" packet with the same schema, ID, and attempt_expires_at_unix_ns, requires every index from zero through count - 1, joins the decoded payloads in index order, and verifies that their full SHA-256 equals record_id. It accepts at most 12,288 packet bytes; a chunk must have exactly the documented keys, a count from 1 through 128, a non-Boolean index in [0, count), and canonical Base64 whose 1–10,924 ASCII characters decode to 1–8,192 bytes. A commit has exactly its four documented keys. Both kinds require the exact envelope schema, a 64-character lowercase hexadecimal record_id, and a 1–20-character ASCII decimal expiry. The expiry and count are immutable within an assembly. Identical duplicate chunks are idempotent; conflicting duplicate chunks or inconsistent assembly fields discard that assembly. Before retaining or charging an observation, the disposable consumer rejects packets after its terminal record, packets with a participant sender identity, packets on another topic, empty or non-byte data, and packets over 12,288 bytes. Those packets never enter its packets list.

One attempt performs and retains at most 256 otherwise eligible packet observations, including malformed UTF-8, JSON, schemas, keys, IDs, expiry, Base64, counts, and indexes. The 257th eligible packet makes the attempt terminal, clears every incomplete assembly, and is not retained or processed; all later packets are rejected without retention. This cap covers both pre-terminal floods and every post-terminal packet while leaving room for one maximum-size valid 128-chunk record plus its commit.

One disposable consumer buffers at most 16 incomplete records and at most 1,048,576 decoded bytes across all of them; one record has the same 1,048,576 byte ceiling. A new record that would exceed either aggregate bound is ignored, and a chunk that would take its existing record over a byte bound discards that record. On every otherwise eligible receiver packet—even malformed JSON—the consumer first sweeps all incomplete assemblies whose expiry has arrived. This keeps state bounded under floods of unrelated partial records.

The expiry is an absolute Unix UTC nanosecond instant, encoded as a decimal string so every JSON implementation preserves it exactly, authored once by the worker when post-Closure publication starts, and repeated unchanged on every chunk and commit. The worker uses the same start and duration for its monotonic teardown deadline. A receiver uses its synchronized UTC clock and may buffer or commit a packet only while its receive time is strictly less than that instant; equality is expired. At or after expiry it discards the packet and all buffered chunks for that record, so an in-flight commit released after worker teardown cannot complete it. Expiry is an acceptance bound, not permission for the sender to continue, and missing or inconsistent expiry fields are rejected. The consumer also verifies the exact complete-record schema and keys and the expected correlation ID. Malformed UTF-8, JSON, schemas, keys, IDs, expiry, Base64, counts, indexes, sizes, complete-record digests, or canonical-result digests are ignored fail-closed rather than raising. If a valid record ID identifies malformed or inconsistent state, that incomplete assembly is discarded. JSON recursion or other pathological nesting failures follow the same rejected-record path and clear that assembly instead of escaping from commit handling.

One disposable consumer represents one evaluator attempt. Its first valid correlated record makes it terminal and clears all incomplete state; it rejects every later packet without processing or retention, including a full replay, a commit-only replay, or a different valid body carrying the same correlation ID and a later expiry. This provides lifetime exactly-once acceptance without an evictable replay cache. Digest verification hashes the exact UTF-8 bytes of the embedded canonical call_result value, so canonical decimal numbers are never coerced through binary floating point. A different room participant receives no packet. The resulting UTF-8 JSON is:

{
  "call_result": {},
  "correlation_id": "scenario-42-attempt-1",
  "result_digest": "canonical-call-result-sha256",
  "schema": "vaani.livekit-evaluation-evidence.v1"
}

call_result is the existing provider-neutral canonical CallResult. It contains the completed transcript, normalized tool and lifecycle activity, terminal outcome, language and provider history, timings, usage, and the built-in Judge report permitted by the selected profile. It contains no provider credentials or bot-private objects. call_result.judge remains bounded Judge evidence; the external evaluation system must compute its own scenario-aware verdict.

This seam is an ephemeral correlation and test-call transport, not production analytics delivery: it is development-only, has no durable retry or delivery report, and a recipient that disconnects can miss it. Use a Result Publisher through the completion handler below for durable analytics or storage.

Worker completion handler

run_livekit_worker(..., completion_handler=handler) may receive one async, typed handler for the terminal CallResult. VAANI calls it once after Closure and before the LiveKit job entrypoint returns. Its operation is bounded by the selected profile's existing result_publication_seconds deadline, outside the call lifecycle. The entrypoint and LiveKit shutdown callback share one absolute monotonic cutoff: shutdown cannot restart the budget, and work observed after the cutoff is never accepted as successful. A timeout or exception is logged, and the sealed result and base outcome are never rewritten.

If LiveKit cancels the entrypoint during shutdown, VAANI keeps the call task shielded, waits for the registered JobContext shutdown callback to trigger bounded closure, and then delivers the same sealed result exactly once. A missing or unsupported shutdown callback cannot fabricate a result; it only preserves the normal cancellation semantics. Call-result settlement is bounded by the profile's resource_retirement_seconds; a call that cannot produce a truthful result in that window is logged as terminal-result unavailable and is never published as successful. Timed-out completion code receives one cancellation request and a bounded resource_retirement_seconds settlement window (or the smaller configured operation budget). Completion handlers must allow CancelledError to terminate within that window. Python cannot force-kill an arbitrary in-process coroutine; a handler that violates this precondition is reported as a contract violation and may remain pending until it returns or the worker process exits. Deployments that know a handler cannot cooperate should set handler.__vaani_cancellation_cooperative__ = False; VAANI then rejects it at worker registration before it can run. Omitting the handler preserves the original worker behaviour.

Install the authenticated runtime with the postgres extra and apply the packaged v1.sql migration before starting a worker; see the PostgreSQL guide. The complete wiring below uses the rc43 API and publishes the exact sealed bytes. Create the Assistant and publisher once at worker startup, not once per call:

import logging
import os
from datetime import UTC, datetime

from vaani import (
    Assistant,
    CredentialScope,
    DeliveryPolicy,
    ResultPublicationRecord,
    bootstrap_environment,
)
from vaani.assistant.worker import run_livekit_worker
from vaani.publishers.postgres import (
    PostgresPublisherConfig,
    PostgresResultPublisher,
)

log = logging.getLogger(__name__)

environment = bootstrap_environment()
assistant = Assistant(
    name="Support",
    prompt="Help the caller briefly. Ask one question at a time.",
    greeting="Hello. How can I help?",
    environment=environment,
)


class DatabaseCredentials:
    scope = CredentialScope(
        provider="postgres",
        authentication_scheme="sqlalchemy-url",
        account_boundary="analytics-primary",
        region=None,
    )

    async def resolve(self):
        return os.environ["DATABASE_URL"]


database_credentials = DatabaseCredentials()
publisher = PostgresResultPublisher(
    PostgresPublisherConfig(
        sink_id="analytics-primary",
        table_name="vaani_call_results",
        credential_source=database_credentials,
    )
)


async def publish_completed_call(result):
    body = result.canonical_bytes()
    digest = result.result_digest()
    record = ResultPublicationRecord(
        idempotency_key=f"call-result:{result.call_id}",
        call_id=result.call_id,
        result_schema_version=result.provenance.result_schema_version,
        result_digest=digest,
        canonical_result_bytes=body,
        published_at=datetime.now(UTC),
    )
    report = await publisher.publish(
        record,
        DeliveryPolicy(timeout_seconds=20, max_attempts=1),
    )
    log.info(
        "result publication disposition=%s attempts=%s",
        report.disposition.value,
        report.attempts,
    )


run_livekit_worker(
    assistant,
    agent_name="support-agent",
    completion_handler=publish_completed_call,
)

Keep credentials inside the consumer-owned CredentialSource. A delivery timeout or failure is operational evidence, not permission to mutate the CallResult or report publication success.

Next: result/error reference or PostgreSQL publication.