How to Measure Real-Time AI Avatar Latency

A latency number without measurement boundaries tells an engineer very little. A vendor may report 120 milliseconds inside a video renderer while an application reports 1.2 seconds from the end of a user’s sentence to audible playback. Both numbers can describe the same session.

The user experiences the complete turn. Microphone buffering, end-of-turn detection, speech recognition, agent work, speech synthesis, avatar rendering, network transport, decoding, and playout all sit on the critical path. Some stages run in sequence. A well-designed system overlaps others.

This guide defines that path and shows how to instrument it. For the broader control and media architecture, start with How Real-Time AI Avatar APIs Work.

Human turn-taking sets a hard reference point

People leave surprisingly small gaps in ordinary conversation. A study of question-and-answer sequences across ten languages found that response offsets most often landed between zero and 200 milliseconds. The cross-language median was about 100 milliseconds. The researchers also found variation by language and by the kind of response being given.^[1]

That result is context, not a sensible service-level objective for an AI system. Human listeners predict when a speaker will finish and begin planning a reply before the turn ends. An avatar pipeline may wait for an endpoint decision, finalize a transcript, run an agent, synthesize speech, and render video. Each operation consumes part of the gap the user notices.

The engineering goal is therefore not “match 100 ms.” It is to account for every millisecond between a clear start event and a clear playback event, then decide which delays can be removed, overlapped, or explained by the interface.

The seven-stage latency path

1. 01CaptureMicrophone frames
2. 02EndpointUser turn commits
3. 03RecognizeTranscript is ready
4. 04AgentFirst useful output
5. 05SpeechFirst audio chunk
6. 06RenderFirst video frame
7. 07PlayoutUser hears and sees it

The simple model is:

Tturn = Tendpoint + TASR + Tagent + TTTS + Trender + Ttransport + Tplayout - Toverlap

Do not add stage averages and call the result end-to-end latency unless the stages are strictly sequential. Streaming systems often start speech synthesis from partial agent output. The renderer may begin before the complete audio response exists. Measure the wall-clock turn as one interval, then use stage timings to diagnose it.

Choose the clock boundaries before collecting data

Most latency arguments are really boundary arguments. Define the start and stop events in the metric name or accompanying documentation.

Where does a turn start?

Four events are often confused:

• Last captured speech frame: the final audio frame containing the user’s voice reaches the client pipeline.
• Voice activity ends: a voice activity detector decides that speech has stopped.
• Turn is committed: the application accepts the pause as the end of the user’s turn.
• Transcript is final: the recognizer stops revising the utterance.

The difference between the last speech frame and the commit event is endpointing delay. This delay may be deliberate. A longer silence threshold reduces premature cutoffs, but it also creates dead air after every turn.

For user-perceived turn latency, turn_committed is a practical start boundary because it represents the point where the application has chosen to respond. Keep a separate speech_last_frame mark so endpointing remains visible.

Where does a turn end?

The first response can also mean several things:

• The first agent token reaches the application.
• The first synthesized audio chunk arrives.
• The first audio sample is submitted to the output device.
• The first audio sample is actually played.
• The first decodable video frame arrives.
• The first synchronized audio and video are presented.

Measure at least time to first audio and time to first video. Users may hear a response before the avatar moves, or see motion while the audio output remains buffered. A single number hides that split.

Instrument one turn with a monotonic clock

Browser events should use performance.now(), not Date.now(). The User Timing specification defines a high-resolution monotonic clock and the mark and measure interfaces for recording intervals. A monotonic clock does not jump when the device clock is corrected.^[2]

Keep all client-side marks for a turn under one identifier:

type TurnMark =
  | "speech_last_frame"
  | "turn_committed"
  | "agent_first_output"
  | "audio_chunk_received"
  | "audio_playout_started"
  | "video_frame_presented"
  | "interruption_detected"
  | "avatar_audio_stopped"

const turnMarks = new Map<string, Map<TurnMark, number>>()

function markTurn(turnId: string, name: TurnMark) {
  let marks = turnMarks.get(turnId)
  if (!marks) {
    marks = new Map()
    turnMarks.set(turnId, marks)
  }
  marks.set(name, performance.now())
}

function interval(
  turnId: string,
  start: TurnMark,
  end: TurnMark,
): number | undefined {
  const marks = turnMarks.get(turnId)
  const startAt = marks?.get(start)
  const endAt = marks?.get(end)

  if (startAt === undefined || endAt === undefined) return
  return endAt - startAt
}

// Map these calls to real events from your audio, agent, and avatar layers.
markTurn(turnId, "turn_committed")
markTurn(turnId, "audio_playout_started")

const timeToFirstAudio = interval(
  turnId,
  "turn_committed",
  "audio_playout_started",
)

The event names in this example are a measurement contract, not SDK-specific API names. Attach each mark to the event that actually occurred. A resolved network request does not prove that audio played, and a decoded frame does not prove that the browser presented it.

Client and server clocks should not be subtracted directly unless they are synchronized and you understand the remaining clock error. Use one local clock for the end-to-end browser interval. On the backend, create a distributed trace for the same turn_id and record spans for recognition, retrieval, agent generation, speech, and rendering. The OpenTelemetry trace conventions provide a common model for spans that cross services.^[3]

The client measurement answers, “How long did the user wait?” The trace answers, “Where did the time go?”

Measure the media path with WebRTC statistics

Application events explain the logical turn. Transport statistics explain why identical turns behave differently across networks and devices.

For WebRTC sessions, RTCPeerConnection.getStats() exposes inbound and outbound RTP reports. Useful fields include packet loss, jitter, round-trip time, frames decoded, frames dropped, and jitter-buffer counters. The W3C defines these fields in the WebRTC Statistics API.^[4]

async function readInboundVideoStats(pc: RTCPeerConnection) {
  const report = await pc.getStats()

  for (const stat of report.values()) {
    if (stat.type !== "inbound-rtp" || stat.kind !== "video") continue

    const averageJitterBufferMs =
      stat.jitterBufferEmittedCount > 0
        ? (stat.jitterBufferDelay / stat.jitterBufferEmittedCount) * 1000
        : undefined

    return {
      packetsLost: stat.packetsLost,
      jitterMs: stat.jitter * 1000,
      averageJitterBufferMs,
      framesDecoded: stat.framesDecoded,
      framesDropped: stat.framesDropped,
    }
  }
}

jitterBufferDelay and jitterBufferEmittedCount are cumulative counters. Their ratio gives the session average. To observe a short interval, sample both counters periodically and divide the change in delay by the change in emitted samples or frames.^[4]

Round-trip time is useful context, but it is not turn latency. It measures a network path represented by an RTCP report. The user-visible turn still includes endpointing, compute, media production, buffering, and playout.

The critical path is allowed to overlap

A naive implementation waits for every stage to finish:

final transcript → complete agent response → complete audio → complete video → playback

That design accumulates the full duration of each operation. A streaming pipeline changes the dependency graph:

• Partial transcripts can prepare context before the user finishes.
• Agent text can stream into speech synthesis by phrase.
• Audio chunks can enter the renderer before the response is complete.
• Encoded frames can leave the renderer while later frames are still being generated.
• The client can decode and buffer a small lead instead of waiting for the full turn.

Overlap reduces time to first response, but each boundary needs flow control. Sending text to speech too early can produce a sentence that the agent later revises. A tiny media buffer lowers delay while increasing the chance of stalls. The fastest pipeline in a perfect network may be less usable than a slightly buffered pipeline on mobile.

Record both time to first output and completion time. Fast first audio with a long mid-sentence stall is not a good turn.

Endpointing is often the largest invisible delay

Turn detection sits between listening and responding. It has two failure modes:

• Commit too late: every answer begins with an avoidable pause.
• Commit too early: the system cuts off a user who paused to think.

A fixed silence threshold is easy to implement and hard to tune across microphones, languages, speaking styles, and background noise. Better systems combine acoustic voice activity with transcript state or a semantic signal that estimates whether the utterance is complete.

Measure the distribution of turn_committed - speech_last_frame. Then label false endpoints, user continuations, and long pauses. Optimizing only the median can hide a detector that frequently interrupts slower speakers.

The interface can help without faking speed. A listening state should appear as soon as capture begins. A processing state can confirm that the turn was accepted. Filler speech should be used carefully; a quick “let me check” adds noise if the answer was already available.

Interruption latency needs its own test

An avatar that starts quickly but cannot stop quickly still feels unresponsive. Barge-in crosses several components:

1. The microphone must remain active while avatar audio is playing.
2. Echo cancellation must avoid treating the avatar’s voice as user speech.
3. The application must distinguish a real interruption from background noise.
4. Agent, speech, and rendering work must be canceled or detached from playback.
5. Buffered audio and video must stop without leaving an old response queued.

Start the interruption clock at the first confirmed user-speech frame. Stop it when avatar audio is no longer audible. Record whether the text generator and renderer stopped as separate events. The user cares about silence first; infrastructure owners also care about compute that continued after cancellation.

Test interruptions near the start, middle, and end of an avatar response. Include short acknowledgements such as “right” or “okay,” because they may be feedback rather than a request to take the turn.

Report a distribution, not an average

An average can make a system with occasional long stalls look acceptable. At minimum, report:

• p50: the middle turn and a useful picture of normal behavior.
• p95: the slower edge that regular users will still encounter.
• p99 or maximum: useful during incident analysis, but sensitive to sample size.
• Failure rate: turns that never produced playable media or exceeded the test timeout.

Separate cold session startup from warm turns. Split results by region, device class, browser, network type, and response shape when the sample supports it. A one-sentence answer without tools should not be compared directly with a turn that runs retrieval and several API calls.

Queueing deserves its own field. If a session waits for capacity before connecting, report allocation wait separately from session startup and per-turn latency.

A repeatable benchmark protocol

Use a fixed protocol so a change in the number means a change in the system:

1. Write the boundary contract. State the exact start and stop event for every metric.
2. Fix the test prompts. Include a short factual reply, a longer explanation, and a tool-using turn only when the product supports those jobs.
3. Control the environment. Record client region, browser, hardware, microphone path, and network condition.
4. Run repeated warm and cold sessions. Do not combine them in one percentile.
5. Capture stage marks and media statistics. A slow turn without diagnostic events cannot guide an optimization.
6. Report the sample count with each percentile. Tail percentiles based on a handful of turns are unstable.
7. Review recordings for behavioral errors. Premature endpoints, talk-over, frozen video, and late cancellation may not appear in the main latency number.

Store the raw event timestamps or histogram buckets, not only a dashboard average. A later investigation may need to regroup turns by model, region, release, or transport condition.

Protect user data while doing this. Turn identifiers and timings usually provide enough diagnostic value. Avoid logging transcripts or raw audio by default. If content capture is required for quality review, define retention, access, and consent rules before collection.

How to read a vendor latency claim

Before comparing two avatar systems, ask:

• What event starts the clock?
• Does the clock stop at server output, client receipt, decode, or actual playout?
• Is the number audio-only, video-only, or synchronized media?
• Is it a median, a tail percentile, or the best observed result?
• Were the session and models already warm?
• Did the test include turn detection, speech recognition, an agent, and speech synthesis?
• Where were the client and servers located?
• What happens to the number during packet loss, tool use, or an interruption?

If those details are absent, treat the claim as a component benchmark. It may still be valid, but it cannot predict the complete conversational gap.

What to optimize first

Measure before choosing a target. The largest stage on the critical path is the first candidate, but reliability and conversational behavior constrain every change.

• If endpointing dominates, test adaptive or semantic turn detection and track premature commits.
• If the agent dominates, stream useful output sooner and inspect retrieval or tool startup separately.
• If speech or rendering dominates, send incremental chunks and keep hot resources ready for active sessions.
• If transport or playout dominates, inspect round-trip time, loss, jitter buffers, decode time, and device-specific behavior.
• If p50 is healthy but p95 is poor, look for queueing, cold starts, retries, and regional routing before shaving a few milliseconds from the normal case.

A good latency dashboard lets an engineer open one slow turn and see its stage marks, trace, transport state, client environment, and cancellation history. A single green average cannot do that.

Frequently asked questions

Sources

1. Tanya Stivers et al. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26), 2009. doi:10.1073/pnas.0903616106
2. World Wide Web Consortium. User Timing. W3C technical specification.
3. OpenTelemetry. Trace semantic conventions. OpenTelemetry specification.
4. World Wide Web Consortium. Identifiers for WebRTC's Statistics API. W3C technical specification.