Skip to main content
ElevenLabs Documentation Docs
current

Search documentation

Type to search this documentation.

On this pageOverview

Client events

Understand and handle real-time events received by the client during conversational applications.

Client events are system-level events sent from the server to the client that facilitate real-time communication. These events deliver audio, transcription, agent responses, and other critical information to the client application.

Overview

Section titled “Overview”

Client events are essential for maintaining the real-time nature of conversations. They provide everything from initialization metadata to processed audio and agent responses.

Client event types

Section titled “Client event types”
conversation_initiation_metadata
  • Automatically sent when starting a conversation
  • Initializes conversation settings and parameters
JSON
// Example initialization metadata
{
  "type": "conversation_initiation_metadata",
  "conversation_initiation_metadata_event": {
    "conversation_id": "conv_123",
    "agent_output_audio_format": "pcm_44100",  // TTS output format
    "user_input_audio_format": "pcm_16000"    // ASR input format
  }
}
queue_status
  • Sent only to callers held in the call queue while the agent is at its concurrency limit
  • waiting is sent once, after conversation_initiation_metadata and before any hold audio
  • admitted or timed_out is sent once when the wait ends. timed_out is followed by a WebSocket close with code 4300
  • Always sent to queued callers. It does not need to be enabled in the agent’s client_events configuration
JSON
// Example queue status event structure
{
  "type": "queue_status",
  "queue_status_event": {
    "status": "waiting"  // "waiting" | "admitted" | "timed_out"
  }
}
JavaScript
// Example queue status handler
websocket.on('queue_status', (event) => {
  const { status } = event.queue_status_event;
  if (status === 'waiting') {
    showWaitingState();
  } else if (status === 'admitted') {
    hideWaitingState();
  } else if (status === 'timed_out') {
    showAllAgentsBusyMessage();
  }
});
ping
  • Health check event requiring immediate response
  • Automatically handled by SDK
  • Used to maintain WebSocket connection
JSON
  // Example ping event structure
  {
    "ping_event": {
      "event_id": 123456,
      "ping_ms": 50  // Optional, estimated latency in milliseconds
    },
    "type": "ping"
  }
JavaScript
  // Example ping handler
  websocket.on('ping', () => {
    websocket.send('pong');
  });
audio
  • Contains base64 encoded audio for playback
  • Includes numeric event ID for tracking and sequencing
  • Handles voice output streaming
  • Includes alignment data with character-level timing information
JSON
// Example audio event structure
{
  "audio_event": {
    "audio_base_64": "base64_encoded_audio_string",
    "event_id": 12345,
    "alignment": {  // Character-level timing data
      "chars": ["H", "e", "l", "l", "o"],
      "char_durations_ms": [50, 30, 40, 40, 60],
      "char_start_times_ms": [0, 50, 80, 120, 160]
    }
  },
  "type": "audio"
}
JavaScript
// Example audio event handler
websocket.on('audio', (event) => {
  const { audio_event } = event;
  const { audio_base_64, event_id, alignment } = audio_event;
  audioPlayer.play(audio_base_64);

  // Use alignment data for synchronized text display
  const { chars, char_start_times_ms } = alignment;
  chars.forEach((char, i) => {
    setTimeout(() => highlightCharacter(char, i), char_start_times_ms[i]);
  });
});
user_transcript
  • Contains finalized speech-to-text results
  • Represents complete user utterances
  • Used for conversation history
JSON
// Example transcript event structure
{
  "type": "user_transcript",
  "user_transcription_event": {
    "user_transcript": "Hello, how can you help me today?"
  }
}
JavaScript
// Example transcript handler
websocket.on('user_transcript', (event) => {
  const { user_transcription_event } = event;
  const { user_transcript } = user_transcription_event;
  updateConversationHistory(user_transcript);
});
agent_response
  • Contains the complete agent message
  • Sent once that message is finished, so in voice conversations it usually arrives after the message’s audio has already started streaming.
  • Used for display and history
JSON
// Example response event structure
{
  "type": "agent_response",
  "agent_response_event": {
    "agent_response": "Hello, how can I assist you today?"
  }
}
JavaScript
// Example response handler
websocket.on('agent_response', (event) => {
  const { agent_response_event } = event;
  const { agent_response } = agent_response_event;
  displayAgentMessage(agent_response);
});
agent_response_correction
  • Contains truncated response after interruption
  • Updates displayed message
  • Maintains conversation accuracy
JSON
// Example response correction event structure
{
  "type": "agent_response_correction",
  "agent_response_correction_event": {
    "original_agent_response": "Let me tell you about the complete history...",
    "corrected_agent_response": "Let me tell you about..."  // Truncated after interruption
  }
}
JavaScript
// Example response correction handler
websocket.on('agent_response_correction', (event) => {
  const { agent_response_correction_event } = event;
  const { corrected_agent_response } = agent_response_correction_event;
  displayAgentMessage(corrected_agent_response);
});
agent_response_metadata
  • Contains arbitrary metadata from a custom LLM response
  • Only sent when using a custom LLM
  • Must be explicitly enabled in the agent’s client_events configuration
JSON
// Example agent response metadata event structure
{
  "type": "agent_response_metadata",
  "agent_response_metadata_event": {
    "metadata": {
      // Any key-value pairs returned by your custom LLM
      "key": "value"
    },
    "event_id": 12345
  }
}
JavaScript
// Example metadata handler
websocket.on('agent_response_metadata', (event) => {
  const { agent_response_metadata_event } = event;
  const { metadata, event_id } = agent_response_metadata_event;

  // Use metadata for UI updates, logging, or analytics
  console.log(`Response ${event_id} metadata:`, metadata);
  updateResponseDetails(metadata);
});
client_tool_call
  • Represents a function call the agent wants the client to execute
  • Contains tool name, tool call ID, and parameters
  • Requires client-side execution of the function and sending the result back to the server
JSON
// Example tool call event structure
{
  "type": "client_tool_call",
  "client_tool_call": {
    "tool_name": "search_database",
    "tool_call_id": "call_123456",
    "parameters": {
      "query": "user information",
      "filters": {
        "date": "2024-01-01"
      }
    }
  }
}
JavaScript
// Example tool call handler
websocket.on('client_tool_call', async (event) => {
  const { client_tool_call } = event;
  const { tool_name, tool_call_id, parameters } = client_tool_call;

  try {
    const result = await executeClientTool(tool_name, parameters);
    // Send success response back to continue conversation
    websocket.send({
      type: "client_tool_result",
      tool_call_id: tool_call_id,
      result: result,
      is_error: false
    });
  } catch (error) {
    // Send error response if tool execution fails
    websocket.send({
      type: "client_tool_result",
      tool_call_id: tool_call_id,
      result: error.message,
      is_error: true
    });
  }
});
agent_tool_response
  • Indicates when the agent has executed a tool function
  • Contains tool metadata and execution status
  • Provides visibility into agent tool usage during conversations
JSON
// Example agent tool response event structure
{
  "type": "agent_tool_response",
  "agent_tool_response": {
    "tool_name": "skip_turn",
    "tool_call_id": "skip_turn_c82ca55355c840bab193effb9a7e8101",
    "tool_type": "system",
    "is_error": false
  }
}
JavaScript
// Example agent tool response handler
websocket.on('agent_tool_response', (event) => {
  const { agent_tool_response } = event;
  const { tool_name, tool_call_id, tool_type, is_error } = agent_tool_response;

  if (is_error) {
    console.error(`Agent tool ${tool_name} failed:`, tool_call_id);
  } else {
    console.log(`Agent executed ${tool_type} tool: ${tool_name}`);
  }
});
agent_tool_response_full_payload
  • Mirrors agent_tool_response and additionally streams the tool’s full result payload as a string in full_tool_result.
  • Surfaces tool output in the client for display or downstream processing.
  • Must be explicitly enabled in the agent’s client_events configuration.
JSON
// Example agent tool response full payload event structure
{
  "type": "agent_tool_response_full_payload",
  "agent_tool_response_full_payload": {
    "tool_name": "lookup_order",
    "tool_call_id": "lookup_order_c82ca55355c840bab193effb9a7e8101",
    "tool_type": "webhook",
    "is_error": false,
    "full_tool_result": "{\"order_id\": \"ORD-789\", \"status\": \"shipped\"}",
    "truncated": false
  }
}
React
// Example agent tool response full payload handler (using @elevenlabs/react)
import { ConversationProvider } from '@elevenlabs/react';

function App() {
  return (
    <ConversationProvider
      onAgentToolResponse={(response) => {
        if (!('full_tool_result' in response)) return;
        const { tool_name, tool_call_id, is_error, full_tool_result, truncated } = response;

        if (is_error) {
          console.error(`Agent tool ${tool_name} failed:`, tool_call_id);
        } else {
          console.log(`Tool ${tool_name} returned:`, full_tool_result);
        }

        if (truncated) {
          console.warn(`Tool ${tool_name} result was truncated (exceeded 64 KB).`);
        }
      }}
    >
      <Agent />
    </ConversationProvider>
  );
}
JavaScript
// Example agent tool response full payload handler (using @elevenlabs/client)
import { Conversation } from '@elevenlabs/client';

const conversation = await Conversation.startSession({
  agentId: 'agent_7101k5zvyjhmfg983brhmhkd98n6',
  onAgentToolResponse: (response) => {
    if (!('full_tool_result' in response)) return;
    const { tool_name, tool_call_id, is_error, full_tool_result, truncated } = response;

    if (is_error) {
      console.error(`Agent tool ${tool_name} failed:`, tool_call_id);
    } else {
      console.log(`Tool ${tool_name} returned:`, full_tool_result);
    }

    if (truncated) {
      console.warn(`Tool ${tool_name} result was truncated (exceeded 64 KB).`);
    }
  },
});
vad_score
  • Voice Activity Detection score event
  • Indicates the probability that the user is speaking
  • Values range from 0 to 1, where higher values indicate higher confidence of speech
JSON
// Example VAD score event
{
  "type": "vad_score",
  "vad_score_event": {
    "vad_score": 0.95
  }
}
mcp_tool_call
  • Indicates when the agent has executed a MCP tool function
  • Contains tool name, tool call ID, and parameters
  • Called with one of four states: loading, awaiting_approval, success and failure.
JSON
{
  "type": "mcp_tool_call",
  "mcp_tool_call": {
    "service_id": "xJ8kP2nQ7sL9mW4vR6tY",
    "tool_call_id": "call_123456",
    "tool_name": "search_database",
    "tool_description": "Search the database for user information",
    "parameters": {
      "query": "user information",
    },
    "timestamp": "2024-09-30T14:23:45.123456+00:00",
    "state": "loading",
    "approval_timeout_secs": 10
  }
}
agent_chat_response_part
  • Streams the agent’s response text as it is generated, as start, delta and stop messages
  • Always sent in text-only mode; in voice conversations it must be explicitly enabled in the agent’s client_events configuration
  • Not sent while the agent or an active procedure uses a blocking guardrail, which has to evaluate the whole response before any of it is released
  • response_id identifies the message being streamed and matches the response_id of the agent_response that later commits it
JSON
// Example start event
{
  "type": "agent_chat_response_part",
  "text_response_part": {
    "type": "start",
    "text": "",
    "event_id": 12345,
    "response_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"
  }
}
JSON
// Example delta event with text chunk
{
  "type": "agent_chat_response_part",
  "text_response_part": {
    "type": "delta",
    "text": "Hello, how can I",
    "event_id": 12345,
    "response_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"
  }
}
JSON
// Example stop event
{
  "type": "agent_chat_response_part",
  "text_response_part": {
    "type": "stop",
    "text": "",
    "event_id": 12345,
    "response_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"
  }
}
JavaScript
// Example handler
websocket.on('agent_chat_response_part', (event) => {
  const { text_response_part } = event;
  const { type: partType, text, response_id } = text_response_part;

  if (partType === 'start') {
    initializeResponseBuffer(response_id);
  } else if (partType === 'delta') {
    appendToResponseBuffer(response_id, text);
  } else if (partType === 'stop') {
    finalizeResponse(response_id);
  }
});
agent_reasoning_response_part

agent_reasoning_response_part streams model-provided reasoning during text-only conversations. Enable the event in client_events and turn on Reasoning summary for the agent. The server sends start, delta, and stop messages. It does not send this event during voice conversations or while the agent or an active procedure uses blocking guardrails.

Event payload

JSON
{
  "type": "agent_reasoning_response_part",
  "reasoning_response_part": {
    "type": "delta",
    "text": "The user asked to cancel, so I should verify the account before continuing.",
    "event_id": 123456
  }
}

Start and stop events use an empty text value.

Handle reasoning events

JavaScript
import { Conversation } from '@elevenlabs/client';

const conversation = await Conversation.startSession({
  agentId: 'agent_7101k5zvyjhmfg983brhmhkd98n6',
  textOnly: true,
  onAgentReasoningResponsePart: ({ type, text, event_id }) => {
    if (type === 'start') {
      initializeReasoningBuffer(event_id);
    } else if (type === 'delta') {
      appendToReasoningBuffer(text);
    } else if (type === 'stop') {
      finalizeReasoning();
    }
  },
});
agent_response_complete
  • Fires when the agent has finished its response, including any pending tool calls. After this event, the agent will only produce further output if the user provides new input or a turn timeout triggers a new turn.
  • Must be explicitly enabled in the agent’s client_events configuration
JSON
// Example agent response complete event structure
{
  "type": "agent_response_complete",
  "agent_response_complete_event": {
    "event_id": 12345
  }
}
JavaScript
// Example handler
websocket.on('agent_response_complete', (event) => {
  const { agent_response_complete_event } = event;
  const { event_id } = agent_response_complete_event;

  console.log(`Agent response ${event_id} complete`);
});
guardrail_triggered
  • Fires when a guardrail violation ends the conversation. Not sent when a guardrail triggers a retry that succeeds.
  • The event itself is the signal - it carries no payload beyond the type field.
  • Must be explicitly enabled in the agent’s client_events configuration.
JSON
// Example guardrail triggered event structure
{
  "type": "guardrail_triggered"
}
JavaScript
// Example guardrail triggered handler (using @elevenlabs/client)
import { Conversation } from '@elevenlabs/client';

const conversation = await Conversation.startSession({
  agentId: 'agent_7101k5zvyjhmfg983brhmhkd98n6',
  onGuardrailTriggered: () => {
    console.warn('Guardrail triggered — conversation will end.');
  },
});

Event flow

Section titled “Event flow”

Here’s a typical sequence of events during a conversation:

When an agent is at its concurrency limit and call queueing is enabled, the server sends queue_status events between conversation_initiation_metadata and the first audio event. Hold audio is delivered as audio events until the caller is admitted.

Best practices

Section titled “Best practices”

  1. Error handling

    • Implement proper error handling for each event type
    • Log important events for debugging
    • Handle connection interruptions gracefully

  2. Audio management

    • Buffer audio chunks appropriately
    • Implement proper cleanup on interruption
    • Handle audio resource management

  3. Connection management

    • Respond to PING events promptly
    • Implement reconnection logic
    • Monitor connection health

Troubleshooting

Section titled “Troubleshooting”
Connection issues
  • Ensure proper WebSocket connection
  • Check PING/PONG responses
  • Verify API credentials
Audio problems
  • Check audio chunk handling
  • Verify audio format compatibility
  • Monitor memory usage
Event handling
  • Log all events for debugging
  • Implement error boundaries
  • Check event handler registration
Suggest an edit

Propose a replacement for this page. The site team reviews it before applying any changes.

Export
Documentation menu