@dtelecom/agents

AI voice agent framework for dTelecom rooms. Build real-time voice agents that join WebRTC rooms and interact with participants using speech-to-text, LLMs, and text-to-speech.

Architecture

Participant mic -> SFU -> server-sdk-node (Opus decode) -> PCM16 16kHz
  -> STT plugin -> transcription
  -> LLM plugin -> streaming text response
  -> Sentence splitter -> TTS plugin -> PCM16 16kHz
  -> AudioSource (upsample 48kHz + Opus encode) -> SFU -> Participants

The pipeline uses a producer/consumer pattern: LLM tokens are split into sentences and queued, while a consumer synthesizes and plays audio concurrently. This minimizes time-to-first-audio.

Install

npm install @dtelecom/agents @dtelecom/server-sdk-js @dtelecom/server-sdk-node

Quick Start

import { VoiceAgent, setLogLevel } from '@dtelecom/agents';
import { DeepgramSTT, OpenRouterLLM, CartesiaTTS } from '@dtelecom/agents/providers';

const agent = new VoiceAgent({
  stt: new DeepgramSTT({ apiKey: process.env.DEEPGRAM_API_KEY! }),
  llm: new OpenRouterLLM({
    apiKey: process.env.OPENROUTER_API_KEY!,
    model: 'openai/gpt-4o',
  }),
  tts: new CartesiaTTS({
    apiKey: process.env.CARTESIA_API_KEY!,
    voiceId: 'your-voice-id',
  }),
  instructions: 'You are a helpful voice assistant.',
});

await agent.start({
  room: 'my-room',
  apiKey: process.env.DTELECOM_API_KEY!,
  apiSecret: process.env.DTELECOM_API_SECRET!,
});

Plugin Interfaces

STT

interface STTPlugin {
  createStream(options?: STTStreamOptions): STTStream;
}

interface STTStream {
  sendAudio(pcm16: Buffer): void;
  on(event: 'transcription', cb: (result: TranscriptionResult) => void): this;
  on(event: 'error', cb: (error: Error) => void): this;
  close(): Promise<void>;
}

LLM

interface LLMPlugin {
  chat(messages: Message[], signal?: AbortSignal): AsyncGenerator<LLMChunk>;
  warmup?(systemPrompt: string): Promise<void>;
}

TTS

interface TTSPlugin {
  synthesize(text: string, signal?: AbortSignal): AsyncGenerator<Buffer>;
  warmup?(): Promise<void>;
}

Configuration

Option	Type	Default	Description
stt	STTPlugin	required	Speech-to-text provider
llm	LLMPlugin	required	Language model provider
tts	TTSPlugin	undefined	Text-to-speech provider (text-only if omitted)
instructions	string	required	System prompt for the LLM
respondMode	'always' | 'addressed'	'always'	When to respond to speech
agentName	string	'assistant'	Name for addressed-mode detection
nameVariants	string[]	[]	Additional names to respond to
onDataMessage	DataMessageHandler	undefined	Callback for data channel messages

Events

Event	Payload	Description
transcription	{ text, isFinal, speaker }	STT transcription result
response	string	Full agent response text
speaking	boolean	Agent started/stopped speaking
error	Error	Pipeline error
connected	—	Agent connected to room
disconnected	string?	Agent disconnected

Custom Providers

Implement the plugin interface and pass it to VoiceAgent:

import { BaseSTTStream, type STTPlugin, type STTStreamOptions } from '@dtelecom/agents';

class MySTTStream extends BaseSTTStream {
  sendAudio(pcm16: Buffer): void {
    // Send audio to your STT service
  }

  async close(): Promise<void> {
    // Clean up
  }
}

class MySTT implements STTPlugin {
  createStream(options?: STTStreamOptions) {
    return new MySTTStream();
  }
}

const agent = new VoiceAgent({
  stt: new MySTT(),
  // ...
});

Data Channels

Receive data channel messages from participants:

const agent = new VoiceAgent({
  // ...
  onDataMessage: (payload, participantIdentity, topic) => {
    const message = JSON.parse(new TextDecoder().decode(payload));
    console.log(`${participantIdentity} sent:`, message);
  },
});

License

Apache-2.0