External Platforms

OpenAI Realtime API

Integrate Trulience avatars with OpenAI's Realtime voice API using WebRTC
For LLMs

Overview

OpenAI’s Realtime API provides low-latency voice-to-voice conversations using WebRTC.

Working example: OpenAI integration example

How It Works

  • OpenAI Realtime API handles speech recognition, conversation logic, and voice synthesis
  • Trulience renders the avatar and synchronizes lip movements with OpenAI’s audio output
  • Audio is delivered via standard WebRTC RTCPeerConnection - no SDK required

Prerequisites

  • OpenAI API key with Realtime API access
  • A Trulience avatar configured for External Voice Platforms
  • Basic knowledge of React and WebRTC

Dashboard Configuration

  1. Open your avatar’s settings in the Trulience dashboard
  2. Navigate to the BRAIN tab
  3. Select ‘3rd Party AI’ mode
  4. Set ‘Service provider or framework’ to ‘External Voice Platforms’

This configuration disables Trulience’s built-in STT, LLM, and TTS, allowing OpenAI to handle these components.

Integration Steps

1. Install Dependencies

npm install @trulience/react-sdk

2. Get Ephemeral Key

OpenAI Realtime API requires an ephemeral key from your backend. Create an API endpoint:

// app/api/token/route.ts
export async function GET() {
  const response = await fetch('https://api.openai.com/v1/realtime/sessions', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: 'gpt-4o-realtime-preview-2024-12-17',
      voice: 'verse'
    })
  });

  const data = await response.json();
  return Response.json(data);
}

Then fetch it from the frontend:

const tokenResponse = await fetch('/api/token');
const data = await tokenResponse.json();
const ephemeralKey = data.client_secret.value;

3. Create RTCPeerConnection

const pc = new RTCPeerConnection();

// Add microphone input
const micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
pc.addTrack(micStream.getTracks()[0]);

4. Capture Remote Audio Track

This is where the Trulience integration happens:

// Listen for OpenAI's audio track
pc.ontrack = (event) => {
  const remoteStream = event.streams[0];

  // Route audio to Trulience avatar
  trulienceRef.current.setMediaStream(remoteStream);
  trulienceRef.current.getTrulienceObject().setSpeakerEnabled(true);

  // Mute any auto-created audio elements to prevent double audio
  const audioElement = document.querySelector('audio');
  if (audioElement) audioElement.muted = true;
};

5. Exchange SDP with OpenAI

// Create offer
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

// Send offer to OpenAI
const sdpResponse = await fetch(
  'https://api.openai.com/v1/realtime/calls?model=gpt-4o-realtime-preview-2024-12-17',
  {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${ephemeralKey}`,
      'Content-Type': 'application/sdp'
    },
    body: offer.sdp
  }
);

// Set remote description
const answer = {
  type: 'answer',
  sdp: await sdpResponse.text()
};
await pc.setRemoteDescription(answer);

Complete React Example

Here’s a full working implementation:

import React, { useRef, useState } from 'react';
import { TrulienceAvatar } from '@trulience/react-sdk';

function OpenAIRealtimeIntegration() {
  const trulienceRef = useRef(null);
  const audioRef = useRef(null);
  const [pc, setPc] = useState(null);
  const [connected, setConnected] = useState(false);

  const startSession = async () => {
    try {
      // Get ephemeral key from backend
      const tokenResponse = await fetch('/api/token');
      const data = await tokenResponse.json();
      const ephemeralKey = data?.client_secret?.value;

      if (!ephemeralKey) {
        throw new Error('Failed to get ephemeral key');
      }

      // Create peer connection
      const newPc = new RTCPeerConnection();

      // Set up audio playback (muted to avoid duplicate audio)
      if (audioRef.current) {
        audioRef.current.autoplay = true;
        audioRef.current.muted = true;
      }

      // Capture remote audio track
      newPc.ontrack = (e) => {
        if (audioRef.current) {
          audioRef.current.srcObject = e.streams[0];
        }

        // Route to Trulience
        if (trulienceRef.current) {
          trulienceRef.current.setMediaStream(e.streams[0]);
          trulienceRef.current.getTrulienceObject().setSpeakerEnabled(true);
        }
      };

      // Add microphone input
      const micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
      newPc.addTrack(micStream.getTracks()[0]);

      // Create offer
      const offer = await newPc.createOffer();
      await newPc.setLocalDescription(offer);

      // Send offer to OpenAI Realtime API
      const sdpResponse = await fetch(
        'https://api.openai.com/v1/realtime/calls?model=gpt-4o-realtime-preview-2024-12-17',
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${ephemeralKey}`,
            'Content-Type': 'application/sdp'
          },
          body: offer.sdp
        }
      );

      const answer = {
        type: 'answer',
        sdp: await sdpResponse.text()
      };
      await newPc.setRemoteDescription(answer);

      setPc(newPc);
      setConnected(true);
    } catch (error) {
      console.error('Failed to start session:', error);
    }
  };

  const endSession = () => {
    if (pc) {
      pc.getSenders().forEach(sender => sender.track?.stop());
      pc.close();
      setPc(null);
      setConnected(false);
    }
  };

  return (
    <div className="relative min-h-screen">
      <audio ref={audioRef} style={{ display: 'none' }} />

      <div className="absolute inset-0">
        <TrulienceAvatar
          ref={trulienceRef}
          url={process.env.NEXT_PUBLIC_TRULIENCE_SDK_URL}
          avatarId="your-avatar-id"
          token="your-trulience-token"
          width="100%"
          height="100%"
        />
      </div>

      <button
        onClick={connected ? endSession : startSession}
        className={`absolute bottom-6 left-1/2 -translate-x-1/2 px-6 py-3 rounded-lg text-white font-semibold ${
          connected ? 'bg-red-600 hover:bg-red-700' : 'bg-blue-600 hover:bg-blue-700'
        }`}
      >
        {connected ? 'Disconnect' : 'Connect'}
      </button>
    </div>
  );
}

export default OpenAIRealtimeIntegration;

Key Integration Points

Preventing Double Audio

OpenAI’s audio arrives via WebRTC track. To prevent hearing both OpenAI’s audio and the avatar’s audio:

  1. Create a muted <audio> element to receive the WebRTC stream
  2. Route the same stream to Trulience via setMediaStream()
  3. Enable the avatar’s speaker with setSpeakerEnabled(true)
// Muted audio element prevents browser from playing directly
<audio ref={audioRef} muted autoplay style={{ display: 'none' }} />

// In ontrack handler
audioRef.current.srcObject = e.streams[0];  // Muted playback
trulienceRef.current.setMediaStream(e.streams[0]);  // Avatar playback
trulienceRef.current.getTrulienceObject().setSpeakerEnabled(true);

Using Data Channels

You can also send events to OpenAI via a data channel:

const channel = pc.createDataChannel('oai-events');

channel.addEventListener('message', (e) => {
  const event = JSON.parse(e.data);
  console.log('OpenAI event:', event);
});

channel.onopen = () => {
  // Send initial instruction
  channel.send(JSON.stringify({
    type: 'response.create',
    response: {
      instructions: 'Introduce yourself as a Trulience AI avatar.'
    }
  }));
};

Troubleshooting

Issue: No audio from avatar

  • Verify setSpeakerEnabled(true) is called
  • Check that remote stream has audio tracks: e.streams[0].getAudioTracks().length > 0
  • Ensure the audio element is muted (not the stream)

Issue: Connection fails

  • Ensure ephemeral key is valid and not expired
  • Check browser console for ICE connection errors
  • Verify HTTPS is used (WebRTC requires secure context)

Issue: Avatar not lip-syncing

  • Check that setMediaStream() is called with a valid MediaStream
  • Ensure the avatar is configured for ‘External Voice Platforms’ in the dashboard
  • Verify the audio track is active: stream.getAudioTracks()[0].enabled === true

Example Code Repository

See our OpenAI integration example for a complete working implementation with error handling, state management, and additional features.

Next Steps