## Overview

OpenAI's Realtime API provides low-latency voice-to-voice conversations using WebRTC.

**Working example:** <a href="https://github.com/trulience/integration-examples/tree/main/web/oai" target="_blank">OpenAI integration example</a>

## How It Works

- **OpenAI Realtime API** handles speech recognition, conversation logic, and voice synthesis
- **Trulience** renders the avatar and synchronizes lip movements with OpenAI's audio output
- Audio is delivered via standard WebRTC `RTCPeerConnection` - no SDK required

## Prerequisites

- OpenAI API key with Realtime API access
- A Trulience avatar configured for External Voice Platforms
- Basic knowledge of React and WebRTC

## Dashboard Configuration

1. Open your avatar's settings in the Trulience dashboard
2. Navigate to the **BRAIN** tab
3. Select **'3rd Party AI'** mode
4. Set 'Service provider or framework' to **'External Voice Platforms'**

This configuration disables Trulience's built-in STT, LLM, and TTS, allowing OpenAI to handle these components.

## Integration Steps

### 1. Install Dependencies

```bash
npm install @trulience/react-sdk
```

### 2. Get Ephemeral Key

OpenAI Realtime API requires an ephemeral key from your backend. Create an API endpoint:

```javascript
// app/api/token/route.ts
export async function GET() {
  const response = await fetch('https://api.openai.com/v1/realtime/sessions', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: 'gpt-4o-realtime-preview-2024-12-17',
      voice: 'verse'
    })
  });

  const data = await response.json();
  return Response.json(data);
}
```

Then fetch it from the frontend:

```javascript
const tokenResponse = await fetch('/api/token');
const data = await tokenResponse.json();
const ephemeralKey = data.client_secret.value;
```

### 3. Create RTCPeerConnection

```javascript
const pc = new RTCPeerConnection();

// Add microphone input
const micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
pc.addTrack(micStream.getTracks()[0]);
```

### 4. Capture Remote Audio Track

This is where the Trulience integration happens:

```javascript
// Listen for OpenAI's audio track
pc.ontrack = (event) => {
  const remoteStream = event.streams[0];

  // Route audio to Trulience avatar
  trulienceRef.current.setMediaStream(remoteStream);
  trulienceRef.current.getTrulienceObject().setSpeakerEnabled(true);

  // Mute any auto-created audio elements to prevent double audio
  const audioElement = document.querySelector('audio');
  if (audioElement) audioElement.muted = true;
};
```

### 5. Exchange SDP with OpenAI

```javascript
// Create offer
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

// Send offer to OpenAI
const sdpResponse = await fetch(
  'https://api.openai.com/v1/realtime/calls?model=gpt-4o-realtime-preview-2024-12-17',
  {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${ephemeralKey}`,
      'Content-Type': 'application/sdp'
    },
    body: offer.sdp
  }
);

// Set remote description
const answer = {
  type: 'answer',
  sdp: await sdpResponse.text()
};
await pc.setRemoteDescription(answer);
```

## Complete React Example

Here's a full working implementation:

```jsx
import React, { useRef, useState } from 'react';
import { TrulienceAvatar } from '@trulience/react-sdk';

function OpenAIRealtimeIntegration() {
  const trulienceRef = useRef(null);
  const audioRef = useRef(null);
  const [pc, setPc] = useState(null);
  const [connected, setConnected] = useState(false);

  const startSession = async () => {
    try {
      // Get ephemeral key from backend
      const tokenResponse = await fetch('/api/token');
      const data = await tokenResponse.json();
      const ephemeralKey = data?.client_secret?.value;

      if (!ephemeralKey) {
        throw new Error('Failed to get ephemeral key');
      }

      // Create peer connection
      const newPc = new RTCPeerConnection();

      // Set up audio playback (muted to avoid duplicate audio)
      if (audioRef.current) {
        audioRef.current.autoplay = true;
        audioRef.current.muted = true;
      }

      // Capture remote audio track
      newPc.ontrack = (e) => {
        if (audioRef.current) {
          audioRef.current.srcObject = e.streams[0];
        }

        // Route to Trulience
        if (trulienceRef.current) {
          trulienceRef.current.setMediaStream(e.streams[0]);
          trulienceRef.current.getTrulienceObject().setSpeakerEnabled(true);
        }
      };

      // Add microphone input
      const micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
      newPc.addTrack(micStream.getTracks()[0]);

      // Create offer
      const offer = await newPc.createOffer();
      await newPc.setLocalDescription(offer);

      // Send offer to OpenAI Realtime API
      const sdpResponse = await fetch(
        'https://api.openai.com/v1/realtime/calls?model=gpt-4o-realtime-preview-2024-12-17',
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${ephemeralKey}`,
            'Content-Type': 'application/sdp'
          },
          body: offer.sdp
        }
      );

      const answer = {
        type: 'answer',
        sdp: await sdpResponse.text()
      };
      await newPc.setRemoteDescription(answer);

      setPc(newPc);
      setConnected(true);
    } catch (error) {
      console.error('Failed to start session:', error);
    }
  };

  const endSession = () => {
    if (pc) {
      pc.getSenders().forEach(sender => sender.track?.stop());
      pc.close();
      setPc(null);
      setConnected(false);
    }
  };

  return (
    <div className="relative min-h-screen">
      <audio ref={audioRef} style={{ display: 'none' }} />

      <div className="absolute inset-0">
        <TrulienceAvatar
          ref={trulienceRef}
          url={process.env.NEXT_PUBLIC_TRULIENCE_SDK_URL}
          avatarId="your-avatar-id"
          token="your-trulience-token"
          width="100%"
          height="100%"
        />
      </div>

      <button
        onClick={connected ? endSession : startSession}
        className={`absolute bottom-6 left-1/2 -translate-x-1/2 px-6 py-3 rounded-lg text-white font-semibold ${
          connected ? 'bg-red-600 hover:bg-red-700' : 'bg-blue-600 hover:bg-blue-700'
        }`}
      >
        {connected ? 'Disconnect' : 'Connect'}
      </button>
    </div>
  );
}

export default OpenAIRealtimeIntegration;
```

## Key Integration Points

### Preventing Double Audio

OpenAI's audio arrives via WebRTC track. To prevent hearing both OpenAI's audio and the avatar's audio:

1. Create a muted `<audio>` element to receive the WebRTC stream
2. Route the same stream to Trulience via `setMediaStream()`
3. Enable the avatar's speaker with `setSpeakerEnabled(true)`

```javascript
// Muted audio element prevents browser from playing directly
<audio ref={audioRef} muted autoplay style={{ display: 'none' }} />

// In ontrack handler
audioRef.current.srcObject = e.streams[0];  // Muted playback
trulienceRef.current.setMediaStream(e.streams[0]);  // Avatar playback
trulienceRef.current.getTrulienceObject().setSpeakerEnabled(true);
```

### Using Data Channels

You can also send events to OpenAI via a data channel:

```javascript
const channel = pc.createDataChannel('oai-events');

channel.addEventListener('message', (e) => {
  const event = JSON.parse(e.data);
  console.log('OpenAI event:', event);
});

channel.onopen = () => {
  // Send initial instruction
  channel.send(JSON.stringify({
    type: 'response.create',
    response: {
      instructions: 'Introduce yourself as a Trulience AI avatar.'
    }
  }));
};
```

## Troubleshooting

**Issue: No audio from avatar**
- Verify `setSpeakerEnabled(true)` is called
- Check that remote stream has audio tracks: `e.streams[0].getAudioTracks().length > 0`
- Ensure the audio element is muted (not the stream)

**Issue: Connection fails**
- Ensure ephemeral key is valid and not expired
- Check browser console for ICE connection errors
- Verify HTTPS is used (WebRTC requires secure context)

**Issue: Avatar not lip-syncing**
- Check that `setMediaStream()` is called with a valid MediaStream
- Ensure the avatar is configured for 'External Voice Platforms' in the dashboard
- Verify the audio track is active: `stream.getAudioTracks()[0].enabled === true`

## Example Code Repository

See our <a href="https://github.com/trulience/integration-examples/tree/main/web/oai" target="_blank">OpenAI integration example</a> for a complete working implementation with error handling, state management, and additional features.

## Next Steps

- Review the [Integration Guide](/docs/external-platforms/integration-guide) for general audio routing patterns
- Explore OpenAI's <a href="https://platform.openai.com/docs/guides/realtime" target="_blank">Realtime API documentation</a>
- Check out our [Javascript SDK documentation](/docs/developer/javascript) for more avatar controls