Voice AI

WhatsApp & Voice AI Assistant

Real-time WebRTC audio processing engine powered by Whisper STT, ElevenLabs TTS, and FastAPI websocket networks.

📖 Project Overview

This project showcases a high-throughput, multi-channel conversational system. Deployed using FastAPI, it facilitates continuous audio exchanges via WebRTC websockets. Raw incoming user audio is streamed directly to a self-hosted Whisper model for STT.

The transcribed content initiates a reasoning loop using LLMs that maintains session logs and memory context on Redis. The resulting response is converted back into realistic human speech via ElevenLabs APIs, keeping end-to-end voice latency below 1 second.

⚙️ Deployment Checklist & Runbook

Handle Audio Stream via FastAPI WebSocket

Establish WebRTC connection endpoints and read media binary buffers asynchronously:

python

from fastapi import APIRouter, WebSocket

router = APIRouter()

@router.websocket("/media-stream")
async def handle_media_stream(websocket: WebSocket):
    await websocket.accept()
    while True:
        data = await websocket.receive_bytes()
        # Feed binary data chunks to Whisper processor
        text_segment = await transcribe_audio_chunk(data)
        if text_segment:
            await process_text_and_respond(text_segment, websocket)

Configure ElevenLabs Synthesis

Connect the ElevenLabs API client using WebSocket streams for audio synthesis:

python

import websockets
import json

async def stream_tts(text: str):
    uri = "wss://api.elevenlabs.io/v1/text-to-speech/voice-id/stream-input"
    async with websockets.connect(uri) as ws:
        bos = {"text": " ", "voice_settings": {"stability": 0.5}}
        await ws.send(json.dumps(bos))
        
        # Send text chunks to generate audio bytes
        await ws.send(json.dumps({"text": text}))
        await ws.send(json.dumps({"text": ""})) # EOS

Initialize Redis Sessions

Provision Redis schema to handle temporary text transcripts & conversation parameters:

bash

# Start isolated Redis container:
docker run -d --name voice-redis \
  -p 6379:6379 \
  redis:7.0-alpine

🔄 Configuration & Environment Keys

Environment Variable	Description	Suggested Value
`ELEVEN_API_KEY`	API credential for rendering TTS speech outputs.	`xxxx-eleven-xxxx`
`WHISPER_URL`	URL endpoint pointing to the local Whisper inference instance.	`http://whisper-service.voice-ai.svc:9000/transcribe`
`REDIS_VOICE_URL`	Local Redis URL address for active conversation cache logs.	`redis://voice-redis:6379/1`

📊 Architectural Workflow

graph TD
    Caller[WhatsApp / Call User] -->|WebRTC Audio Stream| FastAPI[FastAPI Gateway]
    FastAPI -->|Speech Chunks| Whisper[Whisper STT Service]
    Whisper -->|Text Transcription| LLM[LLM Reasoning Chain]
    LLM -->|Check context| Redis[(Redis Sessions)]
    LLM -->|Generate Text Response| ElevenLabs[ElevenLabs TTS WebSocket]
    ElevenLabs -->|Synthesized Speech Bytes| FastAPI
    FastAPI -->|Audio Stream| Caller

🛠️ Useful CLI Reference

# Launch local developer server:
uvicorn app.main:app --host 0.0.0.0 --port 5000

# Start Ngrok secure tunnel for webhook routing:
ngrok http 5000

# Test audio stream using Python websockets tool:
python scripts/test_stream.py --file test_query.wav