Skip to main content

System Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                      External APIs                          │
│  (Gemini, OpenAI, Hume, Twilio, Plivo, Gmail, etc.)         │
└────────────────────┬────────────────────────────────────────┘

┌────────────────────▼────────────────────────────────────────┐
│                   API Gateway Layer                         │
│         FastAPI (http://localhost:8010)                     │
│                                                             │
│  ├── REST API Endpoints (/api/agents, /api/calls, etc.)     │
│  ├── WebSocket Streaming (/ws/stream)                       │
│  └── Webhook Receivers (/webhook/twilio, etc.)              │
└─────────┬──────────┬──────────┬──────────┬─────────────────-┘
          │          │          │          │
    ┌─────▼──┐  ┌────▼────┐ ┌──▼────┐ ┌──▼────┐
    │ Agent  │  │  Call   │ │Org &  │ │Batch  │
    │Service │  │ Service │ │Auth   │ │Calls  │
    └─────┬──┘  └────┬────┘ └──┬────┘ └──┬────┘
          │          │         │         │
    ┌─────▼──────────▼─────────▼─────────▼──────┐
    │                                           │
    │    AI Bridges Layer                       │
    │  ┌──────────┐ ┌──────────┐ ┌──────────┐   │
    │  │  Gemini  │ │  OpenAI  │ │   Hume   │   │
    │  └──────────┘ └──────────┘ └──────────┘   │
    │                                           │
    └─────────────┬─────────────────────────────┘

    ┌─────────────▼──────────────────┐
    │   Communication Layer          │
    │  ┌──────────┐ ┌──────────┐     │
    │  │  Twilio  │ │  Plivo   │     │
    │  └──────────┘ └──────────┘     │
    │  ┌──────────────────────────┐  │
    │  │  LiveKit (RTC Media)     │  │
    │  └──────────────────────────┘  │
    └─────────────┬──────────────────┘

    ┌─────────────▼──────────────────┐
    │  Integration Layer (MCP)       │
    │  ┌──────────┐ ┌──────────┐     │
    │  │ Telegram │ │  Gmail   │     │
    │  └──────────┘ └──────────┘     │
    │  ┌──────────┐ ┌──────────┐     │
    │  │WhatsApp  │ │ Shopify  │     │
    │  └──────────┘ └──────────┘     │
    └─────────────┬──────────────────┘

    ┌─────────────▼──────────────────┐
    │  Data Persistence Layer        │
    │  ┌──────────────────────────┐  │
    │  │    Supabase (PostgreSQL) │  │
    │  │  (Organizations, Agents, │  │
    │  │   Calls, Schedules)      │  │
    │  └──────────────────────────┘  │
    └────────────────────────────────┘

Core Services

1. Agent Service (Service/AgentService.py)

Responsibilities:
  • Create and manage AI agents
  • Configure system prompts and behavior
  • Handle agent lifecycle (create, update, delete)
  • Manage conversation context
Key Classes:
class AgentService:
    def create_agent(self, config: AgentConfig) -> Agent
    def get_agent(self, agent_id: str) -> Agent
    def update_agent(self, agent_id: str, config: AgentConfig) -> Agent
    def delete_agent(self, agent_id: str) -> None
Features:
  • Multi-provider AI support (Gemini, OpenAI, Hume)
  • Custom system prompts
  • Tool/function calling
  • Context management

2. Call Service (Service/CallService.py)

Responsibilities:
  • Manage voice call lifecycle
  • Handle call routing and connection
  • Process call recordings
  • Track call metadata
Key Methods:
def initiate_call(self, call_request: OutboundCallRequest) -> Call
def handle_incoming_call(self, twilio_data: dict) -> TwiMLResponse
def process_call_stream(self, websocket: WebSocket, call_id: str) -> None
def end_call(self, call_id: str) -> None
Call Flow:
  1. Create call request
  2. Initialize call connection
  3. Stream audio to AI agent
  4. Process agent responses
  5. Return audio to caller
  6. Log call metadata

3. Organization Service (Service/OrganizationService.py)

Responsibilities:
  • Manage multi-tenant organizations
  • Handle credentials per organization
  • Manage API keys and authentication
  • Organization billing and usage
Features:
  • Organization creation
  • Member management
  • API key generation
  • Usage tracking

4. Scheduler Service (scheduler/scheduler.py)

Responsibilities:
  • Schedule calls at specific times
  • Manage recurring call patterns
  • Trigger batch operations
  • Handle call execution
Implementation:
  • Uses APScheduler for scheduling
  • Stores schedule in Supabase
  • Supports cron expressions

5. Batch Call Service (Service/BatchCallService.py)

Responsibilities:
  • Process multiple calls in bulk
  • Handle CSV upload and parsing
  • Manage batch execution progress
  • Generate batch reports
Workflow:
  1. Upload call list (CSV/JSON)
  2. Validate phone numbers
  3. Schedule calls
  4. Monitor progress
  5. Generate results report

AI Bridges

AI bridges abstract the different AI provider APIs into a common interface.

Supported Providers

ProviderModuleFeatures
Geminibridges/gemini_bridge_base.pyReal-time streaming, multi-modal
OpenAIbridges/open_ai.pyGPT-4, function calling, vision
Hume AIbridges/hume_bridge.pyEmotional intelligence, empathy

Bridge Interface

class AIBridge:
    async def process_message(self, message: str, context: ConversationContext) -> str
    async def stream_response(self, message: str) -> AsyncIterator[str]
    def set_system_prompt(self, prompt: str) -> None
    async def execute_tool(self, tool_name: str, args: dict) -> Any

Communication Layer

Telephony Providers

Twilio:
  • Incoming call handling via webhooks
  • TwiML response generation
  • Real-time media streaming
  • Call recording
Plivo:
  • Similar to Twilio
  • Alternative provider for redundancy
  • REST API for outbound calls

LiveKit Integration

For real-time communication:
  • WebRTC media streaming
  • Low-latency audio processing
  • Video support
  • Room-based communication

MCP Integration Layer

Model Context Protocol services enable integration with external platforms:

Service Architecture

MCP Client (in AgentService)

     └─► MCP Server Process

              ├─► Tool: send_email
              ├─► Tool: read_email
              ├─► Tool: send_message
              └─► Tool: read_message

Supported Integrations

ServicePortTools
Telegram8100send_message, read_message
Gmail8101send_email, read_email
WhatsApp8102send_message, read_message

Data Layer

Supabase Database Schema

-- Organizations
CREATE TABLE organizations (
  id UUID PRIMARY KEY,
  name TEXT NOT NULL,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

-- Agents
CREATE TABLE agents (
  id UUID PRIMARY KEY,
  organization_id UUID REFERENCES organizations,
  name TEXT NOT NULL,
  ai_provider TEXT,
  system_prompt TEXT,
  created_at TIMESTAMP
);

-- Calls
CREATE TABLE calls (
  id UUID PRIMARY KEY,
  agent_id UUID REFERENCES agents,
  from_phone TEXT,
  to_phone TEXT,
  status TEXT,
  duration INTEGER,
  recording_url TEXT,
  created_at TIMESTAMP
);

-- Scheduled Calls
CREATE TABLE scheduled_calls (
  id UUID PRIMARY KEY,
  agent_id UUID REFERENCES agents,
  phone_number TEXT,
  scheduled_time TIMESTAMP,
  status TEXT,
  cron_expression TEXT
);

-- Batch Calls
CREATE TABLE batch_calls (
  id UUID PRIMARY KEY,
  organization_id UUID REFERENCES organizations,
  file_url TEXT,
  status TEXT,
  total_calls INTEGER,
  completed_calls INTEGER,
  created_at TIMESTAMP
);

Request Flow Examples

Outbound Call Flow

Outbound Call Flow Diagram

Incoming WebSocket Stream

1. Client establishes WebSocket connection
2. Audio frames received in real-time
3. Frames sent to AI provider
4. Response streamed back to client
5. Text-to-speech conversion
6. Audio sent to phone

Scaling Considerations

Horizontal Scaling

  • Stateless API servers (multiple instances)
  • Load balancer (nginx, AWS ALB)
  • Shared database (Supabase)
  • Message queue for async tasks (optional: Redis/RabbitMQ)

Vertical Scaling

  • Increase server resources (CPU, memory)
  • Database connection pooling
  • Caching layer (Redis)
  • CDN for media files

Security Architecture

Authentication

API Request

    ├─► Header: Authorization: Bearer <token>

    ├─► Verify Token (Supabase JWT)

    └─► Extract User & Organization

Data Protection

  • Encryption at rest (Supabase)
  • TLS/SSL for transport
  • API key rotation
  • Audit logging

Technology Stack

LayerTechnologies
API FrameworkFastAPI, Uvicorn, Pydantic
Real-timeWebSocket, LiveKit
AI/LLMGemini, OpenAI, Hume
TelephonyTwilio, Plivo, Silero VAD
IntegrationsMCP, Telethon, OAuth2
DatabasePostgreSQL (via Supabase)
DeploymentDocker, Docker Compose
Task SchedulingAPScheduler
Async Processingasyncio, aiohttp

Next Steps

AI Agents

Learn how to create and configure AI agents.

Call Management

Understand call routing and handling.

Services

Explore available services.

Integrations

Integrate external platforms.