Architecture Overview

System Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                      External APIs                          │
│  (Gemini, OpenAI, Hume, Twilio, Plivo, Gmail, etc.)         │
└────────────────────┬────────────────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────────────────┐
│                   API Gateway Layer                         │
│         FastAPI (http://localhost:8010)                     │
│                                                             │
│  ├── REST API Endpoints (/api/agents, /api/calls, etc.)     │
│  ├── WebSocket Streaming (/ws/stream)                       │
│  └── Webhook Receivers (/webhook/twilio, etc.)              │
└─────────┬──────────┬──────────┬──────────┬─────────────────-┘
          │          │          │          │
    ┌─────▼──┐  ┌────▼────┐ ┌──▼────┐ ┌──▼────┐
    │ Agent  │  │  Call   │ │Org &  │ │Batch  │
    │Service │  │ Service │ │Auth   │ │Calls  │
    └─────┬──┘  └────┬────┘ └──┬────┘ └──┬────┘
          │          │         │         │
    ┌─────▼──────────▼─────────▼─────────▼──────┐
    │                                           │
    │    AI Bridges Layer                       │
    │  ┌──────────┐ ┌──────────┐ ┌──────────┐   │
    │  │  Gemini  │ │  OpenAI  │ │   Hume   │   │
    │  └──────────┘ └──────────┘ └──────────┘   │
    │                                           │
    └─────────────┬─────────────────────────────┘
                  │
    ┌─────────────▼──────────────────┐
    │   Communication Layer          │
    │  ┌──────────┐ ┌──────────┐     │
    │  │  Twilio  │ │  Plivo   │     │
    │  └──────────┘ └──────────┘     │
    │  ┌──────────────────────────┐  │
    │  │  LiveKit (RTC Media)     │  │
    │  └──────────────────────────┘  │
    └─────────────┬──────────────────┘
                  │
    ┌─────────────▼──────────────────┐
    │  Integration Layer (MCP)       │
    │  ┌──────────┐ ┌──────────┐     │
    │  │ Telegram │ │  Gmail   │     │
    │  └──────────┘ └──────────┘     │
    │  ┌──────────┐ ┌──────────┐     │
    │  │WhatsApp  │ │ Shopify  │     │
    │  └──────────┘ └──────────┘     │
    └─────────────┬──────────────────┘
                  │
    ┌─────────────▼──────────────────┐
    │  Data Persistence Layer        │
    │  ┌──────────────────────────┐  │
    │  │    Supabase (PostgreSQL) │  │
    │  │  (Organizations, Agents, │  │
    │  │   Calls, Schedules)      │  │
    │  └──────────────────────────┘  │
    └────────────────────────────────┘

Core Services

1. Agent Service (`Service/AgentService.py`)

Responsibilities:

Create and manage AI agents
Configure system prompts and behavior
Handle agent lifecycle (create, update, delete)
Manage conversation context

Key Classes:

class AgentService:
    def create_agent(self, config: AgentConfig) -> Agent
    def get_agent(self, agent_id: str) -> Agent
    def update_agent(self, agent_id: str, config: AgentConfig) -> Agent
    def delete_agent(self, agent_id: str) -> None

Features:

Multi-provider AI support (Gemini, OpenAI, Hume)
Custom system prompts
Tool/function calling
Context management

2. Call Service (`Service/CallService.py`)

Responsibilities:

Manage voice call lifecycle
Handle call routing and connection
Process call recordings
Track call metadata

Key Methods:

def initiate_call(self, call_request: OutboundCallRequest) -> Call
def handle_incoming_call(self, twilio_data: dict) -> TwiMLResponse
def process_call_stream(self, websocket: WebSocket, call_id: str) -> None
def end_call(self, call_id: str) -> None

Call Flow:

Create call request
Initialize call connection
Stream audio to AI agent
Process agent responses
Return audio to caller
Log call metadata

3. Organization Service (`Service/OrganizationService.py`)

Responsibilities:

Manage multi-tenant organizations
Handle credentials per organization
Manage API keys and authentication
Organization billing and usage

Features:

Organization creation
Member management
API key generation
Usage tracking

4. Scheduler Service (`scheduler/scheduler.py`)

Responsibilities:

Schedule calls at specific times
Manage recurring call patterns
Trigger batch operations
Handle call execution

Implementation:

Uses APScheduler for scheduling
Stores schedule in Supabase
Supports cron expressions

5. Batch Call Service (`Service/BatchCallService.py`)

Responsibilities:

Process multiple calls in bulk
Handle CSV upload and parsing
Manage batch execution progress
Generate batch reports

Workflow:

Upload call list (CSV/JSON)
Validate phone numbers
Schedule calls
Monitor progress
Generate results report

AI Bridges

AI bridges abstract the different AI provider APIs into a common interface.

Supported Providers

Provider	Module	Features
Gemini	`bridges/gemini_bridge_base.py`	Real-time streaming, multi-modal
OpenAI	`bridges/open_ai.py`	GPT-4, function calling, vision
Hume AI	`bridges/hume_bridge.py`	Emotional intelligence, empathy

Bridge Interface

class AIBridge:
    async def process_message(self, message: str, context: ConversationContext) -> str
    async def stream_response(self, message: str) -> AsyncIterator[str]
    def set_system_prompt(self, prompt: str) -> None
    async def execute_tool(self, tool_name: str, args: dict) -> Any

Communication Layer

Telephony Providers

Twilio:

Incoming call handling via webhooks
TwiML response generation
Real-time media streaming
Call recording

Plivo:

Similar to Twilio
Alternative provider for redundancy
REST API for outbound calls

LiveKit Integration

For real-time communication:

WebRTC media streaming
Low-latency audio processing
Video support
Room-based communication

MCP Integration Layer

Model Context Protocol services enable integration with external platforms:

Service Architecture

MCP Client (in AgentService)
     │
     └─► MCP Server Process
              │
              ├─► Tool: send_email
              ├─► Tool: read_email
              ├─► Tool: send_message
              └─► Tool: read_message

Supported Integrations

Service	Port	Tools
Telegram	8100	send_message, read_message
Gmail	8101	send_email, read_email
WhatsApp	8102	send_message, read_message

Data Layer

Supabase Database Schema

-- Organizations
CREATE TABLE organizations (
  id UUID PRIMARY KEY,
  name TEXT NOT NULL,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

-- Agents
CREATE TABLE agents (
  id UUID PRIMARY KEY,
  organization_id UUID REFERENCES organizations,
  name TEXT NOT NULL,
  ai_provider TEXT,
  system_prompt TEXT,
  created_at TIMESTAMP
);

-- Calls
CREATE TABLE calls (
  id UUID PRIMARY KEY,
  agent_id UUID REFERENCES agents,
  from_phone TEXT,
  to_phone TEXT,
  status TEXT,
  duration INTEGER,
  recording_url TEXT,
  created_at TIMESTAMP
);

-- Scheduled Calls
CREATE TABLE scheduled_calls (
  id UUID PRIMARY KEY,
  agent_id UUID REFERENCES agents,
  phone_number TEXT,
  scheduled_time TIMESTAMP,
  status TEXT,
  cron_expression TEXT
);

-- Batch Calls
CREATE TABLE batch_calls (
  id UUID PRIMARY KEY,
  organization_id UUID REFERENCES organizations,
  file_url TEXT,
  status TEXT,
  total_calls INTEGER,
  completed_calls INTEGER,
  created_at TIMESTAMP
);

Request Flow Examples

Outbound Call Flow

Incoming WebSocket Stream

Client establishes WebSocket connection
Audio frames received in real-time
Frames sent to AI provider
Response streamed back to client
Text-to-speech conversion
Audio sent to phone

Scaling Considerations

Horizontal Scaling

Stateless API servers (multiple instances)
Load balancer (nginx, AWS ALB)
Shared database (Supabase)
Message queue for async tasks (optional: Redis/RabbitMQ)

Vertical Scaling

Increase server resources (CPU, memory)
Database connection pooling
Caching layer (Redis)
CDN for media files

Security Architecture

Authentication

API Request
    │
    ├─► Header: Authorization: Bearer <token>
    │
    ├─► Verify Token (Supabase JWT)
    │
    └─► Extract User & Organization

Data Protection

Encryption at rest (Supabase)
TLS/SSL for transport
API key rotation
Audit logging

Technology Stack

Layer	Technologies
API Framework	FastAPI, Uvicorn, Pydantic
Real-time	WebSocket, LiveKit
AI/LLM	Gemini, OpenAI, Hume
Telephony	Twilio, Plivo, Silero VAD
Integrations	MCP, Telethon, OAuth2
Database	PostgreSQL (via Supabase)
Deployment	Docker, Docker Compose
Task Scheduling	APScheduler
Async Processing	asyncio, aiohttp

Next Steps

AI Agents

Learn how to create and configure AI agents.

Call Management

Understand call routing and handling.

Services

Explore available services.

Integrations

Integrate external platforms.

​System Architecture Diagram

​Core Services

​1. Agent Service (Service/AgentService.py)

​2. Call Service (Service/CallService.py)

​3. Organization Service (Service/OrganizationService.py)

​4. Scheduler Service (scheduler/scheduler.py)

​5. Batch Call Service (Service/BatchCallService.py)

​AI Bridges

​Supported Providers

​Bridge Interface

​Communication Layer

​Telephony Providers

​LiveKit Integration

​MCP Integration Layer

​Service Architecture

​Supported Integrations

​Data Layer

​Supabase Database Schema

​Request Flow Examples

​Outbound Call Flow

​Incoming WebSocket Stream

​Scaling Considerations

​Horizontal Scaling

​Vertical Scaling

​Security Architecture

​Authentication

​Data Protection

​Technology Stack

​Next Steps

AI Agents

Call Management

Services

Integrations

System Architecture Diagram

Core Services

1. Agent Service (`Service/AgentService.py`)

2. Call Service (`Service/CallService.py`)

3. Organization Service (`Service/OrganizationService.py`)

4. Scheduler Service (`scheduler/scheduler.py`)

5. Batch Call Service (`Service/BatchCallService.py`)

AI Bridges

Supported Providers

Bridge Interface

Communication Layer

Telephony Providers

LiveKit Integration

MCP Integration Layer

Service Architecture

Supported Integrations

Data Layer

Supabase Database Schema

Request Flow Examples

Outbound Call Flow

Incoming WebSocket Stream

Scaling Considerations

Horizontal Scaling

Vertical Scaling

Security Architecture

Authentication

Data Protection

Technology Stack

Next Steps