Skip to main content

System Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                      External APIs                          │
│  (Gemini, OpenAI, Hume, Twilio, Plivo, Gmail, etc.)         │
└────────────────────┬────────────────────────────────────────┘

┌────────────────────▼────────────────────────────────────────┐
│                   API Gateway Layer                         │
│         FastAPI (http://localhost:8010)                     │
│                                                             │
│  ├── REST API Endpoints (/api/agents, /api/calls, etc.)     │
│  ├── WebSocket Streaming (/ws/stream)                       │
│  └── Webhook Receivers (/webhook/twilio, etc.)              │
└─────────┬──────────┬──────────┬──────────┬─────────────────-┘
          │          │          │          │
    ┌─────▼──┐  ┌────▼────┐ ┌──▼────┐ ┌──▼────┐
    │ Agent  │  │  Call   │ │Org &  │ │Batch  │
    │Service │  │ Service │ │Auth   │ │Calls  │
    └─────┬──┘  └────┬────┘ └──┬────┘ └──┬────┘
          │          │         │         │
    ┌─────▼──────────▼─────────▼─────────▼──────┐
    │                                           │
    │    AI Bridges Layer                       │
    │  ┌──────────┐ ┌──────────┐ ┌──────────┐   │
    │  │  Gemini  │ │  OpenAI  │ │   Hume   │   │
    │  └──────────┘ └──────────┘ └──────────┘   │
    │                                           │
    └─────────────┬─────────────────────────────┘

    ┌─────────────▼──────────────────┐
    │   Communication Layer          │
    │  ┌──────────┐ ┌──────────┐     │
    │  │  Twilio  │ │  Plivo   │     │
    │  └──────────┘ └──────────┘     │
    │  ┌──────────────────────────┐  │
    │  │  LiveKit (RTC Media)     │  │
    │  └──────────────────────────┘  │
    └─────────────┬──────────────────┘

    ┌─────────────▼──────────────────┐
    │  Integration Layer (MCP)       │
    │  ┌──────────┐ ┌──────────┐     │
    │  │ Telegram │ │  Gmail   │     │
    │  └──────────┘ └──────────┘     │
    │  ┌──────────┐ ┌──────────┐     │
    │  │WhatsApp  │ │ Shopify  │     │
    │  └──────────┘ └──────────┘     │
    └─────────────┬──────────────────┘

    ┌─────────────▼──────────────────┐
    │  Data Persistence Layer        │
    │  ┌──────────────────────────┐  │
    │  │    Supabase (PostgreSQL) │  │
    │  │  (Organizations, Agents, │  │
    │  │   Calls, Schedules)      │  │
    │  └──────────────────────────┘  │
    └────────────────────────────────┘

Core Services

1. Agent Service (Service/AgentService.py)

Responsibilities:
  • Create and manage AI agents
  • Configure system prompts and behavior
  • Handle agent lifecycle (create, update, delete)
  • Manage conversation context
Key Classes:
class AgentService:
    def create_agent(self, config: AgentConfig) -> Agent
    def get_agent(self, agent_id: str) -> Agent
    def update_agent(self, agent_id: str, config: AgentConfig) -> Agent
    def delete_agent(self, agent_id: str) -> None
Features:
  • Multi-provider AI support (Gemini, OpenAI, Hume)
  • Custom system prompts
  • Tool/function calling
  • Context management

2. Call Service (Service/CallService.py)

Responsibilities:
  • Manage voice call lifecycle
  • Handle call routing and connection
  • Process call recordings
  • Track call metadata
Key Methods:
def initiate_call(self, call_request: OutboundCallRequest) -> Call
def handle_incoming_call(self, twilio_data: dict) -> TwiMLResponse
def process_call_stream(self, websocket: WebSocket, call_id: str) -> None
def end_call(self, call_id: str) -> None
Call Flow:
  1. Create call request
  2. Initialize call connection
  3. Stream audio to AI agent
  4. Process agent responses
  5. Return audio to caller
  6. Log call metadata

3. Organization Service (Service/OrganizationService.py)

Responsibilities:
  • Manage multi-tenant organizations
  • Handle credentials per organization
  • Manage API keys and authentication
  • Organization billing and usage
Features:
  • Organization creation
  • Member management
  • API key generation
  • Usage tracking

4. Scheduler Service (scheduler/scheduler.py)

Responsibilities:
  • Schedule calls at specific times
  • Manage recurring call patterns
  • Trigger batch operations
  • Handle call execution
Implementation:
  • Uses APScheduler for scheduling
  • Stores schedule in Supabase
  • Supports cron expressions

5. Batch Call Service (Service/BatchCallService.py)

Responsibilities:
  • Process multiple calls in bulk
  • Handle CSV upload and parsing
  • Manage batch execution progress
  • Generate batch reports
Workflow:
  1. Upload call list (CSV/JSON)
  2. Validate phone numbers
  3. Schedule calls
  4. Monitor progress
  5. Generate results report

AI Bridges

AI bridges abstract the different AI provider APIs into a common interface.

Supported Providers

ProviderModuleFeatures
Geminibridges/gemini_bridge_base.pyReal-time streaming, multi-modal
OpenAIbridges/open_ai.pyGPT-4, function calling, vision
Hume AIbridges/hume_bridge.pyEmotional intelligence, empathy

Bridge Interface

class AIBridge:
    async def process_message(self, message: str, context: ConversationContext) -> str
    async def stream_response(self, message: str) -> AsyncIterator[str]
    def set_system_prompt(self, prompt: str) -> None
    async def execute_tool(self, tool_name: str, args: dict) -> Any

Communication Layer

Telephony Providers

Twilio:
  • Incoming call handling via webhooks
  • TwiML response generation
  • Real-time media streaming
  • Call recording
Plivo:
  • Similar to Twilio
  • Alternative provider for redundancy
  • REST API for outbound calls

LiveKit Integration

For real-time communication:
  • WebRTC media streaming
  • Low-latency audio processing
  • Video support
  • Room-based communication

MCP Integration Layer

Model Context Protocol services enable integration with external platforms:

Service Architecture

MCP Client (in AgentService)

     └─► MCP Server Process

              ├─► Tool: send_email
              ├─► Tool: read_email
              ├─► Tool: send_message
              └─► Tool: read_message

Supported Integrations

ServicePortTools
Telegram8100send_message, read_message
Gmail8101send_email, read_email
WhatsApp8102send_message, read_message

Data Layer

Supabase Database Schema

-- Organizations
CREATE TABLE organizations (
  id UUID PRIMARY KEY,
  name TEXT NOT NULL,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

-- Agents
CREATE TABLE agents (
  id UUID PRIMARY KEY,
  organization_id UUID REFERENCES organizations,
  name TEXT NOT NULL,
  ai_provider TEXT,
  system_prompt TEXT,
  created_at TIMESTAMP
);

-- Calls
CREATE TABLE calls (
  id UUID PRIMARY KEY,
  agent_id UUID REFERENCES agents,
  from_phone TEXT,
  to_phone TEXT,
  status TEXT,
  duration INTEGER,
  recording_url TEXT,
  created_at TIMESTAMP
);

-- Scheduled Calls
CREATE TABLE scheduled_calls (
  id UUID PRIMARY KEY,
  agent_id UUID REFERENCES agents,
  phone_number TEXT,
  scheduled_time TIMESTAMP,
  status TEXT,
  cron_expression TEXT
);

-- Batch Calls
CREATE TABLE batch_calls (
  id UUID PRIMARY KEY,
  organization_id UUID REFERENCES organizations,
  file_url TEXT,
  status TEXT,
  total_calls INTEGER,
  completed_calls INTEGER,
  created_at TIMESTAMP
);

Request Flow Examples

Outbound Call Flow

Outbound Call Flow Diagram

Incoming WebSocket Stream

1. Client establishes WebSocket connection
2. Audio frames received in real-time
3. Frames sent to AI provider
4. Response streamed back to client
5. Text-to-speech conversion
6. Audio sent to phone

Scaling Considerations

Horizontal Scaling

  • Stateless API servers (multiple instances)
  • Load balancer (nginx, AWS ALB)
  • Shared database (Supabase)
  • Message queue for async tasks (optional: Redis/RabbitMQ)

Vertical Scaling

  • Increase server resources (CPU, memory)
  • Database connection pooling
  • Caching layer (Redis)
  • CDN for media files

Security Architecture

Authentication

API Request

    ├─► Header: Authorization: Bearer <token>

    ├─► Verify Token (Supabase JWT)

    └─► Extract User & Organization

Data Protection

  • Encryption at rest (Supabase)
  • TLS/SSL for transport
  • API key rotation
  • Audit logging

Technology Stack

LayerTechnologies
API FrameworkFastAPI, Uvicorn, Pydantic
Real-timeWebSocket, LiveKit
AI/LLMGemini, OpenAI, Hume
TelephonyTwilio, Plivo, Silero VAD
IntegrationsMCP, Telethon, OAuth2
DatabasePostgreSQL (via Supabase)
DeploymentDocker, Docker Compose
Task SchedulingAPScheduler
Async Processingasyncio, aiohttp

Next Steps