Skip to main content

What is an AI Agent?

An AI Agent is an autonomous system that:
  • Listens to caller input
  • Processes the information intelligently
  • Generates contextual responses
  • Maintains conversation flow
  • Can execute tools and integrations

Creating an Agent

Via API

curl -X POST http://localhost:8010/api/agents \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "name": "Customer Support Agent",
    "initial_message": "Hello! How can I help you today?",
    "ai_provider": "gemini",
    "system_prompt": "You are a helpful customer support representative...",
    "voice_provider": "twilio",
    "temperature": 0.7,
    "max_tokens": 1024
  }'

Via Python

from Service.AgentService import AgentService
from models.models import AgentConfig

service = AgentService()

config = AgentConfig(
    name="Customer Support Agent",
    initial_message="Hello! How can I help you today?",
    ai_provider="gemini",
    system_prompt="You are a helpful customer support representative...",
    temperature=0.7,
    max_tokens=1024
)

agent = service.create_agent(config)
print(f"Agent created: {agent.id}")

Agent Configuration

Basic Properties

PropertyTypeDescription
namestringAgent display name
initial_messagestringFirst message to caller
ai_providerstringgemini, openai, or hume
system_promptstringInstructions for agent behavior
voice_providerstringtwilio, plivo, or livekit
temperaturefloat0.0-1.0, controls creativity
max_tokensintegerMaximum response length

System Prompts

The system prompt defines the agent’s personality and behavior:
You are a helpful customer support representative for TechCorp.

Your responsibilities:
1. Answer customer questions about our products
2. Help with troubleshooting issues
3. Process refund requests
4. Maintain a professional and friendly tone

Guidelines:
- Always be polite and patient
- Ask clarifying questions if needed
- Apologize for any inconvenience
- Offer solutions or escalate if necessary

Available tools:
- Look up order history
- Process refunds
- Schedule callbacks
- Create support tickets

AI Provider Selection

Gemini

Best for:
  • Real-time streaming
  • Cost-effective solutions
  • Multi-modal understanding
agent_config = {
    "ai_provider": "gemini",
    "model": "gemini-2.0-flash-exp",
    "temperature": 0.7,
}

OpenAI

Best for:
  • Advanced reasoning
  • Function calling
  • Fine-tuned models
agent_config = {
    "ai_provider": "openai",
    "model": "gpt-4-turbo",
    "temperature": 0.7,
}

Hume AI

Best for:
  • Emotional intelligence
  • Empathy detection
  • Sentiment analysis
agent_config = {
    "ai_provider": "hume",
    "model": "hume-standard",
    "temperature": 0.7,
    "enable_emotion_recognition": True,
}

Agent Behavior

Conversation Flow

1. Caller connects
2. Agent plays initial_message
3. Agent listens to caller input
4. Input processed by AI
5. Agent generates response
6. Response converted to speech
7. Audio played to caller
8. Loop until call ends

Context Management

Agents maintain conversation context:
context = ConversationContext(
    agent_id=agent_id,
    call_id=call_id,
    conversation_history=[
        {"role": "user", "content": "Hello"},
        {"role": "assistant", "content": "Hi there!"},
    ],
    metadata={
        "customer_id": "123",
        "order_id": "456",
    }
)

Tool Calling

Agents can execute tools:
# Define available tools
tools = [
    {
        "name": "lookup_order",
        "description": "Look up customer order information",
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string"}
            }
        }
    },
    {
        "name": "process_refund",
        "description": "Process a refund for an order",
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string"},
                "reason": {"type": "string"}
            }
        }
    }
]

# Agent calls tools as needed
result = agent.execute_tool("lookup_order", {"order_id": "ORD-123"})

Managing Agents

List Agents

curl -X GET http://localhost:8010/api/agents \
  -H "Authorization: Bearer YOUR_TOKEN"

Get Agent Details

curl -X GET http://localhost:8010/api/agents/{agent_id} \
  -H "Authorization: Bearer YOUR_TOKEN"

Update Agent

curl -X PUT http://localhost:8010/api/agents/{agent_id} \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "system_prompt": "Updated prompt...",
    "temperature": 0.5
  }'

Delete Agent

curl -X DELETE http://localhost:8010/api/agents/{agent_id} \
  -H "Authorization: Bearer YOUR_TOKEN"

Advanced Agent Features

Multi-Language Support

agent_config = {
    "language": "es",  # Spanish
    "timezone": "America/Mexico_City",
}

Custom Voice

agent_config = {
    "voice_provider": "elevenlabs",
    "voice_id": "EXAVITQu4vr4xnSDxMaL",
    "voice_name": "Sarah",
}

Emotion Recognition (Hume)

agent_config = {
    "ai_provider": "hume",
    "enable_emotion_recognition": True,
    "emotion_response_mapping": {
        "frustration": "escalate_to_human",
        "happiness": "provide_upsell",
    }
}

Knowledge Base Integration

agent_config = {
    "knowledge_base_id": "kb-123",
    "knowledge_base_instructions": """
    Use the knowledge base to answer questions about:
    - Product features
    - Pricing information
    - Company policies
    """,
}

Agent Templates

Customer Support Agent

CUSTOMER_SUPPORT_PROMPT = """
You are a friendly and professional customer support agent.

Your goals:
1. Help customers with product questions
2. Troubleshoot technical issues
3. Process orders and refunds
4. Escalate complex issues to a human agent

Always:
- Be empathetic and understanding
- Listen carefully to customer concerns
- Provide clear solutions
- Ask permission before actions
"""

Sales Agent

SALES_PROMPT = """
You are an enthusiastic sales representative.

Your goals:
1. Understand customer needs
2. Present relevant products
3. Overcome objections
4. Close sales

Techniques:
- Ask discovery questions
- Highlight benefits, not features
- Use social proof
- Create urgency appropriately
"""

Technical Support Agent

TECHNICAL_SUPPORT_PROMPT = """
You are a knowledgeable technical support specialist.

Your approach:
1. Understand the technical problem
2. Ask diagnostic questions
3. Provide step-by-step solutions
4. Verify the issue is resolved

Remember:
- Be patient with non-technical users
- Avoid jargon when possible
- Escalate if beyond your scope
- Document the issue
"""

Appointment Scheduling Agent

SCHEDULING_PROMPT = """
You are a professional scheduling assistant.

Your responsibilities:
1. Understand customer needs
2. Check availability
3. Confirm appointment details
4. Send confirmation

Available tools:
- check_availability
- create_appointment
- send_confirmation
"""

Agent Performance

Monitoring

from utils.logger import logger

# Log key metrics
logger.info(f"Agent {agent_id} - Response time: {response_time}ms")
logger.info(f"Agent {agent_id} - Token usage: {tokens_used}")
logger.info(f"Agent {agent_id} - Call duration: {duration}s")

Optimization Tips

  1. Shorter prompts: Reduce tokens, faster responses
  2. Clear instructions: Better understood by AI
  3. Few-shot examples: Improve output quality
  4. Temperature tuning: 0.3-0.7 is usually best
  5. Model selection: Larger models = better quality but slower

Cost Optimization

# Use streaming for real-time feedback
agent_config = {
    "use_streaming": True,  # Reduces token count
    "response_cache": True,  # Cache repeated questions
}

Testing Agents

Local Testing

# Test agent response
from Service.AgentService import AgentService

service = AgentService()
agent = service.get_agent("agent-id")

response = agent.process_message(
    "What are your business hours?"
)
print(response)

Test Call

# Make a test call to the agent
curl -X POST http://localhost:8010/api/calls \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "agent-id",
    "to_phone": "+1234567890",
    "from_phone": "+0987654321",
    "test_mode": true
  }'

Troubleshooting

Agent Not Responding

# Check agent status
agent = service.get_agent("agent-id")
if not agent.is_active:
    print("Agent is disabled")

# Check AI provider connectivity
try:
    response = agent.test_connection()
    print(f"Connection OK: {response}")
except Exception as e:
    print(f"Connection failed: {e}")

Poor Response Quality

  • Review and improve system prompt
  • Increase temperature for more creative responses
  • Use a larger AI model
  • Add few-shot examples to the prompt
  • Enable emotion recognition for better empathy

Token Limit Issues

agent_config = {
    "max_tokens": 256,  # Reduce response length
    "summarize_context": True,  # Compress history
}

Next Steps