Speaker Identification & Diarization

[00:00-00:05] Speaker 1 (Customer): "Hi, I'd like to place an order"
[00:05-00:12] Speaker 2 (Agent): "Hello! I'd be happy to help you"
[00:12-00:20] Speaker 1: "Great, I need..."
[00:20-00:35] Speaker 2: "Perfect, let me look that up for you"

Speakers Detected: Up to 10+ simultaneously
Accuracy: High
Cost: ~$0.0075/min (standard)
Latency: Real-time processing
Languages: All languages
Special Features: Speaker labels automatically assigned

Speakers Detected: Identifies multiple speakers
Accuracy: Good
Cost: Included in standard pricing
Latency: Real-time
Languages: All supported languages
Special Features: Built into standard STT

Speakers Detected: Unlimited
Accuracy: Very high
Cost: Free (self-hosted)
Latency: Depends on infrastructure
Languages: Language-independent
Special Features: Customizable, offline capable

Benefits:
✓ Verify agent followed procedures
✓ Identify who said what in disputes
✓ Measure talk time (agent vs customer)
✓ Training and coaching improvements

Benefits:
✓ Track transfer between agents
✓ Measure hold time
✓ Identify each speaker's role
✓ Billing and routing information

Benefits:
✓ Clear transcript with speaker labels
✓ Legal evidence of who said what
✓ Audit trail for compliance
✓ Risk mitigation in disputes

Benefits:
✓ Agent talk time vs listening time
✓ Customer engagement metrics
✓ Silence/hold time detection
✓ Conversation flow analysis

pip install pyannote.audio
pip install torch

from pyannote.audio import Model
model = Model.from_pretrained("pyannote/speaker-diarization-3.0")

Best For: Unknown number of speakers
Accuracy: High
Performance: Slightly slower
Recommended: Most cases

If you know there are always 2 speakers (agent + customer):
- Set to 2 speakers
- Faster processing
- More accurate identification

Common Scenarios:
- 2: Standard agent + customer call
- 3: Agent + supervisor + customer
- 4+: Conference calls or multi-agent calls

Default: 0.5 seconds
Increase to: 1-2 seconds (reduces fragmentation)
Use: When speakers change rapidly

Automatic (Default): System chooses best method
Spectral Clustering: Better for many speakers
Agglomerative: More stable

[00:00-00:05] SPEAKER_00: "Hello, this is customer service"
[00:05-00:10] SPEAKER_01: "Hi! I have a question about my order"
[00:10-00:15] SPEAKER_00: "I'd be happy to help!"
[00:15-00:25] SPEAKER_01: "Great, I ordered on Tuesday and..."

{
  "transcript": [
    {
      "speaker": "SPEAKER_00",
      "start": 0.0,
      "end": 5.0,
      "text": "Hello, this is customer service"
    },
    {
      "speaker": "SPEAKER_01",
      "start": 5.0,
      "end": 10.0,
      "text": "Hi! I have a question about my order"
    }
  ]
}

SPEAKER_00: "Agent"
SPEAKER_01: "Customer"

SPEAKER_00: "John (Agent)"
SPEAKER_01: "Sarah (Customer)"

Recommended:
- 16kHz or higher sample rate
- Mono or stereo
- Minimal compression
- Clear voice capture

If you know: "Always 2 speakers"
→ Set to 2 (faster and more accurate)

If you're unsure:
→ Use auto-detect

System handles overlaps but:
- Overlapping = harder to distinguish
- Encourage natural turn-taking
- Can affect accuracy

Speakers with clear separation = better accuracy
Continuous talking = harder to distinguish
Multiple speakers = higher chance of errors

Standard: ~$0.0075/minute
With Diarization: Additional $0.005/minute
Total: ~$0.0125/minute

Monthly Cost (1000 calls, 2 min avg):
1000 × 2 × $0.0125 = $25/month

Standard: ~$0.0043/minute
Diarization: Included
Total: ~$0.0043/minute

Monthly Cost (1000 calls, 2 min avg):
1000 × 2 × $0.0043 = $8.60/month

1. Agent + Customer call
2. Speaker count: 2
3. Labels:
   - SPEAKER_00: "Agent (Sales)"
   - SPEAKER_01: "Customer"

[00:00] AGENT: "Hello! Thanks for calling ABC Company"
[00:05] CUSTOMER: "Hi, I saw your ad and wanted to know more"
[00:10] AGENT: "Great! Let me tell you about our products..."

3-way call:
- Initial agent
- Supervisor
- Customer

Speaker count: 3
Labels:
- SPEAKER_00: Agent 1
- SPEAKER_01: Agent 2 (Supervisor)
- SPEAKER_02: Customer

Multiple participants: 4-6
Auto-detect speakers
Let system identify each speaker

Labels:
- SPEAKER_00: Participant 1
- SPEAKER_01: Participant 2
- SPEAKER_02: Participant 3
- etc.

Default to auto-detect
System handles most cases well
Switch to specified count only when needed

Not every call needs diarization
Only enable for:
- Quality assurance calls
- Escalations
- Training calls
- Compliance-critical calls

Saves on cost for standard calls

Weekly Review:
- Sample 5-10 calls with diarization
- Verify speaker identification
- Check for false positives
- Adjust settings if needed

Ensure:
- Good microphone quality
- Minimal background noise
- Proper volume levels
- Good internet connection

When processing transcripts:
- Label speakers by role (Agent, Supervisor, Customer)
- Add timestamps for easy reference
- Create searchable indexes
- Archive for compliance

Benefits:
- Trained on your specific voices
- Better accuracy for known speakers
- Speaker identification by name
- Customized thresholds

Cost: Enterprise pricing
Setup: Contact provider

System learns over time:
- More accurate as it processes your calls
- Adapts to your speakers
- Improves baseline accuracy
- Requires ongoing feedback

1. Diarization Error Rate (DER)
   - Lower is better (< 5% is excellent)
   
2. Speaker Count Accuracy
   - Correct identification of number of speakers
   
3. Speaker Labeling Accuracy
   - Correct assignment of labels to speakers
   
4. Processing Latency
   - Time to generate diarization