Utterance
A single unit of communication in a conversation, either a user's input or the AI's response in a dialogue exchange.
Overview
In conversational AI, an utterance is one turn in the dialogue: what the user says or what the AI responds. Understanding utterances is fundamental to designing tests, analyzing conversations, and evaluating multi-turn interactions.
Utterances in LLM Testing
Single-Turn Context: In single-turn tests, you typically have:
- User utterance: The test prompt or input
- AI utterance: The system's response
Multi-Turn Context: Multi-turn tests involve a sequence of utterances.
Utterance Characteristics
Length:
- Short utterances: "Hello", "Thanks", "Yes"
- Medium utterances: "What's your return policy?"
- Long utterances: Multi-sentence requests or explanations
Complexity:
- Simple: Single request or question
- Compound: Multiple requests in one utterance
- Contextual: Relies on previous utterances
Testing Utterance Handling
Utterance Quality Metrics: Evaluate appropriateness of utterance length and clarity.
Testing Utterance Understanding: Verify the AI correctly interprets user utterances.
Multi-Turn Utterance Patterns
Turn-by-Turn Analysis: Examine each utterance in context.
Utterance Coherence: Ensure responses connect logically to previous utterances.
Common Utterance Issues
Too Short/Terse: Overly brief responses that lack helpful information.
Too Long/Verbose: Responses that include unnecessary detail.
Missing Context: Utterances that don't acknowledge previous conversation.
Best Practices
Testing Coverage:
- Vary utterance length: Test short, medium, and long inputs
- Test utterance types: Questions, statements, requests, etc.
- Include realistic patterns: How users actually communicate
- Test follow-ups: Subsequent utterances that reference earlier ones
Evaluation Criteria:
- Context matters: Evaluate utterances in conversation flow
- Appropriate length: Not too short, not too long
- Natural language: Should sound conversational
- Coherent: Makes sense given previous utterances
AI Response Quality:
- Match user style: Adapt to user's communication patterns
- Provide complete information: Don't require follow-ups for basics
- Be concise: Respect user's time
- Acknowledge context: Reference previous utterances when relevant