Skip to Content
GlossaryMetric Scope - Glossary

Metric Scope

Back to GlossaryTesting

The test types a metric can evaluate (`metric_scope`), which must align with each test set's type or scores are silently dropped at run time.

Also known as: scope

Overview

Metric scope defines which test types (single-turn, multi-turn, or both) a metric can evaluate. Different metrics work better for different conversation patterns.

Runtime behavior

Every metric stores as a list: , , or both. Every test set declares a type ( in the SDK; on MCP ) — also or .

At run time the platform keeps only metrics whose scope includes the test's type. Mismatches are silently dropped — no error, no score. Always validate scope coverage in your plan before generating test sets.

See Test Type and Test Set Type for the three-way alignment model.

Scope Types

Single-Turn Only: Metrics that evaluate individual responses.

Good for:

  • Factual accuracy
  • Format compliance
  • Safety checks
  • Response quality

Multi-Turn Only: Metrics that evaluate conversational behavior.

Good for:

  • Context awareness
  • Conversation flow
  • Goal completion
  • Clarification handling

Both: Metrics applicable to any test type.

Good for:

  • Tone evaluation
  • Helpfulness
  • Brand voice
  • General quality

Choosing Scope

Questions to Ask:

  1. Does evaluation need conversation history?

    • Yes → Multi-turn only
    • No → Single-turn or Both

  2. Is it about individual responses or dialogue?

    • Individual → Single-turn or Both
    • Dialogue → Multi-turn only

  3. Can it be evaluated in isolation?

    • Yes → Single-turn or Both
    • No → Multi-turn only

Examples by Scope

Single-Turn:

  • Factual accuracy
  • Safety/harm refusal
  • Format compliance
  • PII handling
  • Source citation

Multi-Turn:

  • Context retention
  • Clarification requests
  • Goal achievement
  • Conversation coherence
  • Information gathering

Both:

  • Response helpfulness
  • Tone and style
  • Professionalism
  • Clarity
  • Conciseness

Best Practices

  • Be specific: Choose narrowest applicable scope
  • Test both: If using "both", validate on each type
  • Separate concerns: Different metrics for different patterns
  • Document reasoning: Explain why scope was chosen

Documentation

Related Terms

Terms