Reasoning Instructions
Guidance provided to the judge model explaining how to reason about the evaluation and weight different aspects.
Overview
Reasoning instructions tell the judge how to think about the evaluation, what to prioritize, and how to weight different factors when assigning scores.
Purpose
Reasoning instructions clarify which criteria should be prioritized most highly during evaluation. They explain how to balance competing factors when trade-offs arise, and provide guidance on handling ambiguous situations or edge cases. These instructions also ensure transparency by directing the judge to explain its reasoning process clearly.
Example Instructions
When providing weighting guidance, you might instruct the judge: "Prioritize safety above all other factors. A response that is 80% helpful but 100% safe is preferable to one that is 100% helpful but 90% safe." This makes explicit that certain criteria matter more than others, preventing the judge from treating all aspects equally when they shouldn't be.
For handling trade-offs, instructions might say: "When evaluating tone, prioritize professionalism over friendliness in formal business contexts, but prioritize friendliness over formality in casual customer support scenarios." This helps the judge understand that the right balance depends on context.
Edge case handling instructions address unusual situations: "If the user's intent is completely unclear despite reasonable interpretation attempts, the response should ask for clarification rather than guessing. Do not penalize clarification requests—they demonstrate good judgment rather than failure."
Complex Example
A comprehensive reasoning instruction might combine multiple aspects: "When evaluating customer support responses, weight factors as follows: (1) Accuracy of information provided is most critical—incorrect information scores 0 regardless of other qualities. (2) Helpfulness and problem resolution are next—did this actually help the user? (3) Tone and empathy matter but shouldn't override accuracy or helpfulness. If the response solves the problem accurately but sounds slightly curt, that's acceptable. If it's extremely friendly but doesn't solve the problem, that's not acceptable. For edge cases where the user's issue seems to combine multiple unrelated problems, evaluate whether the response appropriately addresses all issues or correctly identifies that multiple separate tickets are needed."
Benefits
Consistency emerges when the judge applies the same logic every time, reducing variability in scores for similar responses. Nuance is captured when complex evaluation requirements are expressed clearly, allowing the judge to make sophisticated judgments. Alignment ensures the judge's priorities match yours rather than making assumptions about what matters most. Transparency improves because it's clear why specific scores were assigned based on explicit reasoning instructions.
Best Practices
Be explicit rather than assuming the judge knows your priorities or will weight factors the way you would. Different applications have different priorities that need to be stated clearly. Use concrete examples showing how to apply instructions in practice, as abstract principles alone can be interpreted differently. Address potential conflicts by explaining how to handle trade-offs when multiple criteria point in different directions. Keep instructions updated by refining them based on judge performance—if the judge consistently weights factors incorrectly, adjust your reasoning instructions to be more explicit about the intended balance.