An AI agent should not be judged solely by how fluent its response sounds. In production, what truly matters is whether the agent routed the intent correctly, followed the right instructions, used the proper tools at the right time, and respected its scope and identity boundaries.
Take Playbook 2.0 as an example. The agent can route a message to the correct opening stage and still fail if it does not execute the mandatory institutional greeting through the required tool. That is exactly why Eval Judge exists: to turn every execution into an objective quality signal.
Most teams start trying to improve AI agents by tweaking the prompt. The best teams start by measuring behavior.
This article explains Eval Judge, the evaluation module of Playbook 2.0, and shows why it is essential for agents running in production. It also connects the approach with research on LLM-as-a-Judge, fine-grained evaluation, and continuous learning.
The papers cited here do not describe Eval Judge directly. They provide the technical foundation for the problems the system is designed to solve.
1. The problem with trusting only the response
In a demo, an AI agent looks good when it produces a fluent, coherent, and polite reply. The problem is that production is not a demo.
In production, the agent operates in an environment with business rules, brand identity, stage protocols, mandatory tools, and channel-specific format constraints. Responding well is not enough. The agent must respond correctly — at the right stage, with the right tool, within the right scope.
Consider an agent configured for commercial customer service on WhatsApp. It receives a greeting like "Good evening." The routing works: the system correctly identifies the initial intent and directs the message to the opening stage. The agent replies "How can we help you today?" in a polite and contextual manner. Everything seems fine.
But the execution was incomplete. The playbook required the agent to use a mandatory institutional greeting tool during the opening — and it did not. The response was correct in content but failed in operational protocol.
This case illustrates a central point of this article: an agent can answer correctly and still execute incorrectly.
Without a structured evaluation system, this type of failure goes unnoticed. The team sees a fluent response and assumes everything is fine. The diagnosis only surfaces when a customer complains or when the sales funnel starts dropping for no apparent reason.
That is why, in production, the right question is not "did it answer well?" but "did it do what it was supposed to do, at the right time, in the right way?".
2. Demo versus production: why evaluation is overlooked
Many product teams skip systematic evaluation because the problem is not visible in the short term. During development, the agent is tested with controlled scenarios. The response looks good. The test passes. The team moves on.
In production, however, the agent faces:
- Input variability: real users write in unpredictable ways.
- Playbook changes: business rules change and are not always reflected in agent behavior in time.
- Behavior drift: language model updates can change responses without warning.
- Unstable tools: external APIs fail, change contracts, or become slow.
- Asymmetric channels: what works in a CMS may not work in WhatsApp.
None of these problems are caught by a unit test or by manually inspecting a few responses. They only become visible when every execution is systematically evaluated against the instructions the agent received.
The LLM-as-a-Judge literature confirms that automated evaluation is feasible but requires clear rubrics and bias mitigation. Zheng et al. (2023) show that strong models can approximate human preferences in open-ended evaluations, while also exposing issues such as position bias, verbosity bias, and self-enhancement bias. Gu et al. (2025) propose a reliability framework that combines consistency, bias mitigation, and adaptation to diverse scenarios — exactly what an operational judge needs to consider.
Eval Judge was designed from these premises. It does not try to replace human judgment. It creates a systematic evaluation layer that turns every execution into objective data.
3. What Eval Judge actually evaluates
Eval Judge has a deliberately narrow scope: it evaluates a single agent execution against the compiled instructions the agent received for that interaction.
This is different from evaluating the entire dialogue or user satisfaction. The judge does not ask "was the user satisfied?". It asks "did the agent do what it was supposed to do in this execution?".
This scope restriction is intentional and has important practical implications:
- Operational focus: the evaluation measures adherence to instructions, not perceived quality. This makes the result more objective and actionable.
- Granularity: each execution generates an individual score. If an agent misses a tool call in one step, the judge detects it, regardless of whether the rest of the dialogue is correct.
- Comparability: because the judge uses the same rubric for all executions within the same playbook, scores are comparable across sessions and over time.
The process is asynchronous and fire-and-forget. The judge runs in the background after the response has already been delivered to the user. Evaluation does not block the synchronous response flow. The user gets a fast answer, and the evaluation happens afterward, feeding metrics, logs, and learning opportunities.
In practice, each Eval Judge execution takes about 5.2 seconds of runtime and uses roughly 5,765 input tokens and 557 output tokens — a low operational cost compared to the value of the diagnostic it produces.
4. The four rubric criteria
Eval Judge organizes evaluation into four fixed criteria. Each criterion receives an individual score from 0 to 10, along with an evidence field and a suggested action.
4.1 Intent Analysis
The first criterion checks whether the agent correctly identified the user's intent. It answers: "Did the agent understand what the user wanted to do?"
This criterion does not evaluate response content. It evaluates whether the intent routing was correct. If the user asks about pricing and the agent routes to a registration step, intent analysis fails, even if the individual reply is polite.
In practice, this criterion depends on Playbook 2.0's intent classifier (Step 4 — Intent Analyst) and the routing confidence. Low scores here indicate the agent did not understand what the user wanted — a different problem from not knowing how to answer.
4.2 Execution
The second criterion checks whether the agent followed the playbook instructions for that step. It answers: "Did the agent execute the correct protocol for this intent?"
Execution includes aspects such as following the step's script, respecting substep order, completing mandatory actions, and aligning with the configured persona. A response can be fluent and still fail on execution if it ignores a playbook instruction.
4.3 Tools
The third criterion checks whether the agent used the right tools at the right time. It answers: "Did the agent call the mandatory tools and avoid calling prohibited ones?"
This criterion is especially important in operational workflows. A sales agent may need to check inventory, register a lead in the CRM, or trigger a scheduling webhook. Mandatory tools that are not called represent operational failure, as do tools called in the wrong context.
4.4 Guardrails
The fourth criterion checks whether the agent respected its scope, identity, and safety boundaries. It answers: "Did the agent stay within its defined limits?"
Guardrails include not impersonating a human, not inventing information, not executing actions outside the authorized scope, not sharing sensitive data, and not making promises the system cannot keep. This is the most sensitive criterion and typically has the strictest threshold.
4.5 Composite score and suggested action
Each criterion produces three outputs:
- Score: a numeric value from 0 to 10.
- Evidence: a textual explanation of what was observed, justifying the score.
- Suggested Action: an operational recommendation, such as "review mandatory tool", "update playbook", or "monitor execution".
The overall score is not a simple average. It can be weighted by criterion, and the approval threshold is configurable per playbook. A result below the threshold triggers an alert and, in cases of recurring failure, can initiate the creation of a learning candidate.
5. The real case: when execution fails despite a correct response
The opening example of this article deserves a detailed examination because it reveals the real value of Eval Judge.
Scenario: An agent configured for commercial service receives the message "Good evening." The system identifies the intent as IDENTIFICAR_INTENCAO_INICIAL (IDENTIFY_INITIAL_INTENT) with 0.95 confidence. The synapse found is IDENTIFICAR_INTENCAO_INICIAL_OPENING (IDENTIFY_INITIAL_INTENT_OPENING). The agent replies "How can we help you today?".
Eval Judge assessment:
| Criterion | Score | Analysis |
|---|---|---|
| Intent Analysis | 10 | Intent was correctly identified |
| Execution | 5 | Greeting was verbal but did not go through the mandatory tool |
| Tools | 1 | No tools were called, violating protocol |
| Guardrails | 10 | Response respected scope and identity |
Result: Overall score 6.5 (threshold 7.0). Outcome: partial (warning).
Diagnosis: The agent understood the intent. It got the persona right. It got the tone right. But it failed on operational execution because it ignored the mandatory institutional greeting tool.
Learning candidate generated: Reinforce mandatory use of the greeting tool in two-step openings, with validation before response delivery.
This case shows something most evaluation systems miss: the problem was not "understanding what to do" but "following the playbook protocol."
In a system without Eval Judge, this execution would go unnoticed. The response is polite. The user did not complain. The team has no evidence that anything went wrong. But the lead was not registered by the mandatory tool. The information was lost. The funnel silently stops working.
The value of Eval Judge here is not about generating a score. It is about cleanly separating:
- Correct routing (intent analysis = 10)
- Incomplete execution (execution = 5)
- Missing mandatory tool (tools = 1)
- Guardrails preserved (guardrails = 10)
This signals product maturity. The team stops treating everything as "bad response" and starts diagnosing the exact type of failure.
6. How evaluation feeds continuous improvement
Eval Judge does not exist in isolation. It is part of Playbook 2.0's asynchronous cycle, alongside Context Memory and Learning Writer.
The flow is:
- The agent executes an interaction and produces a response.
- The response is delivered to the user (synchronous flow).
- In parallel, Eval Judge evaluates the execution against the compiled instructions.
- If the score falls below the threshold, the result is logged with a diagnosis.
- If the same failure pattern recurs, the system generates a learning candidate.
- The learning candidate is stored in a specialized learning store.
- In future executions, the Learning Recall (Step 3) retrieves relevant learnings and adjusts agent behavior.
This cycle aligns with research on continuous learning in agents. Reflexion proposes that agents can improve through linguistic feedback stored in memory, without retraining the model after every adjustment (Shinn et al., 2023). The difference in Eval Judge is that the feedback is not self-generated by the agent but produced by a dedicated judge with a fixed, explicit rubric.
Self-Refine explores iterative refinement using self-feedback during runtime (Madaan et al., 2023). In Eval Judge, refinement happens between executions, not during — which is more realistic for production environments where latency is critical.
The separation between execution and evaluation also allows different people to participate in the cycle. The product team can review learning candidates and approve playbook changes. The engineering team can adjust rubrics and thresholds. The operations team can monitor scores over time.
This connects with the reliability framework proposed by Gu et al. (2025), which argues that automated evaluation must be accompanied by governance and human intervention when necessary.
7. Score with operational meaning
An evaluation score is only valuable if it drives a decision. Eval Judge was designed with that premise.
The overall score and per-criterion scores feed different types of decisions:
- Immediate operational: low scores on tools or execution can trigger real-time alerts for the operations team.
- Tactical (learning): recurring failure patterns generate learning candidates that change future agent behavior.
- Strategic (product): the distribution of scores over time shows whether the system is improving, stabilizing, or degrading.
Without a judge, every improvement becomes opinion. The team tweaks the prompt, tests with a few examples, and decides if it got better based on subjective impression. With Eval Judge, the team has an objective basis for comparing versions, testing hypotheses, and deciding whether a change actually improved execution.
This aligns with the custom score rubric approach proposed by PROMETHEUS (Kim et al., 2024). The authors show that explicit, fine-grained rubrics produce evaluations more aligned with human judges than generic preferences do. Eval Judge applies this principle at operational scale, with a fixed four-criterion rubric covering the most critical aspects of conversational agent execution.
The work by Saha et al. (2025) on EvalPlanner also reinforces the importance of separating evaluation planning from evaluation execution. In Eval Judge, this separation appears in the output structure: first the evaluation plan (rubric), then the execution (scores and evidence), then the final verdict (suggested action).
8. Limitations and precautions
Eval Judge is not absolute truth. It is an automated evaluation layer that requires governance and continuous calibration.
Judge bias. Studies on LLM-as-a-Judge identify multiple biases that affect evaluation reliability. Zheng et al. (2023) document position bias, verbosity bias, and self-enhancement bias. Zhu et al. (2025) add knowledge bias and format bias in the context of fine-tuning judges.
Eval Judge mitigates some of these biases by design: because it evaluates a single execution against fixed instructions, there is no response position to influence the result. The explicit rubric reduces the room for arbitrary preferences. Still, the underlying model's bias is not eliminated — only attenuated.
The rubric does not cover everything. Four criteria cannot capture the full complexity of a conversational interaction. Aspects such as emotional tone, cultural appropriateness, and creativity are left out. Eval Judge covers what can be verified against compiled instructions. The rest remains the responsibility of playbook design and human supervision.
Arbitrary threshold. The 7.0 threshold used in the example is configurable but still arbitrary. A score of 6.5 may be acceptable in some contexts and critical in others. Defining thresholds per playbook and per criterion is a product decision that needs regular review.
Evaluation cost. Although the cost per evaluation is low (about 5.2 seconds and 6,300 tokens), it accumulates at scale. For thousands of executions per day, the operational cost of the judge must be monitored and optimized. Techniques such as selective sampling (evaluating only low-confidence executions) can reduce cost without losing coverage.
Learning requires curation. Automatically generated learning candidates need review before changing agent behavior. The LLM-as-a-Judge literature recommends periodic human validation to calibrate and correct drift (Gu et al., 2025). In Eval Judge, this is handled through confidence levels, approval status, and decision logs.
Whitehouse et al. (2025) show that it is possible to train judges with RL (GRPO) to develop systematic evaluation strategies, including dynamic criterion generation and iterative self-correction. This is a promising future direction for Eval Judge, but the current version prioritizes simplicity, predictability, and low operational cost.
9. Conclusion
Eval Judge does not solve every AI agent evaluation problem. It solves the most immediate one: creating a systematic evaluation layer that turns every execution into an objective quality signal.
The proposal is simple: before optimizing an agent, you need to measure whether it did what it was supposed to do. And measuring does not mean reading a few responses and guessing. It means comparing each execution against compiled instructions, using explicit criteria, and generating scores, evidence, and suggested actions.
What sets Eval Judge apart from generic approaches is its narrow scope (one execution, not the entire dialogue), its fixed four-criterion rubric, its asynchronous execution, and its direct connection to Playbook 2.0's learning cycle.
In products that run conversational AI in production, the question is not "is the model good?". The question is "is every execution correct?". Eval Judge was built to answer the second question — and, from it, improve the first.
The current scientific literature provides a solid foundation for this approach. Research on LLM-as-a-Judge, score rubrics, separation between evaluation planning and execution, and reinforcement learning for judges all point in the same direction: systematic evaluation is not a luxury — it is a production requirement.
If you want better agents, start with what almost no one measures: the quality of every single execution.
References
GU, Jiawei et al. A Survey on LLM-as-a-Judge. arXiv:2411.15594v6, 2025.
KIM, Seungone et al. PROMETHEUS: Inducing Fine-Grained Evaluation Capability in Language Models. arXiv:2310.08491v2, 2024.
LIU, Yang et al. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Proceedings of EMNLP 2023, 2023.
MADAAN, Aman et al. Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651, 2023.
SAHA, Swarnadeep et al. Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge. arXiv:2501.18099v2, 2025.
SHINN, Noah et al. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366, 2023.
WANG, Lei et al. A Survey on Large Language Model based Autonomous Agents. Frontiers of Computer Science, 2024.
WHITEHOUSE, Chen et al. J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning. arXiv:2505.10320v3, 2025.
ZHENG, Lianmin et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685v4, 2023.
ZHU, Liang et al. JudgeLM: Fine-Tuned Large Language Models Are Scalable Judges. ICLR 2025. arXiv:2310.17631v2, 2025.