You built a chatbot. It summarizes meeting notes, answers support tickets, and pulls dates from contracts. It looks great in the demo. But how do you know it’s actually good? If you rely on eyeballing a few outputs, you’re guessing. If you use generic accuracy scores, you’re missing the point. A summary that is factually correct but three times longer than the original isn’t helpful. An answer that is grammatically perfect but misses the specific date requested is useless. This is where task-specific scorecards come in.
Evaluating Large Language Models (LLMs) isn’t a one-size-fits-all problem. The way you judge a summary differs fundamentally from how you judge an extracted entity or a direct answer to a question. In this guide, we’ll break down exactly how to build scorecards for these three core tasks. We’ll move beyond basic ROUGE scores and look at semantic matching, LLM-as-a-judge frameworks, and practical implementation strategies that keep your models honest.
The Four Pillars of LLM Evaluation
Before diving into specific tasks, you need a framework. Most successful evaluation systems don’t just look at "accuracy." They track four distinct dimensions. Think of this as your dashboard’s main tabs.
- Quality: Is the output correct, helpful, and coherent? This is the core semantic assessment.
- Outcome: Did the user actually get what they needed? Did the summary save them time? Did the answer resolve the ticket?
- Performance: How fast was it, and how much did it cost? Latency and token usage matter for production viability.
- Safety & Compliance: Did the model hallucinate private data? Was the tone appropriate? Did it violate any policy?
Ignoring any of these pillars creates blind spots. You might have a high-quality model that is too slow to be useful, or a fast model that leaks customer PII. Your scorecard must balance all four.
Judging Summarization: Beyond Simple Overlap
Summarization is tricky because there are many ways to say the same thing correctly. Traditional metrics like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measure n-gram overlap between your generated summary and a reference summary. It’s the old standard. If your summary uses the exact same words as the human-written reference, ROUGE gives you a high score. But if your model paraphrases well-using different words with the same meaning-ROUGE penalizes you. It’s blind to semantics.
To fix this, modern scorecards use BERTScore. Unlike ROUGE, BERTScore uses contextual embeddings from transformer models. It compares the vector representations of tokens rather than raw strings. This means it recognizes that "car" and "vehicle" are similar, even if they aren’t identical strings. AWS documentation highlights that BERTScore mirrors human judgment more closely by capturing these semantic nuances.
But semantic similarity isn’t enough. A summary can be semantically accurate but terrible because it’s too long. This is where the Ragas framework shines. Ragas introduces a Summarization Score that combines two components:
- QA Score: It extracts key phrases from the source text, generates questions based on those phrases, and checks if the summary can answer them. If the summary contains the necessary info, it passes.
- Conciseness Score: It penalizes verbosity. The formula is roughly
1 - (summary_length / context_length). If your summary is nearly as long as the original article, the score drops.
The final score is a weighted average of these two. This prevents the "copy-paste" failure mode where the model just regurgitates the first paragraph. For abstractive summaries, where the goal is synthesis rather than extraction, this dual-metric approach is far more reliable than ROUGE alone.
Evaluating Question Answering: Precision Matters
In Question Answering (Q&A), brevity is usually a virtue, but only if the answer is complete. Here, the challenge shifts from coverage to precision. Did the model find the right needle in the haystack? Standard metrics like Exact Match (EM) are too rigid. If the gold answer is "New York City" and the model says "NYC," EM fails it. That’s bad UX.
A better approach for Q&A scorecards involves F1 Score at the token level, which balances precision and recall. However, for open-ended Q&A, you often need reference-free evaluation. This is where G-Eval becomes critical. G-Eval uses an LLM (like GPT-4) as a judge. You provide the question, the ground truth (if available), and the model’s answer. You then prompt the judge to score the answer on criteria like correctness, relevance, and completeness using Chain-of-Thought reasoning.
| Metric | Best For | Limitations | Implementation Effort |
|---|---|---|---|
| Exact Match | Short, factual answers (e.g., dates, names) | Fails on synonyms or formatting variations | Low |
| Token F1 | Partial matches and noisy data | Doesn't capture semantic equivalence | Low |
| BERTScore | Semantic similarity in long answers | Can be computationally heavy | Medium |
| G-Eval (LLM Judge) | Complex, nuanced, or creative answers | Costly; requires careful prompt engineering | High |
For production systems, a hybrid approach works best. Use automated metrics for bulk monitoring and trigger an LLM-based review only when automated scores fall below a threshold. This keeps costs manageable while catching subtle errors.
Information Extraction: Handling Boundaries and Coreference
Extraction tasks-like pulling entities, relations, or events from text-present unique challenges. The main issues here are boundary errors and coreference resolution. If a model extracts "Apple Inc." instead of "Apple," is that wrong? Technically, yes, if your schema expects strict string matching. But functionally, it’s correct.
Your scorecard needs to handle partial matches. Instead of binary pass/fail, use Entity F1, which calculates the intersection of predicted and true entities divided by the union. This rewards models that get most entities right, even if they miss a few.
Coreference is another trap. If the text says "Steve Jobs founded Apple. He died in 2011," and you ask for the founder of Apple, the model should link "He" to Steve Jobs. If it returns "He," your extraction failed. Advanced evaluation pipelines now include coreference resolution steps before scoring, ensuring that pronouns are resolved to their antecedents before comparison.
Building the Scorecard: Practical Implementation
So, how do you put this together? You don’t run every metric on every query. That’s expensive and slow. Instead, structure your evaluation pipeline in layers.
Layer 1: Automated Filters. Run cheap, fast metrics like ROUGE-L or Token F1 on 100% of traffic. These catch obvious failures (empty responses, garbage text). Set thresholds. If a response scores below 0.5 on ROUGE, flag it.
Layer 2: Semantic Checks. For flagged items or a random sample (e.g., 5%), run BERTScore or Ragas Summarization Score. These require GPU resources but give you a better view of quality.
Layer 3: Human-in-the-Loop Audit. Take the worst-performing samples from Layer 2 and have humans rate them. Use these ratings to calibrate your automated thresholds. If humans consistently disagree with your LLM judge, tweak the judge’s prompt.
Microsoft’s AI evaluation guidance suggests using Head-to-Head (H2H) scoring for calibration. Instead of asking a judge to score a single output 1-10, show it two outputs (one from Model A, one from Model B) and ask which is better. Relative judgments are often more consistent than absolute ones.
Pitfalls to Avoid
Even with a good scorecard, you can fool yourself. Watch out for these common traps:
- The Reference Bias Trap: If you evaluate against a single human-written summary, you penalize valid alternative phrasings. Use multiple references or reference-free methods like G-Eval to mitigate this.
- The Verbosity Bias: Some LLM judges prefer longer answers, assuming length equals detail. Explicitly instruct your judge to penalize redundancy. The Ragas conciseness score helps automate this.
- Static Benchmarks: Don’t just test on public datasets like CNN/DailyMail. Your users ask weird questions. Build a custom eval set from your actual production logs. Include edge cases, typos, and ambiguous queries.
Conclusion: Iteration is Key
There is no perfect metric. ROUGE misses meaning. BERTScore misses facts. LLM judges are biased. The solution isn’t finding the magic number-it’s building a composite scorecard that triangulates quality. Start with simple automated checks. Add semantic layers as you scale. Regularly audit with human feedback. By tailoring your evaluation to the specific job-whether it’s compressing text, answering questions, or extracting data-you turn vague "model performance" into actionable business insights.
Why is ROUGE not enough for evaluating LLM summaries?
ROUGE measures surface-level word overlap. LLMs often paraphrase content, using synonyms or restructuring sentences. ROUGE sees these as mismatches, giving low scores to high-quality, fluent summaries that don't copy the reference text verbatim.
What is the Ragas Summarization Score?
It's a composite metric that evaluates both content fidelity and conciseness. It generates questions from the source text to check if the summary retains key information (QA Score) and penalizes overly long summaries (Conciseness Score), combining them into a single value.
How does G-Eval work for LLM evaluation?
G-Eval uses a powerful LLM (like GPT-4) as a judge. It provides the input, output, and evaluation criteria to the judge model, which then uses chain-of-thought reasoning to assign a numeric score. It allows for reference-free evaluation and captures nuanced quality aspects that traditional metrics miss.
Should I use human evaluation for every LLM output?
No, it's too expensive and slow. Use automated metrics for bulk monitoring and reserve human evaluation for sampling audits, calibration of automated thresholds, and investigating outliers where automated scores are inconsistent.
What is the biggest challenge in evaluating Information Extraction?
Boundary errors and coreference. Determining whether "New York" and "New York City" are the same entity, or resolving pronouns like "he" to the correct person, requires sophisticated post-processing before standard metrics like Entity F1 can be applied accurately.
