How to Evaluate Document Question Answering on Hugging Face
Build a reproducible evaluation for document question answering that tests answer quality, retrieval, source attribution, and abstention.
Summary · TL;DR
A practical workflow for comparing document QA models on held-out questions while separating retrieval failures, unsupported answers, and misleading citations.
Key Takeaways
- Start with held-out questions representative of the documents and decisions the system will actually handle.
- Compare the base and fine-tuned models on identical questions and frozen contexts before testing the complete retrieval pipeline.
- Score answer correctness, claim support, citation quality, and abstention separately.
- Save model revisions, prompts, generation settings, document versions, outputs, and known limitations with every result.
On this page (7)
Evaluate the answer and its evidence separately
For document question answering, a useful evaluation asks two questions: Is the answer correct, and do its cited passages support it? A correct answer with an irrelevant citation is not a reliable attributed answer. An answer that accurately summarizes the retrieved text can still be wrong if retrieval supplied an outdated document.
Start with a held-out set of real document questions. Compare the base model and the fine-tuned model using the same questions, source passages, prompt, and generation settings. Then evaluate retrieval separately. This follows the task-first approach in a Hugging Face community benchmarking discussion, which recommends a held-out set, a base-model baseline, fixed settings, task-specific metrics, and inspection of individual failures.
Prerequisites and evaluation scope
You need a base model and, if applicable, its fine-tuned counterpart available through Hugging Face, plus permission to use the documents in your evaluation. Install compatible versions of transformers, torch, and any model-specific dependencies. Record the installed versions; hardware and memory requirements depend on the model. The illustrative runner below has not been tested with a particular model.
Decide what you are comparing before running it:
- Generator comparison: Give both models the same frozen passages. This isolates answer generation and attribution from retrieval variation.
- End-to-end comparison: Run the actual retriever and generator together. This measures the deployed workflow, but a score change cannot automatically be attributed to the model.
Run both when possible. A generator comparison tells you whether the fine-tune helps with the same evidence; the end-to-end test tells you whether the whole system works.
Build a held-out document QA set
Hold out questions and relevant documents or document groups before fine-tuning. Check for duplicate and near-duplicate passages across splits: changing the wording of a question does not prevent leakage if the model trained on the same answer passage. For a retrieval test, retain a separate, versioned corpus that includes the documents the system is allowed to search.
Include cases in proportions that resemble expected use, and report difficult slices separately rather than presenting a deliberately hard set as typical traffic.
| Slice | Example evaluation target |
|---|---|
| Ordinary | A direct question answered in one passage |
| Multi-passage | An answer requiring evidence from two passages |
| Unanswerable | The provided context does not contain the answer |
| Conflicting or outdated | Two versions disagree; the question or metadata identifies which applies |
| Regression | A case the base model previously answered and cited correctly |
| Messy or long | Relevant evidence is embedded in a long or poorly formatted document |
For each case, store a stable ID, question, expected answer or grading rubric, source document version, supporting passage IDs, and whether the supplied context is sufficient. Specify acceptable alternative answers and citations where appropriate. If evidence requires interpretation, have a reviewer resolve ambiguous labels before treating them as ground truth.
A compact record for a fixed-context test might look like this. The policy and answer are invented solely to illustrate the format:
{
"id": "policy-001",
"question": "How long are draft reports retained?",
"context": "[policy-v1:p2] Draft reports are retained for 30 days.",
"expected_answer": "30 days",
"supporting_passage_ids": ["policy-v1:p2"],
"answerable": true
}
Use passage identifiers that resolve to the exact document version and text a reviewer can inspect. A document title alone may be too broad to verify a particular claim.
Fix the Hugging Face inference setup
Record model and tokenizer repository names and immutable revisions, any adapter and its revision, whether weights are merged or quantized, the backend, package versions, and the hardware. Also save the exact prompt and chat template, context order, truncation rules, maximum input length, and decoding parameters. For an adapter-based fine-tune, note whether the evaluated configuration is the base model alone or the base plus adapter.
The following is an illustrative, untested runner for compatible causal language models. Replace the placeholders with actual repository names and immutable revisions. It uses greedy decoding to reduce sampling variation and keeps the context fixed; it does not load an adapter, perform retrieval, or score results. Some models require a model-specific chat template or tokenizer settings. If the two models require different templates, document that difference and treat it as a limitation of the comparison.
import json
from pathlib import Path
from transformers import AutoModelForCausalLM, AutoTokenizer
MODELS = {
"base": ("YOUR_BASE_REPO", "BASE_COMMIT_HASH"),
"tuned": ("YOUR_TUNED_REPO", "TUNED_COMMIT_HASH"),
}
cases = [json.loads(line) for line in
Path("eval.jsonl").read_text(encoding="utf-8").splitlines()]
for label, (repo, revision) in MODELS.items():
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
model = AutoModelForCausalLM.from_pretrained(repo, revision=revision)
model.eval()
with Path(f"{label}-outputs.jsonl").open("w", encoding="utf-8") as out:
for case in cases:
prompt = (
"Answer only from the context. Cite passage IDs for factual "
"claims. If the context is insufficient, say so.\n\n"
f"Context:\n{case['context']}\n\n"
f"Question: {case['question']}\nAnswer:"
)
inputs = tokenizer(prompt, return_tensors="pt")
generated = model.generate(
**inputs, do_sample=False, max_new_tokens=160
)
answer_tokens = generated[0, inputs["input_ids"].shape[1]:]
answer = tokenizer.decode(answer_tokens, skip_special_tokens=True)
out.write(json.dumps({"id": case["id"], "answer": answer}) + "\n")
Before a full run, check a few prompts against each model's context limit. Do not silently truncate evidence: either apply the same documented truncation policy or flag the case as not evaluable. Greedy decoding does not guarantee identical outputs across all hardware and software configurations. If sampling is part of the intended deployment, record its settings and seeds, and repeat runs to examine variability.
Score retrieval, answers, and citations
Keep the following measures separate. For each rate, publish the numerator, denominator, and number of excluded cases; also report results by slice.
| Measure | Practical check |
|---|---|
| Retrieval recall | For questions with known supporting passages, how often did the retrieved set contain the required evidence? Define whether all required passages must be present. |
| Answer correctness | Does the answer match the reference or satisfy the case rubric? Use exact match only where wording is sufficiently constrained. |
| Claim support | Are the answer's factual claims supported by the provided passages? Review claims individually when answers contain several. |
| Citation quality | Does each cited ID exist and support the claim attached to it? Also check whether material claims have citations. |
| Abstention | On insufficient-context cases, does the model say the evidence is insufficient? On answerable cases, does it avoid unnecessary abstention? |
For example, “30 days [policy-v1:p2]” passes the illustrative case if that passage contains the stated rule. “30 days [policy-v1:p3]” should fail citation support if p3 does not establish it, even if the duration is correct. An invented passage ID should fail both the ID-validity and support checks.
Automated ID and format checks are useful, but they cannot establish that a passage entails a claim. Use a written rubric and human review for support and correctness; if using a model judge, record its version and prompt and audit a sample of its judgments. Save both model outputs and assign error labels such as missing_evidence, unsupported_claim, wrong_citation, or unnecessary_abstention.
Troubleshoot and report the result
If a correct passage is absent, inspect indexing, chunking, corpus version, and retrieval settings before changing the generator. If the passage is present but the answer is unsupported, inspect the prompt and generation behavior. If the answer is correct but cites the wrong passage, treat attribution as its own failure, not a cosmetic formatting issue.
Publish a short run manifest alongside results: dataset construction and split, document snapshot, model and adapter revisions, prompts, retrieval configuration, decoding settings, scoring rules, per-slice counts, and representative failures. With small sets, a few examples can change a percentage substantially. Repeat sampled runs where relevant, and distinguish run-to-run variation from uncertainty caused by a limited test set.
A public benchmark can add context, but it cannot replace this task-specific evidence. The defensible conclusion is bounded: the result describes these models, documents, questions, and settings—not every document QA workload.
Sources
- Hugging Face community: “What’s your method for benchmarking?” — task-first evaluation, fair base-versus-fine-tuned comparisons, and reporting considerations.
- Hugging Face LLM Course: evaluation — further background on model evaluation.
- Hugging Face Cookbook: LLM-as-a-judge — background on using a model to assist with evaluation.
FAQ
Should I evaluate retrieval or the language model first?
First compare generators on identical, frozen passages to isolate their behavior. Then evaluate the complete retrieval-and-generation workflow, including whether the retriever finds the required evidence.
Does a correct answer with a citation count as a successful result?
Not necessarily. Check that the cited passage exists and actually supports the associated claim. A correct answer with an irrelevant or invented citation fails attribution.
Can I use an LLM judge instead of human review?
An LLM judge can provide a useful signal, but its decisions should not automatically be treated as ground truth. Record its model version and prompt, use a clear rubric, and have people audit a sample.
Related articles
Guides
GPT 6.2 Astra API cost estimate for document question answering
Estimate GPT 6.2 Astra document question-answering costs using explicit token assumptions, context tiers, cache rates, and reproducible arithmetic.
Guides
Estimate Claude Fable 5.2 API Costs for Document Q&A
A reproducible way to estimate Claude Fable 5.2 costs for document question answering, with explicit token assumptions, source-attribution guidance, and dated price figures.
Guides
How to Build a Reproducible Hugging Face Evaluation for Support Triage
A practical workflow for comparing a base model and fine-tune on held-out support tickets, with human approval and transparent limitations.
Guides
GPT Image 2.6 Sunburst API Costs for Document Question Answering
Build a reproducible document question-answering budget using explicit image-input assumptions, while separating unknown answer-generation and attribution costs.