GuidesOneMux

Evaluate Hugging Face Models for Reliable JSON Extraction

A practical, reproducible evaluation plan for comparing Hugging Face base and fine-tuned models on structured extraction, including schema checks, field-level scoring, fixed generation settings, and limitations.

AI

Summary · TL;DR

To tell whether a Hugging Face fine-tune improves structured extraction, compare it with its base model on the same held-out examples, using identical prompts and generation settings. Score JSON parseability and schema validity separately from field correctness, inspect failures, and publish enough configuration detail for someone else to reproduce the run.

Key Takeaways

  • Use held-out examples that were not included in fine-tuning.
  • Evaluate base and fine-tuned model revisions with the same prompt, chat template, decoding parameters, and inference path.
  • Separate valid JSON, schema compliance, and semantic field accuracy into distinct metrics.
  • Include ordinary, difficult, malformed, ambiguous, and not-in-source examples.
  • Report sample-level failures and limitations; a small evaluation set does not establish broad reliability.
On this page (8)

The evaluation question

A structured extraction model can return syntactically valid JSON while extracting the wrong value. It can also identify the right value but omit a required key or add an unsupported field. A useful evaluation therefore answers three separate questions: Can the output be parsed? Does it match the schema? Are the extracted values correct?

For a Hugging Face base model and its fine-tuned version, answer those questions on the same held-out inputs with the same prompt and generation settings. This tests whether the fine-tune helped the extraction task—not whether it improved on an unrelated leaderboard.

This approach follows the distinction in a Hugging Face Community discussion between task-specific evaluation and public benchmarks: public scores can provide orientation, but a held-out set matched to the intended task is the more direct test. The discussion also recommends saving outputs, comparing the base and fine-tuned model fairly, and inspecting individual failures. Source: Hugging Face Community, “What’s your method for benchmarking?”

1. Define the extraction contract

Before assembling examples, write down what the model must extract, what counts as missing, and what it must do when the source does not contain an answer. Decide whether extra keys are forbidden and whether empty values, null, or an explicit abstention value represent missing information.

For example, an invoice task might require a supplier name, invoice date, and total. The evaluation contract should clarify whether dates must use ISO format, how currency is represented, and whether a missing date should be null rather than guessed. These choices affect schema design and scoring; they are not details to leave to the evaluator’s discretion.

Use a JSON Schema as the machine-checkable contract. The following is an illustrative schema, not a tested configuration for a particular Hugging Face model or serving endpoint:

{
  "type": "object",
  "properties": {
    "supplier": {"type": ["string", "null"]},
    "invoice_date": {
      "type": ["string", "null"],
      "format": "date"
    },
    "total": {"type": ["number", "null"]},
    "currency": {"type": ["string", "null"]}
  },
  "required": ["supplier", "invoice_date", "total", "currency"],
  "additionalProperties": false
}

A schema validator can check structure, types, required keys, and extra properties. It cannot establish that a value is supported by the source document. That requires labeled expected values and task-specific comparison rules.

2. Build a held-out set that reflects real inputs

Keep evaluation examples separate from fine-tuning data. If a model has already seen an example during training, a correct answer may reflect memorization rather than generalization. A compact set is useful for an initial check only if its examples and purpose are clear; do not present a small set as proof of performance across all documents.

Include examples from several slices:

  • Routine inputs: representative documents with clearly stated values.
  • Hard cases: unusual layouts, long inputs, OCR noise, or values expressed in unexpected formats.
  • Missing information: fields genuinely absent from the source, to test whether the model avoids guessing.
  • Ambiguity: conflicting values or unclear references, with a predefined expected behavior.
  • Regression cases: examples the base model already handles correctly, so a fine-tune’s regressions are visible.
  • Format stress cases: inputs likely to trigger commentary, Markdown fences, missing keys, or extra properties.

Store the source input, expected output, and any normalization rules with each example. Define rules before scoring. For example, if 1,250.00 and 1250 are equivalent totals, normalize numeric values consistently; do not change the rule after seeing which model benefits.

3. Fix the Hugging Face evaluation setup

Record the exact model ID and revision for both the base model and the fine-tuned model. If the fine-tune uses an adapter, record the adapter revision and whether evaluation loads the base plus adapter, a merged model, or another artifact. The Hugging Face discussion notes that adapters, merged weights, quantization, and serving paths can behave differently, so the evaluated artifact matters.

Keep these settings identical between the two models wherever applicable:

  • system message, prompt template, and chat template;
  • input examples, ordering, and any few-shot demonstrations;
  • decoding parameters, including temperature, top-p, and maximum new tokens;
  • inference backend, quantization, and relevant context handling.

For a repeatable extraction comparison, deterministic decoding is a reasonable starting choice when the selected model and backend support it. Record the actual parameter values rather than assuming every endpoint honors every option. If an endpoint does not document support for a setting, mark it as unknown and do not claim the runs were identical on that setting.

Keep the prompt instruction explicit: return only the requested JSON object, follow the schema’s missing-value convention, and do not infer values absent from the input. This does not guarantee compliance; that is what the evaluation measures.

4. Score validity and correctness separately

Report counts as well as percentages, especially for small datasets. A useful scorecard is:

MeasureWhat it checksWhat it does not prove
Parseable JSON rateOutput can be decoded as JSONKeys or values are correct
Schema-valid rateParsed output meets the chosen schemaValues match the source
Required-field completionRequired keys are presentValues are accurate
Field-level exact matchEach field matches its labeled value after declared normalizationGeneralization beyond the evaluation set
Record-level exact matchEvery evaluated field in a record is correctRobustness to unseen formats

For each measure, report the numerator and denominator—for example, “18 of 20 outputs passed schema validation”—and the percentage. State how you handle outputs with surrounding prose or Markdown fences. One consistent policy is to score the raw response first: if the task requires JSON-only output, extra text is a format failure. Avoid silently repairing model outputs unless repair is part of the real production workflow; if you do evaluate a repair step, report its results separately.

5. Inspect failures, not just aggregate scores

Save each input, reference output, raw base-model output, raw fine-tuned output, validation result, and field-level scores. Assign a short error label such as invalid JSON, missing key, wrong value, unsupported inference, extra property, or incorrect abstention. Then inspect examples where the fine-tuned model improved and where it regressed.

A schema-validity increase can hide worse extraction accuracy. Conversely, a model may extract more fields correctly while still producing occasional malformed JSON. Keep the measures separate so a team can decide which failure matters most to its workflow.

For a small set, a few examples can move the percentage substantially. Report the sample size, and avoid treating a narrow score difference as conclusive. The supplied discussion’s later reply recommends multiple runs with different seeds because small-set scores can vary; whether this applies depends on the model’s sampling behavior and your inference setup. If outputs are deterministic, repeated identical runs may simply reproduce the same results rather than provide independent evidence.

6. Publish a reproducibility note and limitations

A useful report includes the dataset’s source and split, example count, exclusion from training, model and adapter revisions, prompt and chat template, generation settings, inference path, validator and scoring rules, and representative failure examples. Note any settings that could not be confirmed or held constant.

Be precise about the conclusion: “The fine-tuned revision passed schema validation on 18 of 20 held-out examples in this test” is supportable. “The model reliably extracts invoices” is broader than a small, narrow evaluation can establish. Public benchmarks may be a second layer if they match the goal, but they do not replace testing the actual extraction contract.

Conclusion

For Hugging Face JSON extraction, make the evaluation a reproducible comparison, not a demo: hold out representative examples, fix and record the setup, validate structure, score field values, and inspect failures. Report the limits of the dataset and the inference configuration alongside the result. That gives readers a basis to judge whether the observed change is relevant to their own extraction workflow.

FAQ

Is valid JSON enough to show that a Hugging Face model extracts correctly?

No. Parseability only shows that the output can be decoded as JSON. Check schema validity, required fields, and field-level correctness against labeled source examples as separate measures.

Should I compare a fine-tuned model with its base model?

Yes, if the goal is to measure whether fine-tuning helped. Run both on the same held-out examples with the same prompt, chat template, generation settings, and inference path where possible, and record any differences you cannot hold constant.

What belongs in a JSON extraction evaluation set?

Include routine examples, difficult or messy inputs, missing fields, ambiguous cases, format stress tests, and examples the base model already handles. Keep the set separate from fine-tuning data and document how expected values are labeled.

Can JSON Schema validation measure whether extracted values are true?

No. Schema validation checks structure and data types, not whether a value is supported by the input. Use labeled expected values and field-level scoring for extraction correctness.

How should I handle model outputs with prose around the JSON?

Decide before testing. If the task requires JSON only, score surrounding prose as a format failure. If a production parser or repair step removes it, evaluate and report that separate workflow rather than silently cleaning outputs.

Related articles