How to Reproducibly Evaluate Hugging Face Models on Repository Code Changes

The evaluation decision: did the model make a tested repository change?

For repository code changes, evaluate the workflow you expect the model to perform: understand a repository snapshot, propose a patch, and pass relevant tests. A code-completion score or a good-looking diff does not establish that a change works in context.

A practical comparison uses the same held-out tasks, repository state, prompt template, and generation settings for each model. Score at least two outcomes separately: could the output be applied as a patch? and did the selected tests pass? Neither result alone proves that the change is correct or safe.

This follows the central advice in a Hugging Face community discussion on benchmarking: start with the target task, compare models under the same settings, inspect sample failures, and treat public benchmarks as a second layer rather than proof of task-specific improvement. The thread also recommends recording setup details so others can reproduce the evaluation. Source: Hugging Face community discussion, “What’s your method for benchmarking?”

1. Create held-out repository tasks

Each task should include a repository commit, a concise issue description, and a test command or other explicit acceptance check. Keep evaluation tasks out of model training and prompt iteration. If you repeatedly tune prompts against the same tasks, they become development examples rather than a clean held-out test set.

Choose tasks that reflect the changes you care about: ordinary bug fixes, edge cases, changes spanning multiple files, and cases where the right action is to report insufficient information. Prefer tasks with tests that distinguish the intended behavior from the existing behavior. Record known gaps: a test suite may not cover security, performance, compatibility, or every regression.

Use a manifest so each task can be reconstructed:

Do not silently vary the repository state between models. Store the manifest and task inputs with the evaluation results, while respecting any repository licensing or privacy requirements.

2. Pin the Hugging Face model and generation setup

Use a specific model revision rather than an unspecified moving reference. Record the model ID, revision, Transformers version, tokenizer and chat template, hardware, and whether you used an adapter, merged weights, or quantization. A model’s behavior can depend on these choices.

The following is an illustrative, untested configuration for generating one patch proposal with Transformers. Replace the placeholders and confirm the model’s supported architecture and chat template before running it. It generates text; it does not apply or test the resulting patch.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "YOUR_HUGGING_FACE_MODEL_ID"
revision = "YOUR_PINNED_REVISION"

messages = [
    {
        "role": "user",
        "content": (
            "Given this repository context and issue, return a unified diff only. "
            "Do not claim tests passed.\n\n"
            "Repository context:\nYOUR_FIXED_REPOSITORY_CONTEXT\n\n"
            "Issue:\nYOUR_HELD_OUT_ISSUE"
        ),
    }
]

tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    revision=revision,
    torch_dtype="auto",
    device_map="auto",
)

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        inputs,
        do_sample=False,
        max_new_tokens=1200,
    )

new_tokens = output[0, inputs.shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Greedy decoding (do_sample=False) removes sampling as one source of variation, but it does not guarantee identical outputs across all hardware, libraries, or kernels. Record the exact environment. If sampling is part of the intended workflow, fix and record the seed and sampling parameters; repeat runs if you need to understand output variability. Do not assume the example’s settings are appropriate for every model.

3. Apply and test patches in isolation

For each task-model pair, start from a clean checkout of the recorded commit. Save the raw model output before parsing it. Validate that it contains a usable diff, apply it in a disposable worktree or container, and run the predeclared test command. Keep model generation separate from test execution so a failed patch is not mistaken for an infrastructure or test-runner failure.

Record outcomes individually:

A passing test command is evidence that the selected tests passed in the recorded environment. It is not proof of correctness beyond those tests. Report test setup failures separately from model failures, and do not count an unrun test as a pass.

4. Compare models without hiding regressions

Run the same tasks with the same repository snapshots and prompt for the baseline and candidate models. Report the number of tasks, patch-application rate, test-pass rate, and how many tasks were inspected. Include task-level outcomes or a failure summary, not only an average. A model may improve a narrow set of fixes while regressing on formatting, scope control, or another task slice.

For a small evaluation set, describe the result as a result on that set—not as a general ranking. The source discussion notes that repeated runs can matter because small-set scores may move around. If outputs are stochastic, use repeated runs with recorded seeds and report the variation; if generation is deterministic, still be cautious about drawing broad conclusions from few tasks.

5. Publish the limitations with the result

A reproducible report should include model and revision, repository commits, task-selection method, exclusions, prompt, generation parameters, software and hardware environment, patch parser, test commands, and scoring rules. State what the evaluation does not measure: for example, untested regressions, code review quality, security properties, latency, or performance unless you measured them.

The useful conclusion is specific: on these held-out repository tasks, under this recorded setup, the model produced patches with these application and test outcomes. That is more informative than a leaderboard score when the real question is whether a model can make verified changes in your codebase.