How to Configure Cline for Repeatable Coding-Agent Evaluations
Configure ClineCore sessions with fixed inputs, explicit tool boundaries, repeatable test cases, and a troubleshooting checklist.
Summary · TL;DR
Use ClineCore to make coding-agent evaluations comparable: fix the model and workspace, specify task boundaries, constrain tools through supported runtime controls, and record outcomes alongside tests and usage.
Key Takeaways
- Keep the prompt, model configuration, workspace, and test commands constant between evaluation runs.
- ClineCore accepts session-level configuration, tool policies, and runtime capabilities; exact policy and callback schemas should be checked in the SDK version you install.
- Treat a prompt as task guidance, not a security boundary. Restrict executable capabilities and approval paths explicitly.
- Capture test results, session messages, and accumulated usage so failures can be diagnosed rather than judged by the final answer alone.
On this page (10)
The practical answer
For a fair coding-agent evaluation, make each Cline run start from the same repository state with the same prompt, provider and model identifiers, runtime settings, and test commands. Give the agent only the tools the test requires, then record both the test result and the session history. A prompt that says “do not edit files outside this folder” is useful guidance, but it does not itself enforce that boundary.
ClineCore provides session configuration, per-session tool policies and capabilities, session history, message reading, usage retrieval, and abort/stop controls. The reference does not define the detailed shape of policy values or capability callbacks, so verify those against the SDK version in your project before relying on them. ClineCore API reference
Prerequisites and evaluation design
Before writing a harness, decide what counts as success and what the agent is allowed to do.
- Fix the starting state: use a clean checkout or reset the same test fixture before every run. Record the commit or fixture identifier.
- Define one task per case: include the request, expected behavior, and an executable test command. Avoid vague goals such as “improve the code.”
- Set boundaries: decide whether the run may edit files, run tests, access the network, or spawn agents. Enable only what the case needs.
- Choose a stopping rule: for example, stop after the test command completes or when a human reviewer intervenes. ClineCore documents
abortandstopmethods, but the reference does not promise a particular timeout mechanism. - Record the environment: save the Cline SDK version, provider and model IDs, relevant configuration, test output, and session ID.
Run evaluations sequentially if they share a writable workspace. Parallel agents editing the same checkout can contaminate results.
Configure a fixed ClineCore session
The example below follows the documented create and start shape. It is illustrative, not a tested integration recipe; replace the model and provider identifiers with values supported by your chosen provider, and keep secrets outside source control.
import { ClineCore } from "@cline/sdk"
const cline = await ClineCore.create({
clientName: "cline-eval-harness",
backendMode: "auto",
})
const session = await cline.start({
prompt: [
"Implement the requested change in this repository.",
"Do not change files outside the task scope.",
"Run the specified test command and report its exact result.",
"Do not spawn agents or agent teams.",
].join("\n"),
source: "repeatable-evaluation",
interactive: false,
config: {
providerId: "anthropic",
modelId: "claude-sonnet-4-6",
apiKey: process.env.ANTHROPIC_API_KEY,
systemPrompt: "Make the smallest change that satisfies the task.",
cwd: process.cwd(),
workspaceRoot: process.cwd(),
enableTools: true,
enableSpawnAgent: false,
enableAgentTeams: false,
},
})
console.log({ sessionId: session.sessionId, result: session.result })
The provider and model shown are example values from the reference, not a recommendation or proof of availability. ClineCore also documents toolPolicies and capabilities at creation or session level. Use those controls, where supported by your installed SDK, to govern approvals and the tools your client executes. The source summary does not specify a policy schema or callback signatures; do not copy guessed policy keys into a harness.
For provider-independent evaluation settings, keep the task, workspace, test suite, interaction mode, tool boundaries, and recording procedure unchanged. Provider IDs, model IDs, credentials, and provider-specific request parameters are not provider-independent. Confirm that the provider accepts the configuration your runtime sends; compatibility with an OpenAI-compatible endpoint, for example, depends on that endpoint's documented parameter support. No particular third-party endpoint integration is established by the Cline reference.
Make the test case reproducible
Keep each case in a small manifest or version-controlled test file. A useful case records:
| Field | Example value |
|---|---|
| Case ID | add-input-validation |
| Starting state | Commit or fixture identifier |
| Prompt | Exact text supplied to Cline |
| Allowed actions | File edits and specified local tests only |
| Test command | Exact command and expected exit status |
| Review criteria | Required behavior and prohibited changes |
| Run record | Session ID, SDK version, model ID, test output |
The case name and values above are illustrative. For each run, retain the Cline session ID and retrieve messages with readMessages; use getAccumulatedUsage when you need the session's reported accumulated token or cost usage. These records help distinguish an implementation failure from a test that never ran or an agent that misunderstood the task. Avoid comparing raw cost or usage across providers unless their reporting units and accounting are known to be comparable.
Troubleshooting common evaluation failures
The run changes between repetitions
Check for a dirty working tree, changing dependencies, generated files, environment-dependent tests, or a prompt assembled from non-fixed data. Reset the workspace and compare the exact prompt and session config before drawing conclusions.
The agent uses an unexpected tool
Do not rely only on instructions in the prompt. Review the configured tool policies and the client-owned capabilities and approval callbacks. Confirm the installed SDK's supported schemas, then reduce available execution paths or require approval. The reference lists these controls but does not enumerate every tool or policy value.
The run ends without a clear result
Inspect the session messages and result, then check whether the test command actually ran. The documented start result includes a session ID and may include a result; send returns an AgentResult or undefined. Handle the possibility of no result in the harness rather than treating it as a passing test.
You need to stop a run or inspect prior work
Use the session ID with the documented abort or stop method when intervention is needed. For debugging, ClineCore also documents session lookup, message reading, history listing, and checkpoint restoration. Test restore behavior separately before making it part of a formal evaluation protocol.
A compact evaluation checklist
- Reset to a known repository state.
- Pin the SDK version, provider/model identifiers, prompt, and session configuration.
- Set workspace roots deliberately and constrain tool execution through supported controls.
- Run one named case with a fixed test command.
- Save session ID, messages, test output, and relevant usage data.
- Repeat from the same starting state; investigate any environmental difference before comparing outcomes.
ClineCore supplies useful pieces for a repeatable harness, but reproducibility comes from the surrounding test design: fixed inputs, explicit permissions, controlled workspaces, and recorded evidence.
FAQ
Which Cline settings should stay fixed between coding-agent evaluation runs?
Keep the prompt, provider and model identifiers, Cline session configuration, workspace state, tool boundaries, test command, and SDK version fixed. Record any environment values that could affect the result.
Does telling Cline not to edit a file enforce that restriction?
No. A prompt communicates intent but is not an execution boundary. Use the supported tool policies, client-owned capabilities, and approval callbacks for enforcement, after confirming their exact schemas in your SDK version.
Can ClineCore use any provider or OpenAI-compatible endpoint?
The supplied ClineCore reference shows provider and model identifiers in session configuration, but it does not establish universal provider support or a specific endpoint integration. Confirm that your provider and endpoint document support for the configuration and parameters your runtime uses.
How do I investigate a failed or incomplete Cline run?
Check the session result and persisted messages, confirm whether the expected test command ran, and inspect the workspace for unintended changes. ClineCore documents session lookup, message reading, accumulated usage retrieval, and stop or abort methods.
Related articles
Guides
Cline in Production: Configure Sessions and Recover Safely
A practical guide to configuring ClineCore sessions, separating runtime settings from provider configuration, and designing safe retry and recovery rules.
Guides
Configure ClineCore for Privacy-Aware Coding Sessions
A practical guide to configuring ClineCore sessions, limiting sensitive data exposure, and troubleshooting common setup issues.
Guides
Configure Cline to Diagnose an API Incident
Configure a Cline SDK session for API incident diagnosis, choose provider-independent runtime settings, and troubleshoot observable failure signals.
Guides
Configure Cline with an OpenAI-Compatible Provider for JSON Schema Extraction
Set up an OpenAI-compatible provider in Cline, then use a coding-agent workflow to build and validate structured extraction with JSON Schema.