The agent repairs the original checkout
The issue, repository, tools, verifier, and interaction budget follow the standard autonomous setting.
User-agent interaction beyond messages
SWE-Touch tests whether a coding agent can complete a software repair after a user has directly changed the code in their shared workspace.
9coding models
3repair benchmarks
−7.7mean points on Verified
Research question
When a user contributes code during an ongoing task, can the agent interpret that edit against the issue, repository, and tests and still reach a correct solution?
Evaluation design
From autonomous repair to a shared workspace
Each task is evaluated under matched conditions. The user edit is the only change to the live task state.
vs.
A user changes relevant code during the repair
A small, plausible but task-conflicting edit and a contextual user message enter the same live workspace.
Main evidence
User edits expose a gap hidden by autonomous repair
Resolve rates are percentages. Verified results report mean ± standard deviation.
−7.7average change on SWE-bench VerifiedAll nine evaluated models decline under Counter-Edit.
−1.3 to −16.5large variation across modelsStrong autonomous scores do not imply consistent recovery.
63.3%failures retain the conflictMost failed trajectories end with the user-introduced behavior still active.
SWE-bench Verified
Retention is the fraction of tasks solved under Vanilla that remain solved under Counter-Edit.
Resolve-rate decrease
Counter-Edit minus Vanilla, percentage points
GPT 5.5−1.3
Claude 4.8−1.8
GLM 5.1−4.3
Qwen 3.7−4.8
Kimi K2.6−6.0
MiniMax M2.5−9.5
DeepSeek V4−11.0
MiniMax M2.7−13.8
Qwen Coder−16.5
Finding. The ordering changes substantially: Qwen3-Coder-480B and MiniMax M2.7 lose more than 13 points, while GPT 5.5 and Claude Opus 4.8 retain over 95% of their Vanilla-solved tasks.
Harder repair tasks
Selected SWE-Bench Pro and DeepSWE tasks use a longer budget and trajectory-relative delivery.
Finding. The aggregate direction persists on both harder benchmarks, although the effect varies by model and task family.
Message, edit, and frequency
Exploratory resolve rates on SWE-bench Verified. K is the maximum number of interventions.
Finding. Messages alone do not reproduce the effect consistently. Direct changes to executable code are the stronger intervention in this comparison.
Edits stay local
Mean changed lines and files per patch.
Finding. Counter-Edits remain small even when the underlying reference repair spans hundreds of lines and several files.
Trajectory analysis
Failure is not only passive acceptance
Agents sometimes remove or replace the inserted edit and still fail. Recognizing a problematic contribution does not guarantee that the model can restore the repository to a correct state.
63.3%of Vanilla-solved / Counter-Edit-failed runs end with the conflicting behavior retained.
Open release
Reproduce and extend SWE-Touch
The release separates reusable benchmark records from the Harbor-based evaluation implementation.
Code
Runner, schemas, user simulator, validation gates, and reproducibility commands.
Open repositoryDataset
Task records, critical regions, Counter-Edits, and validation evidence.
Browse on Hugging FaceQuick start
Install the package, validate the release, and launch Vanilla or SWE-Touch evaluation.
Read the guide