AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Terse user prompts measurably reduce coding agents’ success

Benchmarks built from detailed GitHub issue reports may be painting too rosy a picture of AI coding agents. Real users tend to write requests that are shorter and more casual, and a new study finds this difference measurably hurts performance: across seven large language models, converting the same software-engineering tasks into realistic user-style prompts lowered resolution rates by 6.4 percentage points on average. That equaled an average 13.6% relative decline from baseline, with drops ranging from 10.3% to 16.2%.

Researchers at Sungkyunkwan University built a new evaluation resource called REALSWE to study the gap. They first analyzed 718 user-authored prompts from SWE-Chat, a dataset of real developer–coding-agent sessions, alongside 1,229 benchmark problems from SWE-Bench Verified and Pro. The mismatch was stark: requests containing only a problem statement, with or without minor extra context, made up 88% of real prompts but just 7% of benchmark problems. The linguistic difference was also large, with 87% of real prompts casual versus 94% of benchmark problems formal.

To make the comparison controlled, the team transformed 381 tasks from SWE-Bench Verified and Pro into multi-variant families. Each family keeps the same underlying repository issue, tests, and gold patch but changes how the request is communicated. They decomposed issue text into categories such as the problem itself, desired behavior, reproduction steps, environment information, and additional context; for feature requests, the key added category was motivation. A rephrasing step rewrote prompts in a casual, imperative user style while preserving code blocks, error messages, and file paths. The fixed REALSWE-Bench set mirrors the sparse composition seen in real chat logs, with most tasks containing only a problem statement or that statement plus minor context.

The drop was not uniform. Bug-fix tasks suffered an average 9.1-percentage-point decline, while feature requests fell only 3.7 points. Model rankings also shifted: MiMo V2.5 Pro moved from fourth place on the original inputs to second under REALSWE-Bench, overtaking Qwen3.7 Plus and DeepSeek V4 Flash. The study’s controlled ablations suggest the cause is not wording but missing key information. For bug fixes, removing Desired Behavior from an otherwise sparse request lowered resolution rates by 7.1 to 8.9 percentage points across four models, a significant drop for each. Removing reproduction steps, environment details, and extra context produced no measurable benefit. Motivation played a similar but more model-dependent role for feature requests.

Linguistic style alone mattered little: with information content held fixed, rephrasing formal issues into casual, imperative style changed bug-fix rates by roughly zero on average and feature-request rates by −1.8 percentage points on average, with no comparison reaching statistical significance.

The authors caution that their evaluation is single-turn, so it does not capture how agents might recover through clarifying questions. It also excludes the strongest frontier systems currently available, meaning the pattern may not fully generalize. Still, the practical signal is clear. Instead of asking users to write polished issue reports, the researchers say coding agents could explicitly request what behavior a bug fix should produce or why a feature is needed. REALSWE, they argue, offers a way to benchmark agents on how people actually ask for help, not just how GitHub issues describe it.