AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Direct Whole-File Generation Beats Iterative Diffs for Flutter/Dart Editing, Except on Local Edits

When a coding model edits a file, should it rewrite the whole file or emit search-and-replace diffs? Production tools often favor diffs because they are shorter and mirror human patches. A new empirical study on Flutter/Dart editing finds that, as a training objective, direct whole-file generation is substantially stronger across two models—though diff-based editing remains competitive on short, spatially local changes.

The paper compares four fine-tuned arms. Direct arms output the complete modified file from the original file and instruction. “Steps” arms emit search/replace edits applied one at a time until completion or a 20-step budget. The backbones are Rainbow-Pony-100M, trained from scratch, and Qwen2.5-Coder-0.5B, fine-tuned from a pretrained code model. All are evaluated on held-out Flutter/Dart tasks: 1,789 examples for Rainbow-Pony and 1,792 for Qwen.

Direct wins on every reported metric. On Dart static-analysis pass rate (dart_pass), Rainbow-Pony direct scores 0.802 versus 0.347 for steps—a 45.5 percentage-point gap. Qwen direct scores 0.900 versus 0.501, a 39.9-point gap. Direct also has lower bits-per-byte and higher character similarity to the reference.

The failure pattern is not mainly budget exhaustion or malformed edits. Most steps trajectories completed normally: 85.2% for Rainbow-Pony and 81.3% for Qwen, and those runs account for roughly 84% and 70% of steps failures. A key mechanism is ambiguity. If a search span appears more than once, the harness uses a first-occurrence heuristic rather than rejecting the edit. Among completed Rainbow-Pony trajectories, dart_pass is 0.570 without fallback edits but 0.123 with them; for Qwen it is 0.800 without and 0.175 with. The authors call this silent content corruption within completed trajectories.

A concrete steps example: for a simple stateless Flutter widget displaying centered text, the model starts with only an import. It first adds the import, then adds a main method by replacing that import with the import plus runApp(MaterialApp(home: SimpleTextWidget())). This illustrates the localized edit chain; the paper separately identifies ambiguous repeated spans that can mislead the harness.

Matched-ID comparisons control for task difficulty. On clean steps IDs versus direct on the same IDs, Rainbow-Pony steps passes 57.2% versus 82.1%, and Qwen steps passes 80.0% versus 92.4%. A blinded LLM judge, scoring only outputs that compile on both sides, also rated direct higher on goal fulfillment, correctness, and code quality; all six differences were significant at p < 0.001.

Diff-based editing is not uniformly worse. Steps wins concentrate on short trajectories and on refactoring and error-handling/edge-case tasks—the two lowest mean-step-count categories for both architectures. The authors call this “task locality” and suggest format choice should be per edit, not per model.

Limitations are substantial. The study covers only Flutter/Dart, uses greedy decoding, and steps mode gets no compiler or test feedback between edits. Steps fine-tuning also used about 50M tokens versus 5M for direct, though the authors argue this asymmetry cuts against their central finding. They flag the fallback heuristic and the lack of human validation for the LLM judge. The broader implication is not that diffs fail universally, but that whole-file generation is a stronger default here, while diff-based methods may suit narrowly localized changes.