AI Papers Reader

Personalized digests of latest AI research

View on GitHub

When an AI Rewrites UI Code, One Small Fix Can Break the Whole Page

Large vision-language models can now translate a screenshot of a webpage into HTML/CSS code. But when such a model tries to improve its own draft, the revision can make things worse. Researchers from the University of Maryland and Microsoft identify why: they call it “visual repair coupling.” In UI code, a local edit can propagate through layout, style, and component dependencies, so fixing one mismatch may damage regions that were already faithful. They propose RubSE, a rubric-guided self-evolution framework that tries to keep each repair focused.

The task, UI-to-code generation, matters for prototyping and developer workflows. The authors studied self-evolution, where a model generates code, renders it, compares the rendering with the target screenshot, and revises the code. In an unstructured loop, this behaves unstably. To make feedback more useful, RubSE works with “rubrics”: short structured notes that describe a visual mismatch, name a visual category, and suggest a correction direction. Categories include layout geometry, spacing and density, typography, visual styling, and completeness.

At each refinement round, an EVOLVE step generates candidate rubrics for remaining mismatches. A SELECT step chooses one prioritized rubric as the target for the next edit. A HISTORY step stores previously selected rubrics and passes them back as an avoid list, discouraging repeated or over-broad changes. The model then rewrites the code with that single rubric as context. This keeps revisions flexible but scoped.

The mechanism is visible in a representative example: a model attempting to fix the parent assignment of a bottom gray bar introduced fresh errors in page scale, bar length, and the background color of a bottom-left container. RubSE is designed to prevent such collateral drift by making the repair target explicit.

The team tested RubSE and naive self-evolution with six models—GPT-5.4, GPT-5.2, Claude-Sonnet-4.5, Qwen3-VL-32B-Instruct, Qwen3.5-9B, and Qwen3.6-35B-A3B—on three benchmarks: Design2Code, its hard subset, and UI2Code-Real. Across 18 model-benchmark settings, RubSE beat naive self-evolution at the final round in 15, with average gains of +1.20 points on a 0–100 visual-fidelity score and +0.11 points on a 1–7 aspect-mean score, the mean of five aspect ratings. On the best intermediate round, it won in 14 settings, with an average gain of +1.13 overall points. In a human evaluation of 60 UI2Code-Real samples, round-10 RubSE outputs were preferred over naive outputs for both GPT-5.2 and GPT-5.4 (p < 0.01). The authors also found that rubrics generated by GPT-5.4 could improve self-evolution in weaker Qwen models, suggesting high-quality repair guidance transfers.

The extra structure has costs and limits. Using GPT-5.4, RubSE cost about 1.6 times as much as naive self-evolution in API fees; with Qwen3.5-9B, inference latency rose only 2.5%. The authors note that RubSE does not guarantee code correctness: a revision can still introduce syntax or runtime errors that break rendering. Automated scoring by a vision-language judge also remains imperfect for subtle visual differences.

RubSE does not eliminate visual repair coupling, but it gives self-evolving systems a way to decide both what to change and what to preserve. For tasks where a small edit can ripple through an entire generated artifact, the authors say structured repair context may be a step toward more reliable AI self-improvement.