Simplicity Prevails: New “Agentless” LLM Approach Outperforms Complex Software Engineers at a Fraction of the Cost
A new approach to automated software development is challenging the industry’s reliance on complex, autonomous AI agents, demonstrating that a simple, highly structured process can fix real-world bugs more efficiently and cost-effectively.
Researchers at the University of Illinois Urbana-Champaign have introduced AGENTLESS, a three-phase system that strips away the complex decision-making and tool-usage requirements typical of high-profile AI software agents like Devin and SWE-agent. Evaluated on the popular SWE-bench Lite benchmark, AGENTLESS achieved a 32.00% success rate—the highest among all open-source techniques—while operating at a fraction of the cost, averaging just $0.70 per resolved problem.
This finding suggests that the iterative planning, complex tool abstractions, and lengthy self-reflection mechanisms used by many advanced LLM agents often confuse the model rather than help it solve problems.
A Structured Approach to Bug Fixing
AGENTLESS operates by breaking the problem down into three non-autonomous stages: localization, repair, and patch validation. The core philosophy is to remove the burden of deciding what to do next from the large language model (LLM) and instead provide it with highly precise, curated context at each step.
1. Hierarchical Localization: To find the bug, AGENTLESS employs a triage system. For a complex bug report spanning a massive codebase, the LLM first analyzes the repository file structure (like a tree command output) to identify the top suspicious files. It then drills down hierarchically, examining only the file skeletons (class and function headers) to localize the relevant code elements, and finally zeroing in on the exact lines of code that require modification.
For instance, rather than asking the LLM to search through thousands of lines, AGENTLESS first directs it to the top two files, then to the Handler class within one file, and finally to the specific edit_matrix function between lines 120 and 230.
2. Efficient Repair Generation: Once the precise location is identified, the LLM generates multiple candidate fixes. Crucially, AGENTLESS uses a simple Search/Replace diff format. Instead of requiring the LLM to output an entire, potentially buggy, replacement file, it asks for a minimal patch: “In this snippet, replace X with Y.” This focus on small, efficient edits drastically reduces costs and the likelihood of the LLM suffering from “hallucination”—making up code that seems correct but isn’t.
3. Robust Validation: The system then filters and ranks the candidate patches. It first requires the patches to pass existing regression tests. For a final check, AGENTLESS asks the LLM to generate new, specific reproduction tests designed to confirm the original issue is fixed.
Resetting the Baseline
The results are stark: AGENTLESS resolves 96 issues on SWE-bench Lite, achieving cost efficiencies far beyond its competitors. The average cost of $0.70 per fix is significantly lower than other open-source agents, some of which require well over $1 per attempt due to complex tool interactions and excessive LLM queries.
The simplistic yet highly effective design of AGENTLESS has already garnered industry attention. OpenAI has adopted the agentless approach to showcase the real-world coding performance of its new GPT-4o and o1 models, confirming its position as a new standard for rigorous evaluation in autonomous software development.
The research not only highlights the overlooked potential of simplicity in AI software engineering but also provides a more rigorous benchmark, SWE-bench Lite-S, by excluding problematic problems that contained misleading descriptions or already included the exact fix. Researchers hope AGENTLESS will serve as the new baseline, refocusing future work on structured, cost-effective techniques rather than increasingly complex autonomous planning.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.