AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Embeddings Versus Grep: How Much Does the Dense Search Advantage Really Measure?

Code-search benchmarks consistently show neural embedding models, which represent text as vectors, far ahead of BM25, a keyword-ranking method, yet coding agents often perform well using plain grep. A study from Ant Group asks what the embedding advantage actually measures. Its answer is that much of the gap comes from how comparisons are set up, and that the remaining advantage is concentrated in queries that do not name the code’s distinctive identifiers.

The researchers built REPOPREMIUM, a benchmark in which a system must rank a repository’s files so that those edited by a described change appear near the top. The core set has 1,265 queries across ten Python repositories, drawn from curated SWE-bench Verified issue statements and from raw pull-request text in three date windows. They measure the “dense premium,” the gap in nDCG@10, a score that rewards placing relevant results in the top ten, between a dense retriever and BM25.

Evaluation choices matter considerably. When BM25 indexes files in the same 1,000-character chunks the encoder uses, the curated-query premium is +0.121. Indexing whole files shrinks it to +0.034. Counting edited test files as relevant raises the tuned premium on curated queries to +0.165, a choice the authors say favors dense retrieval because encoders rank test files highly.

The larger finding concerns wording. The team had an LLM, qwen35-122B, read each query and draft likely identifiers, file hints and API names, a method they call generate-then-grep. With this expansion, whole-file BM25 scored 0.613 on curated queries, against 0.532 for the canonical encoder, jina-code-embeddings-0.5b. The authors attribute this to the expansion supplying missing code vocabulary. The added terms raised BM25 by 0.12 to 0.21 nDCG@10, but raised the encoder by only 0.03 to 0.07.

The paper illustrates the failure mode with a pandas pull request. A curated rewrite describing the bug in code terms let BM25 rank the gold file first, while dense retrieval placed it seventh. The raw PR title, however, contained the words “DOC” and “docs,” which pulled BM25 toward documentation files; the gold file fell to eighth. Dense retrieval ranked it first.

The researchers also tested combining methods. Fusing dense retrieval with BM25 improved nDCG@10 on a held-out “lockbox” set of 539 queries to 0.551, against 0.533 for dense retrieval alone. A per-query router that chose between retrievers did not reliably beat fusion, and no classifier or LLM judge they tried could predict which retriever would win.

In agent experiments on 96 tasks, the choice of search tool changed exact-match localization scores by roughly 0.04, while the choice of language model moved results far more.

The study has clear limits. The core covers only ten Python repositories, BM25’s settings were tuned on the evaluation queries, and the generator’s output varies by model. Expansion also adds an LLM call for every query, a cost the authors suggest is most justified for frequently searched repositories. Labels reflect the files one change edited, and the paper does not test function- or line-level targets.

The work suggests that reported dense-retrieval gains may partly reflect benchmark design and query wording rather than a fixed property of embeddings. Its guidance, which includes fair lexical baselines and fusion, is practical, though the evidence comes from one benchmark and a specific set of models.