AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Knowledge Pull Requests Offer a Reviewable Way to Update Documents

Wikipedia articles, technical manuals, and intelligence reports are never truly finished. New facts arrive from fresh sources, other languages, or later dates, but large language models often either rewrite an entire document without showing which facts changed or regenerate it from scratch, discarding prior work. That makes review difficult and can erase useful content.

Researchers Alexander Martin and Benjamin Van Durme propose Knowledge Pull Requests (KPRs), a framework that borrows the pull-request idea from software engineering. A KPR treats a document update as a reviewable proposal. It decomposes the main document and its sources into claims—atomic, decontextualized factual statements. It then classifies source claims as already covered, absent, or conflicting; filters absent claims for relevance; and routes them to existing sections or new ones. The output is a ChangeLog: a claim proposal of what knowledge changes and a document diff of how text changes. A reviewer could approve, reject, or resolve proposed claims before merging.

The baselines are ConText, which rewrites sections from raw source text; ConClaim, which uses claims but applies no coverage, conflict, or relevance filtering; and Scratch, which regenerates the document from all sources.

The paper’s concrete conflict example comes from RAGTIME, a multilingual report-generation benchmark. In one case, sources reported a margin of victory as 2.7% in one document and 3% in another. Rather than silently choosing one value, a KPR flags the contradiction for review. In the automated experiments, flagged conflicts were withheld; deciding which source to trust remains open.

On cross-lingual Wikipedia revision, using 599 articles and 49 non-English source languages, KPRs integrated more source knowledge while preserving the English article. Its information precision was 0.878, compared with 0.853 for ConClaim and 0.837 for ConText. Its recall of added source claims was 0.891, versus 0.786 and 0.716. Retention of original claims was high for all methods, with KPR at 0.950. Multilingual QA averaged 69.2 for KPR-revised articles, compared with 58.5 for ConClaim, 48.4 for ConText, 29.5 for the original English article, and 36.9 closed-book. English QA slipped slightly: 87.7 for KPR, 88.0 for ConClaim, 87.6 for ConText, and 92.0 for the original article.

On 100 multilingual questions no dense open model answered closed-book, an OLMo-3-7B model grounded on a KPR article answered 50, while Sol-5.6 with web search answered 38 and with translated search 41; Sol-5.6 grounded on the KPR answered 62. In RAGTIME’s temporal, conflict, and balanced settings, KPRs achieved the highest recall of both round-1 and round-2 nuggets, though Scratch retained higher precision while adding much less information.

Costs remain. Wikipedia KPRs preserved 97.3% of the original and required 11.2 contiguous approvals, fewer than ConText’s 14.7 and ConClaim’s 25.3, but added 1,299 tokens versus ConClaim’s 898 and had a word edit rate of 131 versus 92. The pipeline also uses an LLM for decomposition, classification, routing, and rewriting; the authors do not measure that compute cost, and they run no human study of whether a ChangeLog helps editors. The broader promise is a shift from opaque text edits to reviewable knowledge changes, though the work demonstrates automated performance, not yet collaborative editing gains.