A new way to understand how language models use context

📄 Full Paper

💬 Ask

Language models are becoming increasingly powerful, but they are also becoming increasingly opaque. It’s often difficult to understand how they generate their responses, especially when those responses are based on a large amount of text.

A new research paper, “CONTEXTCITE: Attributing Model Generation to Context,” tackles this problem by introducing a new method called “context attribution.” Context attribution aims to pinpoint which parts of a context (if any) are responsible for a specific statement generated by a language model.

Imagine asking a language model to tell you the weather in Antarctica in January. The model might give you a response based on a Wikipedia article about the climate of Antarctica. How can we know for sure that the model actually used the article to come up with its answer? That’s where context attribution comes in.

CONTEXTCITE uses a technique called “surrogate modeling” to learn how a language model’s response changes when different parts of the context are removed. The paper’s authors demonstrate that CONTEXTCITE can effectively identify the parts of the context that are most responsible for a specific statement, even in complex situations involving a large amount of text.

The authors showcase three key applications of CONTEXTCITE:

• Helping verify generated statements: CONTEXTCITE can be used to help verify whether a statement generated by a language model is actually grounded in the provided context. For example, if a language model misinterprets a fact in the context and generates an inaccurate statement, CONTEXTCITE will pinpoint the misinterpreted part of the context. This can help users to better understand when to trust a language model’s responses.
• Improving response quality by pruning the context: Language models often struggle to correctly use information within long contexts. By identifying the most relevant parts of the context, CONTEXTCITE can be used to prune the context, leading to more accurate and concise responses. This is particularly useful in tasks like question answering, where a language model needs to extract information from a large amount of text.
• Detecting poisoning attacks: Language models can be vulnerable to poisoning attacks, where an attacker adversarially modifies the context to control the model’s response. CONTEXTCITE can be used to identify these attacks by pinpointing the parts of the context that are most responsible for the model’s output.

The authors of the CONTEXTCITE paper argue that their method provides a valuable tool for understanding and improving language models. By helping users to better understand how language models use context, CONTEXTCITE can help to build trust in these powerful technologies and ensure that they are used responsibly.

The code for CONTEXTCITE is available on GitHub: https://github.com/MadryLab/context-cite

Chat about this paper

To chat about this paper, you'll need a free Gemini API key from Google AI Studio.

Your API key will be stored securely in your browser's local storage.