AI Papers Reader

Personalized digests of latest AI research

View on GitHub

LLMs Learn to Follow Instructions Better with "Treasure Markers"

Large language models (LLMs) are becoming increasingly powerful, but they often struggle with less common or “long-tail” tasks that appear infrequently in their training data. To address this, researchers at Cohere Labs have developed a novel technique called “Treasure Markers,” which significantly improves LLM performance and controllability, particularly for these underrepresented tasks.

The core idea behind Treasure Markers is to enrich the training data with detailed, task-specific metadata, or “markers.” These markers act like a treasure map, guiding the LLM during training to understand nuanced aspects of the data, such as the desired output format, length, quality, or even the domain of knowledge. For example, a marker could specify that a response should be “concise” and adhere to a “code generation” task.

During the training process, the LLM learns to associate these markers with specific types of outputs. Crucially, the model is also trained to infer these markers automatically, even if they are not explicitly provided at inference time. This approach allows for greater flexibility, as users don’t need to be experts in prompt engineering to guide the model.

The results are impressive. The researchers found that LLMs trained with Treasure Markers showed an average increase of 5.7% in “win rates” (a measure of output quality judged by another model) across various tasks. More significantly, performance on underrepresented domains saw a jump of over 9.1%. For instance, in the challenging “CodeRepair” task, which is particularly rare, there was a relative improvement of 14.1%.

The markers also provide fine-grained control. In length-specific instruction following, the violation rate (meaning the model didn’t adhere to the length constraint) dropped from a substantial 36.58% to a mere 1.25% when markers were used. This improved instruction following also led to a boost in overall generation quality.

A key aspect of the Treasure Marker framework is its flexibility. Markers can be automatically inferred by the model during inference, or they can be explicitly provided by users through prompts. This “on-the-fly” annotation, where another LLM helps add relevant markers to a prompt, further demonstrated significant improvements in reducing instruction violations and enhancing win rates.

In essence, Treasure Markers equip LLMs with a richer understanding of the data and user intent, enabling them to navigate the complexities of diverse tasks more effectively. This research offers a promising path towards more robust, controllable, and universally capable AI systems, particularly by empowering them to excel even on the most obscure corners of the data landscape.