Recent research highlights the persistent challenge of repetitive copying in long-context language models, revealing that even advanced models struggle with differentiating between relevant and irrelevant information. This study demonstrates that while models can effectively process large amounts of text, they often fall into the habit of copying unnecessary filler, which can detract from their accuracy. To address this issue, researchers implemented a simple evidence-aware reinforcement learning reward system that encourages models to focus on significant portions of text, resulting in accuracy improvements of up to 4.6 points. This aligns with a growing trend in evaluating language models to ensure they ground their responses in meaningful evidence rather than superficial copying.

Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning: “Copy Less, Ground More” is a research paper on long-context language models that analyzes how these models often waste their internal reasoning by repetitively copying large portions of their input instead of focusing on key evidence. The paper proposes an evidence‑aware reinforcement learning reward that encourages models to engage with the small subset of relevant text and discourages copying irrelevant filler, thereby improving accuracy in long‑context reasoning tasks.

EvaluationTrend: There is a growing trend of benchmark design and evaluation methods that explicitly test whether language models ground their answers in appropriate evidence rather than superficial copying or pattern matching over lengthy inputs.
EvidenceAwareRL: Evidence‑aware reinforcement learning approaches have gained attention as a way to align model reasoning with human‑identified evidence, by rewarding attention to relevant passages and penalizing reliance on distractor content in complex inputs.
LongContextChallenge: Recent work on long‑context transformers has highlighted that simply extending context windows is not enough, as models often struggle to identify and prioritize the few salient spans of text that matter for a given question.