Benchmark Leakage Inflates Scores

Benchmark leakage turns a test of reasoning into a test of recall. MMLU and GSM8K were published openly and became common reference points for model training and post training, so models could encounter the exact questions or close variants before evaluation. That inflates scores because the model is no longer solving a fresh problem, it is pattern matching against something it has already seen, which is why live systems like Arena matter more for measuring real user facing quality.

• GSM8K was released publicly in 2021 as a training and evaluation dataset. Once a benchmark lives on the open internet, it can flow into web scrapes, synthetic data generation, fine tuning sets, and RL pipelines, even if no one intended to train on the test split directly.
  1 openai 4 openreview
• Researchers have since built contamination focused follow ons like MMLU-CF because standard MMLU had become too exposed. The point of a closed test set is simple, if the questions are public for long enough, leaderboard gains can partly reflect familiarity with the exam instead of deeper capability.
  2 arxiv
• Arena avoids this failure mode by grading models on live prompts that arrive after training. Users type whatever they want, two anonymous models answer, and votes produce rankings from novel interactions rather than from a fixed worksheet that labs can optimize against in advance.
  3 sacra 5 lmarena

The next step in LLM evaluation is moving away from static public exams toward constantly refreshed, harder to game tests. As labs rely more on post training and synthetic data, contamination pressure rises, which makes live preference data and closed or rotating benchmarks more central to how model quality will be measured.