The post has been translated automatically. Original language: English
Do LLMs have the ability to control attention and ignore unnecessary information while focusing on what is needed?
Evaluating the cognitive ability of attention control was the primary goal of our custom benchmark, developed as part of the hackathon "Measuring Progress Toward AGI - Cognitive Abilities" hosted by Google DeepMind and Kaggle.
While most AI benchmarks evaluate models based on "accuracy" or "recall" within clean contexts, true intelligence operates in noisy, chaotic environments. This benchmark focuses on the Attention track - specifically measuring a model's "Attention Control".
To evaluate whether a model can ignore distractors without losing the thread of the main instruction, we employ a multi-layered assessment logic designed to be easy for humans but complex for LLMs. To prevent data contamination and memorization, the benchmark relies on real-time distractors (taken from newsapi.org).
The benchmark is based on 18 tasks (each consisting of subtasks 0.5-0.25-0.25, total score 1.0 per task), and we have tested 16 top LLM models.
Results:
- The leaderboard reveals a contrast in cognitive resilience among frontier models. The Gemini family demonstrated the highest executive control, taking the top three spots, with gemini-2.5-pro leading at a mean score of 0.722, followed closely by gemini-3.1-pro-preview (0.694). Notably, glm-5 (0.611) and the open-weight gemma-4 variants outperformed several major proprietary models. Conversely, highly capable models like claude-opus-4-6-default (0.0 score) and gpt-5.4-mini-2026-03-17 (0.125 score) suffered near-complete systemic failure when confronted with real-time news distractors.
🏆 Top-5 Models (by Mean Score):
1. gemini-2.5-pro - 0.722
2. gemini-3.1-pro-preview - 0.694
3. gemini-2.5-flash - 0.639
4. glm-5 - 0.611
5. gemini-3-flash-preview - 0.556
⚠️ Worst-5 Models (by Mean Score):
1. claude-opus-4-6-default - 0.000
2. gpt-5.4-nano-2026-03-17 - 0.042
3. deepseek-v3.2 - 0.056
4. gemini-3.1-flash-lite-preview - 0.069
5. claude-sonnet-4-6-default - 0.083
Conclusion
The findings demonstrate that scaling context windows alone does not solve the problem of selective focus. Instead, effective reasoning in real-world scenarios requires the ability to dynamically suppress or prune irrelevant information.
Kaggle Writeup: https://www.kaggle.com/competitions/kaggle-measuring-agi/writeups/distraction-agnostic-benchmark
Kaggle Benchmark: https://www.kaggle.com/benchmarks/armanzhalgasbayev/distraction-agnostic-bench
Do LLMs have the ability to control attention and ignore unnecessary information while focusing on what is needed?
Evaluating the cognitive ability of attention control was the primary goal of our custom benchmark, developed as part of the hackathon "Measuring Progress Toward AGI - Cognitive Abilities" hosted by Google DeepMind and Kaggle.
While most AI benchmarks evaluate models based on "accuracy" or "recall" within clean contexts, true intelligence operates in noisy, chaotic environments. This benchmark focuses on the Attention track - specifically measuring a model's "Attention Control".
To evaluate whether a model can ignore distractors without losing the thread of the main instruction, we employ a multi-layered assessment logic designed to be easy for humans but complex for LLMs. To prevent data contamination and memorization, the benchmark relies on real-time distractors (taken from newsapi.org).
The benchmark is based on 18 tasks (each consisting of subtasks 0.5-0.25-0.25, total score 1.0 per task), and we have tested 16 top LLM models.
Results:
- The leaderboard reveals a contrast in cognitive resilience among frontier models. The Gemini family demonstrated the highest executive control, taking the top three spots, with gemini-2.5-pro leading at a mean score of 0.722, followed closely by gemini-3.1-pro-preview (0.694). Notably, glm-5 (0.611) and the open-weight gemma-4 variants outperformed several major proprietary models. Conversely, highly capable models like claude-opus-4-6-default (0.0 score) and gpt-5.4-mini-2026-03-17 (0.125 score) suffered near-complete systemic failure when confronted with real-time news distractors.
🏆 Top-5 Models (by Mean Score):
1. gemini-2.5-pro - 0.722
2. gemini-3.1-pro-preview - 0.694
3. gemini-2.5-flash - 0.639
4. glm-5 - 0.611
5. gemini-3-flash-preview - 0.556
⚠️ Worst-5 Models (by Mean Score):
1. claude-opus-4-6-default - 0.000
2. gpt-5.4-nano-2026-03-17 - 0.042
3. deepseek-v3.2 - 0.056
4. gemini-3.1-flash-lite-preview - 0.069
5. claude-sonnet-4-6-default - 0.083
Conclusion
The findings demonstrate that scaling context windows alone does not solve the problem of selective focus. Instead, effective reasoning in real-world scenarios requires the ability to dynamically suppress or prune irrelevant information.
Kaggle Writeup: https://www.kaggle.com/competitions/kaggle-measuring-agi/writeups/distraction-agnostic-benchmark
Kaggle Benchmark: https://www.kaggle.com/benchmarks/armanzhalgasbayev/distraction-agnostic-bench
Share
