PAPERZILLA
Crunching Academic Papers into Bite-sized Insights.
About
Sign Out
← Back to papers
Physical Sciences › Computer Science › Artificial Intelligence
The Illusion of the Illusion of Thinking: A Comment on Shojaee et al. (2025)
★
★
★
☆
☆
SHARE
Overview
Paper Summary
Conflicts of Interest
Identified Weaknesses
Rating Explanation
Good to know
Topic Hierarchy
File Information
Paper Summary
Paperzilla title
LLMs Not Dumb, Just Too Wordy? Study Finds AI Can Solve Puzzles If We Don't Make Them Write Novels About It
The paper argues that a previous study's findings of "accuracy collapse" in Large Reasoning Models on complex planning puzzles are due to experimental design limitations, specifically output token limits and unsolvable problem instances. By using alternative representations that bypass these limitations, the authors suggest that models can solve tasks previously deemed too complex.
Possible Conflicts of Interest
The authors are affiliated with Anthropic and Open Philanthropy, which may have interests in promoting positive views of AI capabilities. However, the critique primarily addresses methodological concerns, reducing the likelihood of significant bias.
Identified Weaknesses
Lack of Novel Contribution
The paper primarily critiques a previous study's methodology, rather than presenting novel research. The focus is on demonstrating how experimental design flaws led to misinterpretations of LLM capabilities, specifically regarding output constraints and problem solvability.
Weak Supporting Evidence
The authors mention conducting "preliminary testing" with alternative representations, but the details are scarce. The sample size is acknowledged as insufficient for statistical significance, limiting the strength of their counter-arguments. More robust experimentation is needed to support their claims of restored performance.
Over-Reliance on Anecdotal Evidence
The paper heavily relies on anecdotal evidence, such as a tweet and model outputs expressing awareness of length limits. While illustrative, these examples don't constitute rigorous scientific proof.
Rating Explanation
The paper provides a valuable critique of experimental design in AI research, highlighting the importance of considering output constraints. However, its lack of novel findings, reliance on preliminary tests, and anecdotal evidence limits its impact.
Good to know
This is our free standard analysis. Paperzilla Pro fact-checks every citation, researches author backgrounds and funding sources, and uses advanced AI reasoning for more thorough insights.
Explore Pro →
Topic Hierarchy
File Information
Original Title:
The Illusion of the Illusion of Thinking: A Comment on Shojaee et al. (2025)
File Name:
2506.09250v1.pdf
[download]
File Size:
0.22 MB
Uploaded:
July 08, 2025 at 12:15 PM
Privacy:
🌐 Public
© 2025 Paperzilla. All rights reserved.