Large Reasoning Models (LRMs) fail to develop generalizable problem-solving capabilities in complex puzzle environments, eventually reaching zero accuracy beyond certain complexity thresholds. They also exhibit a counterintuitive behavior, reducing their reasoning effort (thinking tokens) as problem complexity increases despite having available compute budget, suggesting inherent scaling limitations.

The authors work at Apple, which may have interests in LLM development, but the study does not directly evaluate Apple's models, minimizing the direct COI.

This is a well-designed study that provides interesting insights into LRM limitations. The controlled experiments and detailed trace analysis offer valuable data. While the scope is limited to puzzle environments, the findings on scaling limitations and reasoning patterns have broader relevance. Minor limitations related to reliance on closed-source LLMs and limited evaluation metrics prevent a top rating.