Large Reasoning Models (LRMs), despite self-reflection mechanisms, face accuracy collapse beyond certain puzzle complexities and exhibit counterintuitive scaling limits, reducing thinking effort as difficulty increases. Three reasoning regimes emerge: standard LLMs outperform LRMs in simple puzzles, LRMs excel in moderately complex ones, and both fail in highly complex puzzles, highlighting fundamental limitations in their generalizable reasoning capabilities.

Authors are affiliated with Apple, which has vested interests in the development and application of advanced language models. This potential COI is acknowledged in the paper.

The paper presents a well-designed controlled study on the reasoning capabilities of Large Language Models using algorithmic puzzle environments. The methodology enables systematic investigation into how complexity affects solution accuracy and the thinking process. The findings, including the identification of three distinct reasoning regimes and the counterintuitive scaling limit of thinking tokens, are valuable contributions to the field. While the puzzle environment focus introduces limitations in generalizability, the rigorous methodology, insightful analysis, and practical implications for LRM development warrant a strong rating. The potential conflict of interest with Apple affiliation is acknowledged and considered in the rating.