ChatGPT demonstrated performance equivalent to a passing score for a third-year medical student on USMLE Step 1 and Step 2 practice questions, exceeding the accuracy of other large language models like GPT-3 and InstructGPT. While demonstrating logical reasoning in all responses and using internal information effectively, the model's reliance on external information was stronger for correct answers, highlighting a potential link between knowledge access and performance.

The authors acknowledge funding from the Yale School of Medicine and the National Institutes of Health, but declare no specific conflicts of interest.

This study provides a valuable early assessment of a large language model's capabilities in a critical domain, demonstrating promising results while acknowledging limitations. The methodology is sound, though constrained by the model's closed nature. No obvious attempts to manipulate the rating were detected.