Large Language Models (LLMs) exhibit significantly lower performance in multi-turn conversations compared to single-turn interactions, primarily due to a substantial increase in unreliability rather than a loss in aptitude. This "lost in conversation" phenomenon stems from LLMs making early assumptions, prematurely proposing solutions, and struggling to incorporate new information effectively. The study employed simulated conversations across six diverse generation tasks, revealing consistent performance degradation across various LLMs, regardless of size or reasoning capabilities.

Authors are employed by Microsoft Research and Salesforce Research, organizations with vested interest in LLM development and performance.

This paper presents a comprehensive and large-scale study highlighting a critical issue in current LLM performance: unreliability in multi-turn conversations. The methodology is sound, and the findings are significant and potentially impactful for future LLM development. The limitations regarding simulated conversations and task scope are acknowledged.