LLM Performance Drops by 39% in Multi-Turn Conversations with Tool Use, Researcher Warns

AI researcher Tahmid Tapadar has highlighted a significant challenge in the evaluation of large language models (LLMs), noting that "The space of valid tool sequences grows fast as conversations get longer, making these traditional evals even more fragile." This observation points to a critical issue impacting the reliability and effectiveness of AI agents in complex, multi-step interactions. The expanding complexity of tool-using LLMs in extended dialogues is revealing limitations in current assessment methodologies.

Recent academic research corroborates Tapadar's concern, with a 2025 paper titled "LLMs Get Lost In Multi-Turn Conversation" by Laban et al. reporting an average performance degradation of 39% for LLMs in multi-turn, underspecified conversations compared to single-turn, fully-specified interactions. This substantial drop is primarily attributed to increased model unreliability rather than a decrease in inherent aptitude. Issues such as premature answer attempts, "answer bloat" where responses become unnecessarily long, and a "loss-in-middle-turns" phenomenon contribute to this fragility, as models struggle to maintain context and adapt to evolving user needs.

The growing use of LLMs as agents capable of employing various tools, from web search to API calls, further complicates evaluation. A 2025 survey on "Evaluating LLM-based Agents for Multi-Turn Conversations" emphasizes the need for robust methods to assess dialogue coherence, context maintenance, and effective tool utilization over extended interactions. Current evaluation datasets often lack comprehensive coverage for these complex scenarios, posing a hurdle for developers aiming to build reliable AI systems.

In response to these challenges, industry players are actively developing solutions. Amazon Bedrock, for instance, has introduced a multi-turn conversation agent node that allows AI flows to intelligently pause, request clarification, and seamlessly resume interactions. This capability aims to create more interactive and context-aware experiences, validating the real-world impact of the issues raised by researchers like Tapadar. The ongoing evolution of LLMs necessitates a parallel advancement in evaluation techniques to ensure these powerful AI systems can consistently perform in dynamic conversational environments.