New Research Reveals LLM Query Energy Significantly Lower Than Widely Believed, Up to 20x Efficiency Gains Possible

New research, titled "Energy Use of AI Inference: Efficiency Pathways and Test-Time Compute" and highlighted by Rohan Paul, challenges common perceptions regarding the energy consumption of large language model (LLM) queries. The paper, available on arXiv, indicates that a typical frontier model query in real deployments uses approximately 0.34 Watt-hours (Wh). This figure stands in stark contrast to many public numbers, which Paul noted are often "4x to 20x too high because of tiny tests and unclear scopes."

The study's methodology for estimating energy consumption involves analyzing token speed, NVIDIA H100 server power, and data center overhead. Researchers found that the decoding process, where the model generates tokens sequentially, accounts for the majority of the energy expenditure during an LLM query. However, for tasks requiring very long outputs, such as complex reasoning or agent runs, the median query energy can increase to about 4.32 Wh.

Significant opportunities for energy reduction exist through both model and serving optimizations. Model changes, including techniques like distillation, low-bit formats, mixture of experts (MoE), and faster attention mechanisms, are projected to yield energy savings ranging from 1.5x to 10x. These advancements focus on making the core LLM operations more energy-efficient.

Further efficiency improvements can be achieved through serving-side strategies, such as split prefill and decode, speculative decoding, KV cache tuning, and intelligent routing, which are estimated to save 1.5x to 5x energy. Additionally, continuous advancements in GPU hardware are expected to contribute 1.5x to 2.5x in energy efficiency. These combined efforts provide a multi-faceted approach to sustainable AI.

Collectively, the research suggests that implementing these model, serving, and hardware enhancements could reduce the energy required for LLM inference by an impressive 8x to 20x. This substantial potential for energy conservation is achievable even as the complexity and length of LLM outputs continue to grow, offering a clear pathway for more sustainable AI development and deployment.