AI Policy Expert Challenges Prevailing AGI Narratives, Advocates for Nuanced Governance

Séb Krier, a Policy Development & Strategy Manager at Google DeepMind and a prominent AI policy researcher, has publicly articulated a critical perspective on the current discourse surrounding Artificial General Intelligence (AGI). Krier, formerly with the UK Government’s Office for Artificial Intelligence and Stanford University’s Cyber Policy Center, suggests that the prevailing ontology for discussing and conceptualizing AGI is fundamentally flawed. He anticipates future reflection will reveal the primitiveness of today's ideas.

In a recent social media post, Krier stated, "I think our entire ontology for how we talk about and conceptualise A[G]I is confused." He elaborated that the tendency to reify future AGI systems as "self-sovereign entities with their own goals and incentives" constitutes a "category error." While acknowledging this view is not impossible, Krier expressed growing uncertainty, emphasizing that agents could remain tools rather than evolving into distinct species.

Krier also criticized the abstract nature of some AGI and ASI claims, which he finds difficult to parse or falsify, likening it to vague statements about "Finance." He argued that such broad generalizations hinder the development of actionable prescriptions, leading to generic calls for balancing risks and opportunities. Instead, Krier posits that AGI will likely manifest as "a distributed ecosystem of different models, built by different companies and state actors," each with varied capabilities and incentive structures.

Addressing the challenge of AI alignment, Krier suggested that viewing it as a one-off "fix" is misguided. He drew parallels to "solving truth" or "solving conflict," asserting that alignment is intrinsically about governance, politics, and power dynamics. Krier concluded that it will be "a messy, perpetual process of negotiation, regulation, and adaptation, much like law, democracy, or international relations," underscoring a perceived "dearth of imagination" in current governance approaches compared to the ambitious goals for AI's capabilities.