AI Advancement Correlates with Five Key Task Verifiability Properties

AI researcher Jason Wei has unveiled a new blog post delving into the "asymmetry of verification" and "verifier's law," concepts increasingly vital as reinforcement learning (RL) capabilities expand. Wei posits that the ease with which AI can be trained to solve a task is directly proportional to how verifiable that task is, marking a significant insight into the future trajectory of artificial intelligence development.

Asymmetry of verification describes tasks that are considerably easier to verify than to solve. Wei highlighted examples such as Sudoku puzzles, writing code for a website like Instagram, and OpenAI's BrowseComp problems, which are challenging to complete but simple to check once an answer is proposed. Conversely, some tasks exhibit near-symmetry, like summing two 900-digit numbers, while others, such as fact-checking a lengthy essay, are much harder to verify than to generate.

"Verifier's law states that the ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI," Wei explained in his blog post.

The ability to train AI to solve a task, according to Wei, hinges on five critical properties: objective truth, fast verification, scalable verification, low noise in verification, and continuous reward for solution quality. He noted that most popular AI benchmarks over the past decade adhere to these criteria, underscoring their importance in driving AI progress.

The profound significance of verifiability lies in its ability to maximize the amount of learning in AI systems. When these criteria are met, AI can undertake numerous gradient steps, each providing a strong signal for improvement, thereby accelerating iteration speed. Google's AlphaEvolve is cited as a prime example of leveraging this asymmetry, focusing on problems where the training set is equivalent to the test set, leading to breakthroughs in mathematics and other fields.

Wei's insights suggest a future of "jagged intelligence," where AI excels disproportionately at tasks that are easily measurable and verifiable. This paradigm shift emphasizes that anything that can be reliably measured will eventually be within AI's problem-solving grasp, reshaping how AI capabilities evolve across various domains.