Artificial Intelligence

Mathematicians Test AI With Contamination-Free Problems

Has AI achieved genuine mathematical breakthroughs or are these claims premature without rigorous verification?

Published FEB 16

Mathematicians Test AI With Contamination-Free Problems

Story

Image credit: Omar Marques/SOPA Images/LightRocket/Getty Images

The Facts

A group of 11 mathematicians launched the First Proof initiative on Feb. 5, presenting 10 mathematical problems drawn from their unpublished research across fields including stochastic analysis, spectral graph theory, symplectic geometry and algebraic topology.
The New York Times
United Nations University
PHYS
The problems were designed to avoid data contamination by ensuring none had been posted online, with encrypted solutions scheduled for release on Feb. 13, giving AI systems one week to attempt solutions without access to answers in their training data.
The New York Times
United Nations University
arXiv
Preliminary testing using GPT-5.2 Pro and Gemini 3.0 Deep Think showed that current AI systems struggled to solve many problems correctly on a single attempt, without iterative refinement or human assistance.
The New York Times
Scientific American
PHYS
OpenAI ran its internal model throughout the challenge week and posted results on Feb. 14, claiming six of 10 solutions had a high chance of being correct, though mathematicians identified potential holes in at least one of those submissions.
Openai
Scientific American
NewsIQ
The First Proof team found that only two AI generated proofs were correct in their preliminary tests, with one nearly identical to an existing proof. The first problem was contaminated by an archived sketch from author Martin Hairer's website.
The New York Times
Scientific American
arXiv
The mathematicians plan to release a second set of questions within months and aim to develop a more formal benchmark — potentially testing frontier AI systems before public release while maintaining contamination-free properties.
Scientific American
United Nations University
arXiv

Sources Split

LEFT

CENTER

RIGHT

ESTAB
CRIT

CENTER

PRO
ESTAB

The Spin

Techno-optimist narrative

AI has crossed a genuine threshold by producing new mathematical knowledge, even if modest. This milestone deserves serious recognition as frontier research evaluation, with models solving multiple expert-level problems and demonstrating real capability advancement.

Jakub Pachocki on X

Sam Altman on X

Techno-skeptic narrative

Celebrating AI math breakthroughs remains premature without rigorous verification and transparency. Assisted solutions with human supervision don't prove autonomous reasoning, and vague methodologies demand skepticism until peer review confirms these claims actually highlight independent problem-solving.

Aakash Gupta on X