AI in math reveals a hard verification problem
A thread on AI in mathematics surfaced a clean statement of a problem that applies well beyond math: AI can produce a convincing-looking proof with a subtle flaw, and you need to be at Terence Tao's level to catch it. The article asked whether AI would be a tool, collaborator, or oracle. The HN crowd answered by pointing out that 'oracle' is dangerous precisely because the error modes are not obvious.
This is a specific instance of a broader reliability question that keeps surfacing across AI threads. The math case is just unusually stark because correctness is binary in formal proofs. The same dynamic applies to legal reasoning, financial modeling, or any domain where confident-sounding wrong answers are worse than no answer.
There is no clean resolution in the thread. People agree the problem is real and disagree about whether it is a temporary limitation or a structural one.
So what?
For founders building AI into high-stakes workflows, the verification gap is a product design problem, not just a model problem. If your users cannot independently verify outputs, you need to build verification tooling or constrain the scope of what the model is allowed to decide. Skipping this step is how you get liability.