How Do You Test Whether an AI Actually Knows What It Doesn’t Know?
Overview
Every serious attempt to measure artificial general intelligence today asks the same kind of question: what can this system do? Can it answer hard questions across many fields? Can it work autonomously for hours? Can it match or beat human experts on difficult tasks? These are real capabilities and they are worth measuring.
Our paper, An Epistemic Autonomy Framework for AGI Evaluation, argues that all of these frameworks share a blind spot. They measure what a system currently knows or can currently do. None of them measures the mechanism by which a system extends what it knows into territory it has never seen. We call that mechanism epistemic autonomy, and we make the case that no performance benchmark can capture it, even in principle.
The Problem with Measuring Performance
A system that has memorized the right answers to a huge range of problems will score exactly the same as a system that genuinely understands those problems, as long as you only test performance. On any fixed benchmark, the two are indistinguishable. We formalize this as the static competence problem: a system can look brilliant across every domain in its training distribution and collapse to chance the moment it steps outside it, and performance measurement has no way to tell the difference.
This is not a hypothetical failure. It is what current systems do when they produce fluent, confident, and false answers without flagging that they have no evidence. The distinction we draw is between knowing a lot and being able to recognize a new kind of problem and figure out how to approach it. Those two abilities are independent. Measuring the first tells you nothing about the second.
A Framework for the Mechanism
We define epistemic autonomy as a five-phase loop. A system recognizes the limits of its own knowledge, identifies what it would need to close the gap, acquires that information on its own, updates its beliefs based on what it found, and then repeats the whole process in a genuinely new domain without being retrained in between. We call this the Epistemic Autonomy Loop, and the phases are sequential: each one depends on the one before it. Bad calibration poisons gap identification, which poisons acquisition, and so on down the chain.
What makes the framework more than a description is how each phase is pinned to something objective. Calibration is measured through behavior rather than self-report. Instead of asking a system how confident it is, our protocol puts it in a decision with real costs and rewards and watches what it commits to, when it seeks more information, and when it abstains. We designed the structure so that the only way to pass is to actually have well-calibrated beliefs, not to produce confident-sounding numbers.
The same objectivity runs through the rest of the loop. Gap identification is scored against a formal derivation of what a task genuinely requires, computed from the domain’s rules rather than an evaluator’s opinion. Belief revision is checked against priors derived from the domain’s own structure, so the ground truth never depends on what the system happened to believe going in.
Building Domains That Can’t Be Memorized
Any evaluation on real-world domains faces an obvious objection: how do you know the system didn’t just see this during training? Our answer is the Constructed Domain Protocol. These are synthetic rule systems whose vocabulary is generated fresh at evaluation time and verified absent from every indexed corpus that predates the test. Lexical contamination becomes something you can actually check and rule out.
We do not claim to have eliminated all transfer. A constructed domain that uses a familiar logical step still benefits any system trained on logic, whatever you rename the symbols. What the protocol does is bound that structural transfer, decompose it, and disclose it alongside every score. We think being honest about this limit is part of doing the measurement properly.
One Deliberate Quirk in the Score
All of this rolls up into a single number, the Epistemic Autonomy Score, built from a formula we prove is the only one satisfying five stated axioms. The score has a property that looks like a bug and is actually a design decision: it can go down when a system improves. If a system is already strong in one domain and gets even stronger there while staying weak elsewhere, the composite score drops. Our reasoning is that widening the gap between your best and worst domains is evidence against general intelligence, not for it. Consistency across genuinely different domains is the thing we are measuring, and we penalize inconsistency on purpose.
Why This Matters for Safety
We make a point of separating capability from alignment. The score measures only whether a system can manage its own knowledge and extend it. It says nothing about what goals that capability serves. We think this makes the measurement directly relevant to safety rather than orthogonal to it. A system that knows exactly what it lacks, knows how to get it, and updates cleanly, all in service of misaligned goals, is more dangerous than a system with the same goals and no such capability. For that reason the framework requires every published score to either include an alignment assessment or state plainly that none was done.
Conclusion
A high score does not prove a system is AGI. Epistemic autonomy is a necessary condition, not a sufficient one. The human reference points are predictions, not measurements, and the whole framework rests on validation studies we have not yet run. What we are offering is a falsifiable protocol for something the field currently has no way to test: the mechanism that produces new knowledge rather than retrieving stored knowledge.
The argument lands in a single line near the end of the paper. A system that reaches the profile of an educated adult without the mechanism by which an adult became educated is not generally intelligent. This framework is our attempt to measure that mechanism directly.
The full paper is available below.