The "System 2" AI blueprint is still a blueprint, five years on

A proposal, filed once, never revised

On 5 October 2021, a large group of researchers posted a paper to arXiv titled Thinking Fast and Slow in AI: the Role of Metacognition (arXiv:2110.01834). Its authors include Marianna Bergamaschi Ganapini, Murray Campbell, Francesco Fabiano, Lior Horesh, Jon Lenchner, Andrea Loreggia, Nicholas Mattei, Francesca Rossi, Biplav Srivastava and Kristen Brent Venable. The abstract does one thing clearly: it argues and it proposes. It does not report a benchmark, a system, or an evaluation. The arXiv record shows a single version, v1, a 560 KB submission, and no subsequent revision.

That matters because the vocabulary in this paper — fast agents, slow agents, deliberate activation of reasoning — has since become the default marketing language for a generation of AI products. The paper it is often traced back to is a position paper. Worth reading on its own terms; not evidence that anyone shipped the thing.

What the architecture actually says

The starting diagnosis is straightforward. The authors write that despite dramatic advances, we are "still mostly seeing instances of narrow AI": systems built for a limited set of competencies such as image interpretation, natural language processing, classification and prediction. They also make a point that has aged well — that these successes are tied not only to better algorithms but to the availability of huge datasets and computational power. State of the art AI, they argue, still lacks capabilities a person would naturally include in the idea of intelligence.

Their proposed fix borrows Daniel Kahneman's distinction between two modes of human thought. System 1 is fast, automatic, pattern-driven — recognising a face, catching a dropped glass. System 2 is slow, effortful and deliberate — doing long division, planning a route through an unfamiliar city.

The paper maps this onto a multi-agent architecture. Incoming problems go to either "system 1" agents, which react by exploiting only past experience, or "system 2" agents, which are, in the authors' words, "deliberately activated when there is the need to reason and search for optimal solutions beyond what is expected from the system 1 agent."

Two shared resources sit underneath both kinds of agent. A model of the world holds domain knowledge about the environment. A model of "self" holds information about the system's own past actions and about the skills of its solvers. That second component is the interesting one, and it is where the word metacognition earns its place: a system that keeps a record of which of its own components are good at what can, in principle, decide when its cheap reflex is out of its depth.

The hard part the abstract does not resolve

Routing is the whole game, and it is genuinely difficult. To know that a fast answer is inadequate, something must estimate the quality of an answer it has not yet computed the alternative to. Get that wrong in one direction and you burn compute deliberating over trivia. Get it wrong in the other and the system confidently reflexes its way into a bad decision — the failure mode anyone running a model in production already recognises.

The abstract states the criterion at the level of intent, not mechanism. It says slow agents are activated when there is a need to reason beyond what is expected of the fast agent. How that expectation is formed, measured, calibrated or audited is not something the abstract commits to. Treat any product that claims to have solved this as making a claim, and ask for the numbers.

Questions You Should Be Asking

• When a vendor says their system "thinks slowly when needed," what specific signal triggers the switch — and can you see the log of when it fired and when it did not?
• What does the system believe about its own competence, and who wrote that belief? A "model of self" is a configuration artefact; if a supplier populated it, it encodes their assumptions about their own tool.
• What is the cost profile of a wrong routing decision in your workflow — wasted compute, or an unreviewed decision that reaches a customer?
• Is the architecture you are being sold evaluated against a baseline, or described against an analogy to human cognition? Those are different kinds of argument.
• If a fast path answers wrongly, does anything in the system detect this afterwards and update — or does the error simply become part of "past experience"?

What To Watch Next

The signal to watch is not more papers invoking Kahneman. It is whether any deployed system publishes its routing decisions as an auditable trace — a record of which requests were escalated to deliberate reasoning, which were not, and how often that choice turned out to be wrong. Until that exists, the metacognitive layer is something you are asked to trust rather than something you can inspect.