The Next AI Frontier: Systems That Learn Like Our Brains, Fast, Slow and Continuously

In 2017, eight Google engineers wrote an article called “Attention is all you need”, which would have far-reaching consequences. Applied to machine translation, it argued that a neural network could learn by “paying attention” to different parts of a sequence by expanding the context window. This apparently “inoffensive” article led to the creation of transformers, the architecture that today supports ChatGPT, Claude, Gemini and practically all modern generative AI.

Eight years on, the same company, (different engineers; Google has never been good at retaining talent) has come up with another potentially disruptive idea. “Nested Learning: the illusion of deep learning architectures”, is the paper you need to be reading and understanding now, and it starts with a provocative claim: what we call deep learning, might not be that deep at all.

The authors argue that neural networks do not learn because they are multi-layered, but because each of these layers, and the algorithms that train them, operate at different speeds. In reality, what we have is not a stack of transformations, but a nested learning system: processes that adjust to each other on different time scales, as if some parts of the model are thinking faster and others more slowly.

The result is that what we saw as “depth” was an illusion: a consequence of different learning frequencies interacting with each other. If this hypothesis is correct, the future of AI might not be about adding more layers or parameters, but about designing systems that learn at several rates at once.

In addition, Google highlights something crucial: the paradigm of nested learning not only redefines architecture, it addresses head-on one of the great Achilles’ heels of current models: catastrophic forgetting. The article argues that traditional models treat architecture and optimization algorithm as two separate pieces, instead recommending merging both into a single optimization hierarchy. This fusion gives rise to a “continuous” memory system where different modules are updated at different rates (i.e., fast, medium and slow learning coexisting simultaneously). The paper defines the so-called “HOPE architecture,” which is both an engineering experiment and a manifesto: if we want machines to learn more like us, not just train once and freeze, but learn, forget, relearn, adapt, then we have to design for multiple cadences of change, not just for deeper layers.

Another interesting insight from the paper is that current models are trapped in an “eternal present”: they process input with fixed weights, can adapt minimally within the context window, and then forget everything. Learning happens before the model is deployed, and from there, it hardly changes. The nested learning approach proposes to break this limitation by providing the system with several learning rhythms: some modules that react immediately, others that adjust in the medium term and others that evolve more slowly. This overlapping speed would allow for a response to the stimulus of the moment, as well as building a lasting memory and modifying its behavior over time, a capacity much closer to the way humans learn.

That’s where the comparison with the human brain becomes inevitable: our nervous system also works in overlapping layers of time: reflexes that react in milliseconds, learning consolidated during sleep, habits that are formed after months of repetition, and personality traits that change slowly over the years. Intelligence, ultimately, may not be a matter of size or depth, but of pace: of how fast and slow learning processes are integrated into the same structure.

Google seems to think it’s on to a winner. Just as transformers redefined the notion of “attention,” nested learning could redefine the very notion of architecture. If algorithms and optimizers stop being separate pieces and become a living system that continuously modifies itself, we will be much closer to a model that learns like us: not retraining each time from scratch, but continuously readjusting its memory and behavior.

Of course, this idea is still in its infancy. The paper’s proofs are conceptual, and it remains to be seen whether the idea scales up to today’s gigantic models. But history teaches us that when Google launches a theory about how a machine should learn, it pays attention to it: the last time this happened, it changed the course of the entire industry.

If nested learning works out, we could be facing a new paradigm shift whereby AI stops looking like a calculating machine, and starts looking more like an evolving brain.

—

This post was previously published on Enrique Dans’ blog.

***