Recursion Is The Next Scaling Law In AI

Current large language models (LLMs) are primarily feed-forward systems that process inputs in a one-shot manner. While they can perform 'chain of thought' or use tools, these are 'hacks' that simulate recursion in the discrete token space. This limits their ability to solve incompressible problems (like sorting or Sudoku) from first principles or discover new algorithms, as their reasoning is bounded by the human knowledge they were trained on.

The guest explains that LLMs cannot map an unsorted list to a sorted list in a one-shot basis due to theoretical lower bounds on comparison sorts (n log n steps). If a transformer has 30 layers and a list is 31 elements long, it runs out of layers to do the necessary comparisons. Sudoku and mazes are cited as other incompressible problems. Chain of thought and tool use rely on existing human knowledge of solutions.

Hierarchical Reasoning Models (HRM) leverage a bio-inspired architecture with three levels of recursion (low-level, high-level, and outer refinement steps) to achieve 'compute depth.' Crucially, HRM uses a Deep Equilibrium (DEQ) learning method that performs fixed-point iteration during training. Instead of backpropagating through all recursion steps, it stops gradients and reuses the updated hidden states for subsequent iterations, effectively creating 'mini-batches' from different memory states to circumvent the vanishing/exploding gradient problems of traditional RNNs.

HRM is described as directly in the lineage of RNNs, inspired by brain regions operating at different frequencies (00:07:47). It involves TL steps with a low-level module (LNET), TH steps with a high-level module (HNET), and N_sub outer refinement steps (00:08:47). The key trick is using DEQ's fixed-point iteration, where gradients are stopped, and the same batch is re-passed with updated hidden states 16 times, treating each pass as a new 'batch' in latent space (00:11:51). This model achieved state-of-the-art on ARC prize 1 and 2 with only 27 million parameters, trained on just 1000 tasks (00:09:52).

Tiny Recursive Models (TRM) build upon HRM by simplifying the architecture and optimizing the backpropagation process. TRM collapses the two separate networks (LNET and HNET) into a single, weight-shared network and uses only one transformer layer instead of four. Its key training innovation is backpropagating through one full latent recursion step, which is more extensive than HRM's truncated backprop (t=1). This simplification and optimized training allow TRM to achieve even higher performance (87% on ARC prize 1) with a significantly smaller model (7 million parameters), demonstrating that compute depth through recursion is a powerful scaling law.

TRM simplifies HRM by collapsing LNET and HNET into a single 'net' and using just one transformer layer (00:31:29). Alexia's work shows that going deeper didn't help, and on some tasks like Sudoku, an MLP even outperformed attention (00:31:42). TRM's optimization involves backpropagating through one full latent recursion step after a detach operation (00:32:11). This 7-million-parameter model gets from 70% (HRM) to 87% on ARC prize 1 (00:33:33).

The most promising direction for AI research is to integrate the strengths of both large language models and tiny recursive models. Large LLMs excel at learning vast embedding representations from massive datasets, effectively mapping raw inputs (text, pixels) into semantically rich latent spaces. Within these high-dimensional latent spaces, small, specialized recursive models can then perform deep, iterative reasoning to solve complex problems, discover new solutions, and move beyond the limitations of human-bounded knowledge.

The host notes that TRMs and HRMs are task-specific, while LLMs are general-purpose (00:36:20). The guest suggests that LLMs are great at finding amazing embedding representation spaces, but reasoning within that space is not done much (00:37:00). The proposed future is to use LLMs to map inputs into a 'really cool latent space' where 'things are just nicely semantically separated,' and then use 'tiny reasoning models' with recursion within that space (00:37:12).