Three loops.
Three different kinds of progress.

Start with what changes.

An AI can spend more time on a problem, learn from experience, or rewrite some of the software it uses. In casual conversation, all three can sound like “the model is improving itself.” That phrase hides the engineering distinction that matters most: what is being changed, and does the change survive the current task?

A model’s weights are its learned numerical parameters. Its activations are the temporary numerical representations computed while processing a particular input. Its surrounding system includes prompts, tool definitions, executable code, memory, retrieval, and the rules that control execution.

A useful analogy is a programmer. Thinking longer about today’s bug resembles extra inference computation. Learning a better debugging habit through practice resembles training. Writing a testing utility that helps build better testing utilities resembles a limited form of recursive self-improvement. The analogy is imperfect, but it separates three kinds of progress that deserve different measurements.

These categories overlap. A looped transformer can be trained with RL. A tool-building system can use RL to learn which tools are worth creating. A system can improve its software while keeping the underlying language model completely frozen. None of those combinations makes the three terms interchangeable.

Looped transformers: more computation with shared weights.

A conventional transformer passes its representations through a sequence of layers. Typically, each layer has its own parameters. A looped transformer instead applies a shared layer or block repeatedly, feeding its updated representation into the next pass. Some architectures combine ordinary input and output layers with a recurrent middle block.

Ordinary depth:
h1 = block_1(h0)
h2 = block_2(h1)
h3 = block_3(h2)

Shared, recurrent depth:
h1 = shared_block(h0, input)
h2 = shared_block(h1, input)
h3 = shared_block(h2, input)

The same function can produce a different result on each pass because its input representation has changed. The architectural family has a substantial history: Universal Transformers explored recurrence over depth with shared transformations and mechanisms for adaptive computation. The important axis is repeated processing within the network, distinct from moving through successive words in a sentence.

The hidden state changes; the weights do not change during inference.

In compact notation, a recurrent block might be written as h[r+1] = Fθ(h[r], x). Here, x is the input, h[r] is the representation after pass r, and θ denotes the shared parameters. This is a schematic: actual designs differ in input injection, recurrence embeddings, normalization, and attention.

Increasing the number of passes spends more computation on the current input. It does not automatically save a better model for tomorrow. Once the request is over, its internal activations ordinarily disappear unless the surrounding system explicitly stores something.

In Scaling up Test-Time Compute with Latent Reasoning, Geiping and colleagues study a recurrent-depth language model that iterates in latent space. Their proof of concept demonstrates that extra recurrent computation can improve reasoning performance on evaluated tasks. That is evidence for a particular architecture and training setup, not a promise that arbitrary extra passes improve any transformer.

Looping is different from asking for another draft.

A chatbot can write an answer, critique it, and write another answer. That is an outer loop over text and model calls. A looped transformer’s defining recurrence happens inside its computation, potentially without emitting intermediate words. A model can also generate a long chain of thought using an otherwise ordinary transformer. More reasoning tokens and more recurrent depth are different ways to spend inference compute.

Weight sharing can reduce the number of distinct parameters relative to a network with separately parameterized blocks at every effective depth. It does not remove arithmetic, attention operations, or data movement. Training must teach the recurrent block useful behavior, and evaluation must determine where extra passes help, saturate, or hurt. Simply putting an existing model in a software loop does not reproduce an architecture trained for recurrent depth.

Reinforcement learning: how rewards become learning.

RL is a framework for learning behavior from the consequences of actions. An agent interacts with an environment and seeks to increase its expected cumulative reward. The basic concepts are an observation or state, an action, a policy that selects actions, and a reward signal. Sutton and Barto’s textbook is the foundational reference.

For a scheduling agent, observations include the conversation and tool results. Actions might include asking a question or calling an API. The agent may not see the entire database, so its accessible history is more accurate than pretending it has perfect knowledge of the world. The policy describes how it chooses among possible actions.

RL is usually treated separately from supervised and unsupervised learning. Unsupervised learning still has training objectives and optimization signals; it is not literally “learning with no feedback whatsoever.” The distinctive RL signal evaluates actions through their consequences, often with a delay. RL also need not use a neural network: tabular methods can update values in a table. In language-model RL, however, parameter updates are central.

A score alone does not update a language model.

Telling a model “that was wrong” may change its next answer because the feedback enters its context. Its weights can remain unchanged. A training system must connect collected experience and rewards to an optimization procedure for the result to become a parameter update.

repeat over training batches:
    collect attempts from the current policy
    execute their actions in isolated environments
    score the resulting behavior and outcomes
    estimate which choices were better than expected
    compute a training loss and update parameters

save the resulting checkpoint
evaluate on tasks excluded from training

This is conceptual pseudocode, not a complete algorithm. Credit assignment is the difficult middle step: which earlier decisions contributed to a good or bad outcome? If an appointment fails at the end, penalizing every preceding action equally may discourage useful steps. A policy-gradient method uses estimates of return or advantage to shape updates; an advantage expresses how an outcome compares with an expected baseline.

Proximal Policy Optimization (PPO) alternates experience collection with optimization of a surrogate objective and uses a constrained style of update to discourage excessively large policy changes. DeepSeekMath introduced Group Relative Policy Optimization (GRPO), a PPO variant that uses group-relative reward comparisons to estimate advantages without a separate critic in its original formulation. Their details matter when implementing training; a proposal should explain why its chosen method fits the environment and feedback.

What the reward measures determines what can go wrong.

Consider a request to move an appointment from Monday to Friday. A persuasive confirmation message is weak evidence. The evaluator can instead check that Friday is booked at the requested time, Monday is cancelled, the patient has no unintended duplicate, and unrelated records remain intact. This is an illustrative design, not a reported experiment.

Efficiency belongs in the objective, but it should not pay the agent to skip necessary work. A reusable function that hides six database calls is still doing those calls. Measure actual latency, execution cost, and errors, including calls inside helpers. Calibrate penalties according to impact: a recoverable lookup error is different from cancelling the wrong appointment.

A completion reward encourages behavior that earns that reward. It does not guarantee that the evaluator perfectly captures the user’s intent. Test the evaluator itself against known good and bad outcomes, and inspect cases where high reward coexists with obvious failure. Freeze the evaluation criteria while comparing models so the measuring instrument does not drift with the system being judged.

Recursive self-improvement: changing the ability to improve.

Recursive self-improvement is a broader systems idea. A system changes some part of itself, and the resulting version participates in producing later improvements. Different discussions use the term at different strengths, from bounded software improvement to speculative claims about accelerating general intelligence. It is useful to state the exact mechanism being claimed.

Creating one helper function is tool generation. Building a collection of helpers is capability accumulation. The recursive connection becomes stronger when a new helper improves the system’s ability to create, test, repair, or select further helpers. For example, an agent might build a tool-testing utility, use it to find defects in later generated tools, and then improve that utility using the failures it discovers.

system version 0 proposes a tool-testing improvement
independent checks validate the candidate
version 1 retains and uses the improved tester
version 1 creates or repairs further tools more effectively
the next validated improvement becomes version 2

The persistence is in the retained artifact: a tool, code patch, workflow, or training procedure. Base-model weights may remain fixed. RL is one possible way to learn improvement decisions, but search, program synthesis, evolutionary selection, and ordinary software testing can also drive parts of the process.

Two research examples clarify the boundary.

Voyager maintains a growing library of executable skills in Minecraft and uses environmental feedback to improve programs without fine-tuning its underlying language model. It illustrates how persistent code can expand an agent’s capabilities. Skill accumulation alone should not be taken as proof of an unrestricted recursive improvement process.

The Darwin Gödel Machine explicitly investigates agents that modify their code, evaluate candidates, and maintain an archive of agent variants. It provides a more direct example of improvements to an agent affecting subsequent improvement attempts. Its reported benchmark gains are empirical results within a defined setup, not a theorem that each generation must improve or that progress continues indefinitely.

Any such system can plateau, spend more resources than its improvements justify, or overfit the tests used to select candidates. A system that writes a faster but incorrect checker might appear to improve while making the measurement worse. A credible demonstration keeps independent evaluation outside the candidate’s control and preserves earlier versions for comparison.

One appointment, three different mechanisms.

Imagine an agent operating in a synthetic appointment system. It can inspect availability, check coverage, reserve slots, cancel bookings, and send confirmations. The user wants Friday afternoon, the same clinician, and a virtual visit. These are constraints the final booking must satisfy.

With a looped transformer, repeated internal computation may help reconcile those constraints before the next question or tool call. The benefit would appear within the current attempt. The model’s weights remain fixed during that inference run.

With RL, the agent makes many training attempts across appointment scenarios. The environment scores their outcomes. The training algorithm updates the policy so useful choices become more likely across future attempts. A model does not have to start by trying every possible action: a pretrained policy already supplies substantial prior knowledge.

With tool evolution, the agent might compose primitive calls into reschedule_appointment(). A robust version would check a replacement before releasing the old slot, handle failures according to the sandbox’s transaction rules, and report the actual outcome. A helper’s name is not evidence of correctness; execution is.

To train tool creation with RL, creation itself must be among the decisions that training can influence. The system might expose choices to create a candidate, reuse a registered tool, modify one, or continue with primitives. Generated code must be executed and tested, and its downstream performance must feed back into the relevant training objective. If an external script always creates the tools, the policy may be learning only tool selection.

With a recursive extension, the agent could improve a reusable transaction wrapper or testing utility that then makes future generated tools more reliable. Now a retained improvement helps the machinery responsible for later improvements. That is a more precise claim than merely saying that the agent wrote a rescheduling function.

Design experiments that separate the gains.

A useful first pilot can use one synthetic scheduling sandbox and 10–20 task templates. That is enough to test plumbing, graders, and failure handling. It is not a claim that 10–20 examples suffice for effective neural RL training or statistically strong conclusions. Expanded training and evaluation should follow only once the small system behaves coherently.

Start by saving initial states and specifying acceptable outcomes. Multiple final states may be valid: if the user accepts any Friday afternoon slot, do not require one arbitrary slot. Verify constraints and invariants rather than enforcing an exact database snapshot that rejects equally correct solutions.

Measure success rate, invalid actions, unintended state changes, wall-clock time, tokens, actual primitive operations, and tool reuse on unseen scenarios. Include the cost of generating and validating tools. A helper that saves a little execution time after an expensive development cycle may only pay off after substantial reuse.

Hold out task families, not merely different patient names attached to identical scripts. Repeatedly testing the same benchmark during candidate selection can turn it into a development set. Reserve a separate final test set. Repeat runs to quantify variability and report both average outcomes and failures; a tiny pilot cannot justify precise claims about broad capability.

For the recursive comparison, hold the base model, budgets, initial tools, and evaluation conditions constant where possible. Track an explicit causal chain: improvement A made it easier to discover or validate improvement B. Rising scores alone may instead reflect more sampling, more compute, or a larger library assembled by unchanged machinery.

The infrastructure determines whether the loop is useful.

Each loop creates a different resource burden. Recurrent inference repeats computation inside a request. Neural RL adds experience generation, parameter updates, checkpoints, and many environment instances. Tool evolution adds code execution, artifact storage, dependency tracking, and candidate evaluation.

For looped models, variable pass counts can complicate batching and tail latency. Fewer distinct weights can help parameter storage, but activations, attention, and cache behavior still depend on the architecture. The recurring block is not guaranteed to remain in the fastest hardware cache, and latent computation is not free merely because it emits no visible tokens.

For RL, the simulator can become the bottleneck. A fast accelerator does little good if every rollout waits on a slow environment. Keep environments isolated, make initial states reproducible, and record which policy checkpoint generated each trajectory. Otherwise data from incompatible versions can make training and debugging difficult to interpret.

For evolving tools, a registry needs more than names. Store schemas, versions, dependencies, tests, execution limits, and measured behavior. A new helper can change the meaning of an old policy’s action space. Freezing the tool registry within a training phase, then validating changes between phases, is one possible way to make comparisons easier to interpret.

The appointment example also shows why valid outcomes can require process constraints. A final state may look correct after an unauthorized cancellation was briefly made and repaired. If such intermediate actions matter, score the execution trace as well as the final state. Outcome checks and trajectory checks answer complementary questions.

Ask what survives the loop.

When someone says an AI “learns by thinking longer,” ask whether its parameters change. If only activations change, the claim concerns inference computation. When a system receives rewards, ask where the optimization step happens and what it updates. A scored conversation can remain ordinary inference.

When an agent creates tools, ask whether those tools persist, generalize, and improve later tool-building attempts. A useful artifact can increase system capability without changing the model. A recursive claim requires a connection between improvements and the production of further improvements.

These distinctions create a practical design choice. A difficult individual request may call for more inference compute. A recurring behavior problem may call for better data, objectives, and training. Repeated engineering work may benefit from validated reusable tools and an improvement process around them. The right loop depends on the object you need to change and the evidence you can collect.

Sources and further reading

The mechanisms are explained using primary research and a standard textbook. The appointment system and experimental plan are illustrative designs; no new benchmark results are claimed here.

1. Dehghani et al., Universal Transformers (2018). Shared transformations and recurrence over depth.
2. Geiping et al., Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (2025). Latent recurrence and test-time computation.
3. Sutton and Barto, Reinforcement Learning: An Introduction, second edition (2018). Policies, rewards, returns, and learning from interaction.
4. Schulman et al., Proximal Policy Optimization Algorithms (2017). Policy optimization through sampled experience.
5. Shao et al., DeepSeekMath (2024). Introduction of GRPO.
6. Wang et al., Voyager (2023). Persistent executable skills without base-model fine-tuning.
7. Zhang et al., Darwin Gödel Machine (2025; revised 2026). Empirical evolution of agents that modify their code.