Spherical Fixed-Point Reasoners

Looped transformers have to know when to stop

Scaling test-time compute is the main lever we have for making sequence models reason better. Chain-of-thought does it in token space. Looped transformers do it in latent space: a single weight-tied block is applied repeatedly, so depth becomes an iteration count chosen at inference time, and compute can be allocated per input according to how hard it is. On algorithmic puzzles such as Sudoku, mazes and ARC, this recipe lets models with a few million parameters compete with far larger ones.

A looped model raises two questions. When should it stop? And will it stay well-behaved as it runs? The fixed-point view answers the first: treat the loop as a solver for and halt once the iterate stops moving. Fixed-Point Reasoners (FPRM) turn this into a practical recipe: a damped iteration whose step size decays on a patience rule, a relative-norm residual as the halting test, and a boundedness theorem that keeps the recurrent state from diverging.

The second question is where the Euclidean formulation shows its cost. Nothing in a pre-norm residual stream stops the norm of the state from growing as the block is applied again and again. To contain it, FPRM rescales the residual stream with four coefficients that a boundedness theorem ties together,

and it has to divide the halting residual by the norm of the proposal, , to make the test scale-invariant. All of this machinery exists for one reason: the norm is not fixed.

The same loop, on the sphere

The normalized transformer (nGPT) gets stable dynamics by changing the geometry rather than adding control. Every hidden state lives on the unit hypersphere , and each sub-layer is a learned angular move followed by renormalization, a retraction. The divergence that FPRM has to guard against simply cannot occur. Spherical Fixed-Point Reasoners (SFPRM) run FPRM's damped solver on this sphere, so that every control quantity of the loop becomes an angle.

The one primitive we need is spherical linear interpolation. For unit vectors at angle ,

moves a fraction of the way along the shortest great-circle arc from to and stays on the sphere. Applied token by token, it gives the four steps of one SFPRM iteration:

1. INJECT   : move each token a fraction of the arc toward its input token. One angle replaces FPRM's affine mix and the coupling that came with it.
2. BLOCK   : the weight-tied nGPT block proposes a new point. Its attention and MLP sub-layers are retractions, so the proposal is on the sphere by construction.
3. DAMP   : rotate a fraction of the way to the proposal. Each token turns by exactly , so is an angular step size that FPRM's patience-based decay literally shrinks.
4. HALT   stop once , i.e. once every token's proposal lies within an angle of its iterate (; we use , about ). The residual is an angle; there is nothing to normalize.

Everything else is FPRM. The step size starts at and is multiplied by a decay factor whenever the residual has not improved for five consecutive iterations; training uses deep supervision with truncated backpropagation through one segment of six geodesic steps per optimizer step, with the carried state detached between segments. The readout is nGPT's: logits are with a learned scale , and the rows of the head are left unnormalized so the head can set its own per-class scale while the state stays on the sphere.

What the geometry buys analytically

Boundedness is now structural: injection and damping return unit vectors, the block does too, so the state stays on the sphere for every iteration and every . Recurrent magnitudes can neither explode nor vanish. But boundedness alone says nothing about whether a fixed point exists or is reached, so the paper proves three things about the exact recurrence.

On a sphere, continuity alone does not give you a fixed point: rotating a circle moves every point. Injection repairs this by keeping every injected state inside a cap around the input, which, unlike the sphere, can be flattened into a closed ball where Brouwer's theorem applies. No Lipschitz constant, contraction modulus or spectral bound is needed. FPRM proves existence only through global convergence, hence under a contraction assumption.

The proof linearizes the damped step to , which moves each eigenvalue a fraction from toward ; the window above is exactly what keeps them all inside the unit disk. Damping tames modes that oscillate or spiral away under the raw map, not expansive ones, which training has to remove. This is the spherical counterpart of FPRM's stabilization theorem, now with an explicit window and with unit norm preserved.

In logarithmic coordinates at , injection multiplies by : for near one, the first injection pulls every admissible state into a small ball around the input on which the map is a Banach contraction. Together the three results cover every injection strength: a fixed point exists for all , strong injection makes it a unique global attractor, and damping gives local convergence under weaker injection.

We checked the last theorem on the trained Sudoku model by overriding at inference and running the undamped map from eight initializations, including the input and its antipode, which the theorem excludes. For all eight end states coincide on every puzzle. At the learned the raw map already reaches an attracting fixed point on 91% of the puzzles, and on the puzzle shown above all eight paths end inside the exact-match region. There is a trade-off hiding in the figure, though: for the iteration converges within a few steps to a unique fixed point that no longer solves the puzzle. Injection that is strong enough to make the map a contraction leaves too little room for computation.

Results

We evaluate on Sudoku-Extreme, Maze-Hard, Mini-ARC and ARC-1, which test generalization from a small number of training examples. SFPRM and FPRM share a 6.83M-parameter transformer backbone and FPRM's hyperparameters. SFPRM trains its weight matrices with Muon and everything else with AdamW, learns the injection fraction except on Sudoku where it is fixed at , and halts at . Every number is a single seed.

Published numbers from the respective papers; values in parentheses are our own runs under the same protocol as SFPRM. Within the 5–7M group the best result is bold and the second best underlined. Our own FPRM runs fell short of the published 94.2 and 87.0, so we compare against the published numbers and use the released checkpoints for all FPRM evaluations below.

Confining the state to the sphere does not cost accuracy; it improves it. On Sudoku-Extreme SFPRM reaches 96.6% exact accuracy, 2.4 points above FPRM, trained on 5k base puzzles augmented 200× rather than FPRM's 1k puzzles augmented 1000×, the same number of training examples but five times as many distinct puzzles. On Maze-Hard it reaches 87.9% on the 1,000 test mazes, above TRM and FPRM and behind only the 27M-parameter Attractor model. On Mini-ARC it reaches 59.1% Pass@1, above the best published result in its size group and above every baseline retrained under our protocol. On ARC-1 it matches FPRM.

Halting

To see how the two solvers converge and when they stop, we run each trained model with halting disabled, decode every iterate, and record the residual its solver tests. Both residuals fall quickly, and both models saturate their accuracy within 20 to 35 iterations, so every later loop is spent after the answer has stopped changing. SFPRM converges faster and stops sooner. A detail worth knowing: on Sudoku, the puzzles a solver halts on are solved in 99.7 to 100% of cases, so a halt certifies the answer, whereas a missing halt does not certify a failure.

Swapping the solver at inference

Because the loop is a genuine fixed-point solve, the damped step can be replaced at inference by a standard solver without touching the model or the halting rule. Anderson acceleration halts earlier than the default solver on every Maze example while staying within one point of its accuracy, and on Sudoku it brings the median halt from 435 iterations down to 37, nine times fewer evaluations of the block averaged over all examples. Limited-memory Broyden, the solver used for deep equilibrium models, reaches the tolerance about as fast but then leaves the fixed point and degrades, so it is usable only with halting. Without damping, 37 to 100% of the examples never reach the tolerance at all. Training with these solvers, rather than swapping them in at inference, is left for future work.

The geometry is readable

On the sphere a token state has only one free quantity, its direction, so we can probe what the loop writes into the state through a single number per cell: the anchor angle between a cell's iterate and its own unit-normalized input embedding, the quantity that injection acts on.

Each loop iteration writes the cell's class into the direction of its state. FPRM, by contrast, encodes the Sudoku clue mask mainly in the norm of its state, a scale set by training rather than by the geometry; its anchor-angle map shows only small differences between clues and blanks, and on Maze it separates walls from free cells but not path cells from empty ones. On Mini-ARC the cells a task rewrites end furthest from their input under SFPRM, with an AUC of 0.81 for telling rewritten cells from the rest, while FPRM's ordering varies from task to task (AUC 0.51).

The final angle predicts correctness

The same quantity flags wrong answers without any label. On the Sudoku puzzles SFPRM fails to solve, both cell classes end farther from their anchors: the clues at about instead of , almost disjoint from the solved puzzles, and the blanks at instead of . FPRM's final norms separate the two outcomes far less; its clue and blank norms overlap widely.

Citation

@article{zela2026sfprm,
  title   = {Spherical Fixed-Point Reasoners: Adaptive Loop Halting on the Hypersphere},
  author  = {Zela, Arber and Maranjyan, Artavazd and Puigdemont, Pol and Berghaus, David
             and Boreiko, Valentyn and Cevher, Volkan},
  journal = {arXiv preprint arXiv:2610.25646},
  year    = {2026}
}