A floor that survives depth

Stack enough layers and the weakest direction in the activations quietly dies. Suppose each block shrinks its flattest direction to 60% of what came in. That sounds harmless, but it compounds: after 10 blocks the direction is down to 0.6% of its original size, and after 24 blocks to about 5 parts in a million. A direction that started out perfectly visible has been squeezed below anything you could measure. So in a plain deep network the smallest singular value races to zero with depth, and a reading at layer 30 has nothing to do with a reading at layer 3.

What σ_min measures

The quantity being squeezed is $\sigma_{\min}$, the smallest singular value of a layer's activations across a batch of inputs. It measures the flattest direction of the representation: the one along which the layer's outputs barely move, so different inputs land in nearly the same place. A healthy layer spreads its inputs out in every direction and $\sigma_{\min}$ sits comfortably above zero. A layer that has stopped using some direction has a near-zero $\sigma_{\min}$, a dead direction showing up in the representation itself.

The identity path holds the floor up

A transformer does not stack its blocks plainly. It carries a residual stream: each block adds its output onto the running activations and carries the sum forward,

$$ x_{\ell+1} = x_{\ell} + f(x_{\ell}). $$

Picture two kinds of relay station passing a signal down a line. A plain block is a relay that re-encodes the signal and sends on only its own version; if it has learned to ignore the weak direction, that direction leaves as nearly zero. A residual block is a relay that adds its version on top of a copy of whatever came in. Even when its own contribution along the weak direction is near zero, the copy carries that direction forward.

In symbols, take the weakest direction of the stream. Through a plain block it becomes $f(x) \approx 0$ and is gone. Through a residual block it becomes $x + f(x) \approx x$ and passes through essentially intact. A block can nudge the floor, but it cannot compound it away, because the identity carries every direction forward. So $\sigma_{\min}$ stops short of zero: it settles near a single block's squeeze and holds there, however deep the stack.

Start with the residual connection on, depth 24, and a per-block squeeze of 0.6: $\sigma_{\min}$ drops a little, then holds flat across every layer. Switch the connection off and the same blocks send the floor toward zero. Turn it back on and drag the depth out to 48, and the floor barely moves; drag the squeeze down to 0.3, an aggressive block, and it still holds. The protection is the skip connection, and removing it breaks the invariance in front of you.

Why a depth-invariant floor matters

Because $\sigma_{\min}$ stays the same order of magnitude at every layer, you can read it at layer 3 and at layer 30 and compare the two directly. A near-dead direction that forms at one depth stays legible at another, where in a plain stack the floor at layer 30 would only reflect 30 compounded squeezes and tell you nothing about the direction's own structure. That comparability is what makes $\sigma_{\min}$ a practical dead-direction signal at the scale of a real transformer: it falls out of a single forward pass, with no curvature matrix to form and no posterior to sample.¹

The prediction has been put to real models. Across fourteen pretrained transformers, from 160-million-parameter language models to a 31-billion-parameter multimodal one, the floor holds in thirteen; and the one exception is the best part of the story, because it was not a failure of the prediction but a genuine dead direction in that model, which the same reads then pinpointed to a single sub-layer.²

Still open / underspecified. The protection is approximate. It holds while each block's contribution stays modest relative to the stream, which pre-normalisation and careful initialisation arrange in practice, and the fourteen-model survey above says that is how real models behave. What stays open is the fine grain: exactly which directions the identity path protects, and what a large or adversarially aligned block update would do to them. And the read has a natural limit at the other end: at a deeply singular checkpoint the raw value of $\sigma_{\min}$ can sink below what floating point resolves even though the rank-level structure is intact. Used the other way round, that sensitivity is an asset: a $\sigma_{\min}$ still rising says the network has not yet formed any structure to read (nothing to read yet#20).

This makes $\sigma_{\min}$ the cheapest dead-direction read that works on any architecture: the K-FAC bridge needs a layer's curvature, the order needs a decay fit, and a posterior read needs sampling, while $\sigma_{\min}$ needs only the activations a forward pass already produces. A LayerNorm model admits a cheaper read still#10, taken straight from the normalization weights with no forward pass. ↩︎
The fourteen-transformer depth-invariance survey, and the diagnosis of the one exception, are from Tejas Pradeep Shirodkar and P. J. Narayanan, Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale, arXiv:2606.19491 (2026). ↩︎