Reading a dead direction from both sides

A dead direction leaves a track on each side of the network: one in what the layer computes, one in what the loss sends back. The program rests on one bridge: a dead direction is at once a flat direction of the loss, the way singular learning theory sees it, and a collapse of the Fisher information, the way information geometry sees it. Here the bridge becomes a working measurement, two cheap reads, one per side, that lock together whenever they are looking at the same death, and whose disagreement, when it comes, is not a failure but a further reading.

Two reads, one from each side

There are two cheap-ish handles on a dead direction, coming from opposite sides of the network.

The activation side reads $\sigma_{\min}$, the smallest singular value of a layer's activations: the flattest direction of what the layer outputs. It comes from a single forward pass.

The Fisher side reads $\lambda_{\min}(G)$, the smallest eigenvalue of the layer's gradient Gram $G = \mathbb{E}[g\,g^\top]$: how flat the loss is along its weakest direction. This is the backward read, the gradient side that the layer-by-layer factorisation#6 isolates as the $G$ factor, taken here as one Gram over the layer rather than a per-weight factor.

These are not independent. The activation side is fixed by the activation Gram $A = \mathbb{E}[a\,a^\top]$, whose weakest eigenvalue is the activation singular value squared, $\lambda_{\min}(A) = \sigma_{\min}^2$. The gradient that builds $G$ flows through those same activations, so along a squeezed direction the backward read inherits the squeeze, squared:

$$ \lambda_{\min}(G) \;\propto\; \sigma_{\min}^2. $$

As the activation direction is squeezed toward zero, the curvature along it falls as the square.

This tracking holds while the two sides are reading the same dead direction: a forward death, where low activation variance starves the gradient along it too. A backward death is the other case. The loss stops constraining an output direction while its activations stay full-rank, so $G$ grows a weak direction that $A$ does not have. Now $\lambda_{\min}(G)$ falls while $\sigma_{\min}$ holds, the points lift off the slope-2 line, and the cheap forward read misses a degeneracy the gradient side still sees. That lift-off is the signature that the dead direction sits on the backward side, and the reason the layer-by-layer factorisation#6 insists on checking both.

Drag the approach. Both reads fall, and plotted against each other they stay on a line of slope 2: $\lambda_{\min}(G)$ tracks $\sigma_{\min}^2$ all the way down. This run is a forward death, so the two sides stay locked. The activation side and the gradient side are reading the same dead direction here, which is why a cheap forward-pass $\sigma_{\min}$ stands in for the expensive gradient read. The next demo breaks that lock.

Hold the activation side fixed and starve the gradient instead. The activation $\sigma_{\min}$ barely moves, but $\lambda_{\min}(G)$ keeps falling, and the point peels below the slope-2 line. That gap is a backward death: a degeneracy the forward read cannot see, so a layer that looks healthy from the activation side still has to be checked from the gradient side.

When a backward death happens

A backward death is not exotic. It is what you get whenever the loss stops sending gradient along an output direction of a layer while that layer's inputs keep varying, so the backward factor $G$ loses rank where the forward factor $A$ stays full. Four common cases:

These span both kinds of dead direction: a gauge and a genuine flat direction#3 can each surface as a backward death. Forward and backward name which factor has gone slack, the activations $A$ or the gradients $G$, a separate question from what kind of flat direction it is. In every case a forward read calls the layer healthy, and the degeneracy lives in $G$, which is why the layer-by-layer factorisation#6 reads both factors of a layer, and why a single backward pass, $u^\top G u$, earns its place next to the cheaper $\sigma_{\min}$. Where the candidate direction $u$ comes from is its own story: at a frozen checkpoint the two factors are enough to construct it, which is how reading at a standstill#21 finds dead directions that no eigensolve of the bottom of the spectrum would surface.

Why it holds when the rate breaks

The correlation is geometric. It only needs the two observables to be reading the same rank-deficient structure, with no requirement on how the network got there. So it survives in regimes where the decay-rate reading from earlier breaks down. The measured numbers put the pattern plainly, in rank correlation: $\rho = +1.0$ in the clean analytic case, $\rho \ge +0.83$ on trained testbeds whose coordinates stay aligned with the geometry, and $\rho = +0.75$ even under Adam with cross-entropy, where the per-coordinate preconditioner scrambles the trajectory so badly that the rate reading fails outright yet the static correlation keeps its sign and most of its strength.1 That robustness is what turns the bridge into a dependable detector: read a dead direction from whichever side is cheaper at the scale you are working at, and trust that the other side would have told you the same thing.

Still open / underspecified. The proportionality $\lambda_{\min}(G) \propto \sigma_{\min}^2$ is exact in the clean canonical case and holds at the rank-correlation level across the testbeds measured so far; the constant in front, and how tightly the two reads agree, vary with architecture and the finite-sample budget. A backward-side death is the known way the line breaks, where $\lambda_{\min}(G)$ outruns $\sigma_{\min}^2$. Mapping where the correlation is tight enough to substitute one read for the other, where it only ranks, and where the gradient side has to be read directly, is part of the measurement work.

  1. The structural correlation between activation and Fisher spectra, and its validation across architectures and optimizers, are from Tejas Pradeep Shirodkar and P. J. Narayanan, Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity, arXiv:2606.21158 (2026). ↩︎