Every way a direction dies
Point the frozen read at a direction and it comes back with more than a number: it comes back with a kind. Dead structure is not one phenomenon. Some directions are dead because a piece of the network genuinely died; some are dead because the architecture never let them matter in the first place. The reads of this section, slope and magnitude together, sort every direction into a small taxonomy, and the taxonomy is where the geometry starts naming parts of the network you can point to.1
The kinds that carry an order
A finite slope marks a genuine singularity, and here is the striking part: the value of the order is fixed by the architecture, not by the training run.
The commonest kind is a node-death: a hidden unit whose incoming and outgoing weights have both collapsed. Its order is set by how the unit's activation switches on at zero. An activation that rises linearly there, like gelu or ordinary ReLU, dies at $k = 2$; one that rises quadratically, squared ReLU, at $k = 3$; cubically at $k = 4$. One extra power in the activation, one extra step in the order, and the training run has no say in it: swap the activation and the integer moves with it, verified at $r^2 = 1.000$ on constructed cells.
A depth singularity belongs to compositions: a deep linear chain that collapses to zero carries the order of its depth, $k = L$, one factor of flatness per layer, tested at depths three through five. A unit overlap, two units merging into one, dies at the curvature order $k = 2$ along the direction that splits them apart. And a convolutional channel-death is the node-death's image in a conv layer, read through a spatial-patch factor instead of the activation factor, same verdict downstream.
The kind that does not
A scan that stays flat all the way is not a failed reading; it is a gauge#12. A symmetry direction never carried curvature to lose, so its directional Fisher sits at a floor rather than rising with any exponent: the LayerNorm kernel#10, the attention rotations, the cross-entropy shift, the ReLU rescale. These add to the dead count without carrying a finite order, and the assembly#23 has to book them separately.
The subtle case is the trap. An attention head's rotation orbit is curved, and a curved orbit read along its tangent rises with slope 2, exactly the slope a $k = 2$ node-death shows. On slope alone the two are indistinguishable. What separates them is magnitude: the gauge's Fisher floor sits orders of magnitude below anything alive (on the measured transformer, a factor of about $10^6$ below a live direction), while a genuine node-death rises from a floor toward live values. The read's verdicts always weigh slope and magnitude, and this one distinction is why.
Pick a structure and scan it. Every genuine kind rises with its architecture-fixed slope; every gauge sits at its floor; and the curved-orbit case shows you slope 2 with a floor so deep the verdict panel refuses to call it a death. The taxonomy is not a table to memorise, it is a decision the two numbers make.
The order is an invariant
If the architecture fixes the order, then training the same architecture with different optimizers should change everything except the integer. That is exactly what happens. Take one network that prunes to a node-death and read it after training under SGD, Adam, AdamW, and RMSProp: every run returns $k \approx 3$, the activation's order. What differs is everything else, and the differences are the optimizers' signatures. One leaves the dead subspace clean on the coordinate axes; another leaves it rotated, where only the constructed read#21 can follow it; a third leaves it sharper or shallower. The architecture fixes the order; the optimizer chooses the basis and the sharpness. On a deep transformer an optimizer can even leave the structure too diffuse to carry any order, vanilla Muon does exactly that, and the verdict there is that the solution holds nothing to read, not that the instrument failed.1
A real network, read whole
Put the taxonomy on a real model and it earns its keep immediately. A fine-tuned DINOv2 vision transformer, read block by block, direction by direction, turns out to carry no node-death at all: not one genuinely dying feature anywhere in twelve blocks. Its entire dead structure is gauge, the LayerNorm kernel, matched at $|\cos| = 1.0000$ per block with projection variance at machine precision, plus the attention rotations. Everything flat about that model is symmetry, nothing is dying, and a monitor that only counted near-zero eigenvalues would have called it riddled with dead directions.
Train a similar transformer from scratch under weight decay and the picture flips: as it compresses its over-provisioned MLP it forms a genuine rotated node-death, which the read recovers at the activation's predicted order across conditions ($\hat k = 3.00$ and $2.02$ for the two activations), with a coordinate-concentration of $0.07$ confirming the optimizer left it far off the axes.1 Same architecture family, same instruments, opposite verdicts, and both are right: fine-tuning had no reason to kill capacity that pretraining already shaped, while from-scratch compression did. The taxonomy is what lets the reading say that, instead of "both models have small eigenvalues."
Still open / underspecified. The taxonomy covers the structures this program has constructed, predicted, or found; it is a catalogue, not a completeness theorem. Attention-head death and expert death in mixtures of experts are catalogued with detectors in hand but no verified read yet. The magnitude criterion draws the gauge-versus-death line with a floor that is architecture- and precision-dependent, and structure sitting near that line inherits its ambiguity.
- The dead-structure taxonomy, the magnitude criterion, the optimizer-invariance experiment, and the fine-tuned versus from-scratch ViT readings are from Tejas Pradeep Shirodkar, Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment, arXiv:2607.00603 (2026). ↩︎