Why singular models generalize
Time to collect on the opening puzzle. This section began with models that should overfit and do not, and every explainer since has added a piece: the map from weights to functions is many-to-one, the loss valley is flat along its dead directions, and each direction's flatness is an integer $k$, which fixes the real log canonical threshold $\lambda = 1/(2k)$. This is where the pieces pay out. What does $\lambda$ actually buy?
The short answer is that $\lambda$ is the exchange rate between the geometry of the valley and the statistics of generalisation. It shows up in two places at once, and they turn out to be the same number.
The volume behind λ
Fix a small tolerance $\varepsilon$ and ask how much of parameter space fits the truth that well: the volume of the set $\{\theta : K(\theta) \le \varepsilon\}$, where $K$ is the KL divergence to the true distribution. Along an order-$k$ dead direction $K(w) = w^{2k}$, so the good set is $\{\,|w| \le \varepsilon^{1/2k}\,\}$, and the size of that good set scales as
$$ \mathrm{vol}\{K \le \varepsilon\} \;\propto\; \varepsilon^{\lambda}, \qquad \lambda = \frac{1}{2k}. $$
A flatter valley (larger $k$, smaller $\lambda$) makes this volume larger, so more of parameter space sits near-optimal. This is the singular-learning form of Occam's razor: a simple model is one whose well-fitting parameters fill a lot of room, the prior places a lot of mass there, and the model is preferred without any explicit penalty on parameter count.
The price of a model
The same $\lambda$ sets the cost side. Watanabe's free energy, the negative log marginal likelihood that Bayesian model selection minimises, behaves as
$$ F_n \;\approx\; n\,L_n \;+\; \lambda \log n \;+\; O(\log\log n), $$
so the complexity penalty is $\lambda \log n$. Classical model selection (the BIC) uses $\tfrac{d}{2}\log n$ with $d$ the raw parameter count. For a singular model $\lambda$ can fall far below $d/2$, so the price the network actually pays is much smaller than its parameter count suggests. Watanabe's theorem ties the cost to the error: the expected generalisation error $K_n$, the KL divergence from the truth that remains after training on $n$ examples, carries the very same coefficient,1
$$ \mathbb{E}\big[K_n\big] \;\approx\; \frac{\lambda}{n}. $$
A smaller $\lambda$ therefore means a smaller expected error. The two meet because the same flatness does both jobs: more near-optimal volume puts more of the prior's weight on parameters that fit, and that is what lowers both the price the model pays and the error it makes. The flatness you can read off a single dead direction is the same quantity that governs how well the whole model generalises.
Raise the order $k$ and the floor flattens along the dead direction, so the good-fit basin on the left stretches and a larger share of candidate models fall inside it: more near-optimal volume. On the right, the same $\lambda = 1/(2k)$ drops the whole error curve $\lambda/n$ below the classical rate $\tfrac{1}{2n}$ that a single regular parameter would cost. The two learning curves run parallel, so the advantage is a constant factor of $k$, and it reads either way: $k$ times less error at any data size, or the same error from $k$ times less data. One knob moves both panels, geometry and generalisation at once.
The fluctuation that measures overfitting
$\lambda$ controls how much of parameter space fits, but it is not the only constant the geometry hands you. The Bayesian posterior does not settle on a single point in the flat valley; it spreads over the whole near-optimal set, and the singular fluctuation $\nu$ measures how widely it spreads. Together with $\lambda$ and the multiplicity $m$, it forms the Watanabe triple $(\lambda, m, \nu)$, the three constants that summarise how a singular model learns.
The fluctuation is what lets you estimate generalisation without knowing the truth. Different settings in that spread make slightly different predictions, and they disagree most on data the model has not seen, so the spread costs you there: the expected gap between test loss and training loss is about $2\nu/n$. The WAIC estimator measures that spread directly and adds it back, turning the training loss into an estimate of the test loss.
Widen the fluctuation and the gap grows; add data and the posterior tightens onto the best fits, so the fan narrows and the gap closes as $1/n$. WAIC tracks the test loss the whole way, because it measures the very spread that creates the gap. A regular model has $\nu = \lambda = d/2$, where this reduces to the classical AIC, and the singular case is where $\nu$ and $\lambda$ part ways and the raw parameter count fails. Measured on real trained networks, $\nu$ holds one more surprise: the fluctuation a dead direction should carry can be partly absorbed by the live structure around it, so the realized value falls below the clean theory. That story is taken up in adding up the orders#23.
Back to the puzzle
This is the payoff of the whole chain. A dead direction is a flat direction of the loss; its order $k$ measures how flat; the Fisher decay slope $2(k-1)$ lets you read $k$ from a trained network; and $k$ fixes $\lambda = 1/(2k)$, which is at once the effective parameter count and the generalisation coefficient. Reading the slope is reading the generalisation.
What stays hard is that a real network carries many dead directions of different orders at once, and its global $\lambda$ is set by the smallest of their per-direction values $1/(2k)$, so by its flattest structure, together with how many directions attain it (the multiplicity). Turning a spectrum of local orders into the single $\lambda$ that governs the model is where the measurement work of this section lives, and it gets its own explainer at the far end of the arc: adding up the orders#23.
- The free-energy expansion and the $\lambda/n$ generalisation rate are Watanabe's; see Algebraic Geometry and Statistical Learning Theory (Cambridge University Press, 2009) and Mathematical Theory of Bayesian Statistics (CRC Press, 2018). ↩︎