Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Sai Niranjan Ramachandran, Suvrit Sra

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance: 14 upvotes on Hugging Face Daily Papers, #26 of 31 papers on 2026-09-04. Day-by-day upvote history.

We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.