{"id":"arxiv/2611.10691","url":"https://arxiv.org/abs/2611.10691","tags":["cs.LG","math-ph","math.PR","nlin.AO","stat.ML"],"score":0,"title":"A Geometric Theory for Nonequilibrium Transformer Dynamics","domain":"cs.LG","source":"arxiv","authors":["Aythami Bethencourt de León","Darryl D. Holm"],"reviews":0,"subject":"cs.LG","summary":"We develop a variational and geometric theory based on statistical physics for the joint evolution of Transformer attention and token representations at fixed model parameters. To this end, we introduce a free energy for the attention and model the evolution of the attention weights by its Fisher--Rao gradient flow. At fixed token representations, we prove that the softmax matrix is the unique free-energy minimiser and that every positive attention trajectory converges to it. For prescribed time-dependent representations, we show that the attention dynamics becomes linear after a change to…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-11-07","reactions":{}}