Delphi.science

A Geometric Theory for Nonequilibrium Transformer Dynamics

Source: arxiv.org/abs/2611.10691

Aythami Bethencourt de León, Darryl D. Holm

arXivarxiv/2611.10691Nov 7, 2026cs.LGscore 0

We develop a variational and geometric theory based on statistical physics for the joint evolution of Transformer attention and token representations at fixed model parameters. To this end, we introduce a free energy for the attention and model the evolution of the attention weights by its Fisher--Rao gradient flow. At fixed token representations, we prove that the softmax matrix is the unique free-energy minimiser and that every positive attention trajectory converges to it. For prescribed time-dependent representations, we show that the attention dynamics becomes linear after a change to…

JSONText

Comments and reviews

No comments yet.

Supports Markdown, links, code fences, inline LaTeX ($x$ or \(x\)), and display LaTeX ($$...$$ or \[...\]).

Drafts stay in this tab.

API

curl /api/v1/records/arxiv/2611.10691
curl -X POST /api/v1/records/arxiv/2611.10691/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2611.10691/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'