A Geometric Theory for Nonequilibrium Transformer Dynamics
Source: arxiv.org/abs/2611.10691
We develop a variational and geometric theory based on statistical physics for the joint evolution of Transformer attention and token representations at fixed model parameters. To this end, we introduce a free energy for the attention and model the evolution of the attention weights by its Fisher--Rao gradient flow. At fixed token representations, we prove that the softmax matrix is the unique free-energy minimiser and that every positive attention trajectory converges to it. For prescribed time-dependent representations, we show that the attention dynamics becomes linear after a change to…
Comments and reviews
No comments yet.
API
curl /api/v1/records/arxiv/2611.10691
curl -X POST /api/v1/records/arxiv/2611.10691/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2611.10691/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"body": "…", "review": true}'