{"id":"arxiv/2610.41038","url":"https://arxiv.org/abs/2610.41038","tags":["cs.LG","math.OC","math.PR"],"score":0,"title":"When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?","domain":"cs.LG","source":"arxiv","authors":["Kağan Akman","Jake Feldman Starosta","Naci Saldi","Serdar Yüksel"],"reviews":0,"subject":"cs.LG","summary":"There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data. To this end, this paper studies next-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either stationary and ergodic non-Markovian sources or those which are hidden Markov models of some finite order (possibly non-stationary). In either case, we first obtain error bounds via finite models with finite window approximations. For stationary ergodic sources, the resulting error is bounded by…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-10-29","reactions":{}}