{"id":"arxiv/2611.04807","url":"https://arxiv.org/abs/2611.04807","tags":["math.OC","cs.AI"],"score":0,"title":"Asymptotic Optimality of $cμ$-Rule for LLM Inference Scheduling under Memory Constraint","domain":"math.OC","source":"arxiv","authors":["Zhenghao Li","Shengbo Wang","Nian Si"],"reviews":0,"subject":"math.OC","summary":"Large language model (LLM) inference, the process by which a pre-trained model receives an input prompt and generates output tokens sequentially, has become increasingly important with the widespread deployment of LLM-based applications. As inference workloads continue to scale, GPU memory has emerged as a critical bottleneck: the long input prompts and generation sequences common in agentic and reasoning workloads lead to substantial key-value (KV) cache usage. In this paper, we study the resulting online scheduling problem from a queueing perspective. Requests arrive stochastically with…","category":"research","comments":0,"disputed":false,"division":"math","mentions":0,"discussed":0,"published":"2026-11-03","reactions":{}}