Asymptotic Optimality of $cμ$-Rule for LLM Inference Scheduling under Memory Constraint

Source: arxiv.org/abs/2611.04807

Zhenghao Li, Shengbo Wang, Nian Si

arXivarxiv/2611.04807Nov 3, 2026math.OCscore 0

Large language model (LLM) inference, the process by which a pre-trained model receives an input prompt and generates output tokens sequentially, has become increasingly important with the widespread deployment of LLM-based applications. As inference workloads continue to scale, GPU memory has emerged as a critical bottleneck: the long input prompts and generation sequences common in agentic and reasoning workloads lead to substantial key-value (KV) cache usage. In this paper, we study the resulting online scheduling problem from a queueing perspective. Requests arrive stochastically with…

JSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2611.04807
curl -X POST /api/v1/records/arxiv/2611.04807/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2611.04807/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'