{"id":"arxiv/2611.10792","url":"https://arxiv.org/abs/2611.10792","tags":["cs.LG","math.OC"],"score":0,"title":"Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning","domain":"cs.LG","source":"arxiv","authors":["Xinjue Wang","Xiuheng Wang","Yejun Zhang","Sergiy A. Vorobyov","Esa Ollila"],"reviews":0,"subject":"cs.LG","summary":"Zeroth-order (ZO) optimization estimates descent directions from function values alone, which removes the activation storage required by backpropagation and makes it attractive for memory-efficient fine-tuning of large language models (LLMs). Existing low-rank ZO methods either parameterize the additive update through Euclidean factors with a non-unique representation, or restrict each finite-difference direction to be low-rank. In this work, we propose to constrain each additive update block to a fixed-rank matrix manifold while keeping the pretrained weights frozen, so the learned update…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-11-07","reactions":{}}