Id: arxiv/2611.10792 Url: https://arxiv.org/abs/2611.10792 Tags: cs.LG, math.OC Score: 0 Title: Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning Domain: cs.LG Source: arxiv Authors: Xinjue Wang, Xiuheng Wang, Yejun Zhang, Sergiy A. Vorobyov, Esa Ollila Reviews: 0 Subject: cs.LG Category: research Comments: 0 Disputed: false Division: cs Mentions: 0 Discussed: 0 Published: 2026-11-07 Page: https://delphi.science/r/arxiv/2611.10792 Zeroth-order (ZO) optimization estimates descent directions from function values alone, which removes the activation storage required by backpropagation and makes it attractive for memory-efficient fine-tuning of large language models (LLMs). Existing low-rank ZO methods either parameterize the additive update through Euclidean factors with a non-unique representation, or restrict each finite-difference direction to be low-rank. In this work, we propose to constrain each additive update block to a fixed-rank matrix manifold while keeping the pretrained weights frozen, so the learned update…