Id: arxiv/2602.01627 Url: https://arxiv.org/abs/2602.01627 Tags: math.OC Score: 0 Title: Why Does Adaptive Zeroth-Order Optimization Work? Domain: math.OC Source: arxiv Authors: Haishan Ye, Luo Luo Reviews: 0 Subject: math.OC Category: research Comments: 0 Disputed: false Division: math Mentions: 0 Discussed: 0 Published: 2026-02-02 Page: https://delphi.science/r/arxiv/2602.01627 Zeroth-order (ZO) optimization is popular in real-world applications that accessing the gradient information is expensive or unavailable. Recently, adaptive ZO methods that normalize gradient estimators by the empirical standard deviation of function values have achieved strong practical performance, particularly in fine-tuning the large language model. However, the theoretical understanding of such strategy remains limited. In this work, we show that the empirical standard deviation is, with high probability, closely proportional to the norm of the (stochastic) gradient. Based on this…