Id: arxiv/2607.15450 Url: https://arxiv.org/abs/2607.15450 Tags: math.ST, cs.LG, stat.ML Score: 0 Title: Prediction-Only Distillation in Linear and Logistic Regression Domain: math.ST Source: arxiv Authors: Hien Dang, Pratik Patil, Alessandro Rinaldo Reviews: 0 Subject: math.ST Category: research Comments: 0 Disputed: false Division: math Mentions: 0 Discussed: 0 Published: 2026-07-16 Page: https://delphi.science/r/arxiv/2607.15450 Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional…