SciPost Submission Page
Simplified derivations for high-dimensional convex learning problems
by David G. Clark, Haim Sompolinsky
Submission summary
| Authors (as registered SciPost users): | David Clark |
| Submission information | |
|---|---|
| Preprint Link: | https://arxiv.org/abs/2412.01110v4 (pdf) |
| Date submitted: | 2025-02-11 14:34 |
| Submitted by: | Clark, David |
| Submitted to: | SciPost Physics Lecture Notes |
| Ontological classification | |
|---|---|
| Academic field: | Physics |
| Specialties: |
|
| Approach: | Theoretical |
Abstract
Statistical-physics calculations in machine learning and theoretical neuroscience often involve lengthy derivations that obscure physical interpretation. We present concise, non-replica derivations of key results and highlight their underlying similarities. Using a cavity approach, we analyze high-dimensional learning problems: perceptron classification of points and manifolds, and kernel ridge regression. These problems share a common structure--a bipartite system of interacting feature and datum variables--enabling a unified analysis. For perceptron-capacity problems, we identify a symmetry that allows derivation of correct capacities through a naïve method.
Current status:
Reports on this Submission
Report
Summary : These lecture notes revisit three celebrated problems in high-dimensional statistical learning, first studied in their respective works [5,8,9] through the lens of the replica method of statistical physics, using a cavity approach. The computation presents the advantage of being less lengthy, and overall more intuitive. It leverages the observation that all these problems admit reformulations with a bipartite structure.
Evaluation : As such, these notes propose a concise and insightful approach, and will prove of interest to researchers working on these topics. The manuscript is very well written, and sufficient discussion of all technical steps is provided. I list a few minor presentation comments below, but recommend that the work be accepted, even in its current state.
Comments:
- more explanations on the self-averaging of the self-responses (e.g. below (33)) could prove helpful.
- to the best of my reading, the expression (58) for the number of supporting points is not established before (58), and could gain to be briefly discussed.
-"due to the bipartite structure, perturbations to other datum variables do not affect the [cavity variable]": is this statement true to leading order or in general ? If the former, it would be clearer to make the precision.
Recommendation
Publish (easily meets expectations and criteria for this Journal; among top 50%)