SciPost Submission Page
Dense Hopfield Networks in the Teacher-Student Setting
by Robin Thériault, Daniele Tantari
Submission summary
| Authors (as registered SciPost users): | Daniele Tantari · Robin Thériault |
| Submission information | |
|---|---|
| Preprint Link: | scipost_202402_00010v2 (pdf) |
| Code repository: | https://github.com/RobinTher/Dense_Associative_Network |
| Date submitted: | 2024-05-21 16:42 |
| Submitted by: | Thériault, Robin |
| Submitted to: | SciPost Physics |
| Ontological classification | |
|---|---|
| Academic field: | Physics |
| Specialties: |
|
| Approaches: | Theoretical, Computational |
Abstract
Dense Hopfield networks with p-body interactions are known for their feature to prototype transition and adversarial robustness. However, theoretical studies have been mostly concerned with their storage capacity. We derive the phase diagram of pattern retrieval in the teacher-student setting of p-body networks, finding ferromagnetic phases reminiscent of the prototype and feature learning regimes. On the Nishimori line, we find the critical amount of data necessary for pattern retrieval, and we show that the corresponding ferromagnetic transition coincides with the paramagnetic to spin-glass transition of p-body networks with random memories. Outside of the Nishimori line, we find that the student can tolerate extensive noise when it has a larger p than the teacher. We derive a formula for the adversarial robustness of such a student at zero temperature, corroborating the positive correlation between number of parameters and robustness in large neural networks. Our model also clarifies why the prototype phase of p-body networks is adversarially robust.
Author indications on fulfilling journal expectations
- Provide a novel and synergetic link between different research areas.
- Open a new pathway in an existing or a new research direction, with clear potential for multi-pronged follow-up work
- Detail a groundbreaking theoretical/experimental/computational discovery
- Present a breakthrough on a previously-identified and long-standing research stumbling block
List of changes
Sections: Divided section 3 into 3.1 and 3.2. Split 4.1 into 4.1 and 4.2. Divided ex-section 4.4 (now 4.5 because of renumbering) into 4.5.1 and 4.5.2.
Line 51-53: Clarified a sentence in the introduction.
Line 83-85: Added a few lines describing additions to Section 2.
Line 136 - 160: Added a discussion about RSB, dynamical 1RSB and the spin-glass phase in the direct model.
Lines 188 - 189, 204, 247: Clarified the scaling of $\alpha$ at a few places.
Lines 193 - 194, 200, 228: Added references pointing to Fig. 1 to clarify the meaning of $T_{\mathrm{crit}}$.
Line 201: Added a reminder that $p^* = p$.
Lines 226 - 228: Explained why we cannot have $q^* \neq 0 $ and $m \neq 0$ at the same time.
Lines 269, 277, 295, 296: Clarified that we need $T > T_{\mathrm{crit}}$ for the argument of section 4.2 to hold.
Lines 321 - 324: Added a brief discussion about RSB and dynamical 1RSB in the inverse model (and how they relate to the direct model).
Lines 494 - 496: Added a comparison between adversarial attacks and random perturbations.
Lines 550 - 553: Added a few lines about learning more than one pattern.
Fig. 2: Specified that the plot is on the Nishomori line and added a reference pointing to Fig. 1 in order to explain how to find the $eR$ transition temperature from the $\alpha = 0$ axis. Added phase transition lines of nRSB where $n > 1$, 1RSB and d1RSB in the direct model. Modified the caption correspondingly.
Fig. 4: Defined $\varepsilon$.
Citations: Citation 26 is now referred to as K \& H's work instead of Krotov's work. Added seven references to support the new discussion at lines 136-160.
Anonymous on 2024-05-22 [id 4506]
In the list of changes, we state that we added the multiple-RSB, 1RSB and d1RSB transitions of the direct model to Fig. 2. In reality, we added the multiple-RSB and 1RSB lines to Fig. 1, and the 1RSB and d1RSB lines to Fig. 2. We apologize for the inaccuracy.