SciPost Submission Page
Dense Hopfield Networks in the Teacher-Student Setting
by Robin Thériault, Daniele Tantari
Submission summary
| Authors (as registered SciPost users): | Daniele Tantari · Robin Thériault |
| Submission information | |
|---|---|
| Preprint Link: | scipost_202402_00010v1 (pdf) |
| Code repository: | https://github.com/RobinTher/Dense_Associative_Network |
| Date submitted: | 2024-02-06 12:12 |
| Submitted by: | Thériault, Robin |
| Submitted to: | SciPost Physics |
| Ontological classification | |
|---|---|
| Academic field: | Physics |
| Specialties: |
|
| Approaches: | Theoretical, Computational |
Abstract
Dense Hopfield networks with p-body interactions are known for their feature to prototype transition and adversarial robustness. However, theoretical studies have been mostly concerned with their storage capacity. We derive the phase diagram of pattern retrieval in the teacher-student setting of p-body networks, finding ferromagnetic phases reminiscent of the prototype and feature learning regimes. On the Nishimori line, we find the critical amount of data necessary for pattern retrieval, and we show that the corresponding ferromagnetic transition coincides with the paramagnetic to spin-glass transition of p-body networks with random memories. Outside of the Nishimori line, we find that the student can tolerate extensive noise when it has a larger p than the teacher. We derive a formula for the adversarial robustness of such a student at zero temperature, corroborating the positive correlation between number of parameters and robustness in large neural networks. Our model also clarifies why the prototype phase of p-body networks is adversarially robust.