SciPost Submission Page
Learning Summary Statistics for Bayesian Inference with Autoencoders
by Carlo Albert, Simone Ulzega, Firat Ozdemir, Fernando Perez-Cruz, Antonietta Mira
This Submission thread is now published as
Submission summary
| Authors (as registered SciPost users): | Carlo Albert |
| Submission information | |
|---|---|
| Preprint Link: | https://arxiv.org/abs/2201.12059v2 (pdf) |
| Code repository: | https://renkulab.io/gitlab/bistom/enca-inca |
| Date accepted: | 2022-07-01 |
| Date submitted: | 2022-05-24 15:03 |
| Submitted by: | Albert, Carlo |
| Submitted to: | SciPost Physics Core |
| Ontological classification | |
|---|---|
| Academic field: | Physics |
| Specialties: |
|
| Approach: | Computational |
Abstract
For stochastic models with intractable likelihood functions, approximate Bayesian computation offers a way of approximating the true posterior through repeated comparisons of observations with simulated model outputs in terms of a small set of summary statistics. These statistics need to retain the information that is relevant for constraining the parameters but cancel out the noise. They can thus be seen as thermodynamic state variables, for general stochastic models. For many scientific applications, we need strictly more summary statistics than model parameters to reach a satisfactory approximation of the posterior. Therefore, we propose to use the inner dimension of deep neural network based Autoencoders as summary statistics. To create an incentive for the encoder to encode all the parameter-related information but not the noise, we give the decoder access to explicit or implicit information on the noise that has been used to generate the training data. We validate the approach empirically on two types of stochastic models.
Author comments upon resubmission
List of changes
1) We have modified our definition of the concentration property. In the previous version, we had used a definition that worked for discretized variables only.
2) We have removed a minor inconsistency between the manuscript and the code. The nonlinear function we are using in Eq.9 is f(x) = x(1-x), not f(x) = x^2exp(-x) as previously written in the manuscript.
3) We added a distinguishing feature of our approach to the Introduction:
“What distinguishes our approach from other information-theoretic machine learning algorithms such as the ones recently presented in \cite{cvitkovic_2019_MinSuffStatML} and \cite{chen2020MLsufficientStats}, is the possibility to use {\em explicit} noise information. This makes our approach also applicable in situations where only noise-, but no parameter-information is available, as could be the case in observed rather than simulated data. As an example, we might want to use it to remove rain-features (rain playing the role of the noise) from hydrological runoff time-series in order to distill runoff-features that stem from the catchments themselves. We will examine this application in future publications.”
4) At the end of Section 2, we further extended the analogy between summary statistics learning and thermodynamics. In particular, we stress the importance of considering the entropy of summary statistics, when doing inference with high-dimensional correlated data.
5) We added a new figure (Fig. 5), showing the reconstruction capabilities of the decoder for the first model.
6) We added new inference results, for the case when only 2 summary statistics are learned, for the first model: Figs. 5 and 6 (left panel).
7) In Appendix 6.2, we are much more explicit about how the Autoencoders are trained. We also added a short discussion on the scalability of our methods.
Published as SciPost Phys. Core 5, 043 (2022)
Reports on this Submission
Strengths
1. The authors improved their manuscript given the referee's remark. I feel that the article is clearer now.
2. They add some other results confirming their previous results.
Report
The authors have answered positively to the referee's comment. On my side, as not expert on the field, I see no particular reason not to accept the manuscript.