SciPost logo

SciPost Submission Page

Learning Summary Statistics for Bayesian Inference with Autoencoders

by Carlo Albert, Simone Ulzega, Firat Ozdemir, Fernando Perez-Cruz, Antonietta Mira

This Submission thread is now published as

Submission summary

Authors (as registered SciPost users): Carlo Albert
Submission information
Preprint Link: https://arxiv.org/abs/2201.12059v2  (pdf)
Code repository: https://renkulab.io/gitlab/bistom/enca-inca
Date accepted: 2022-07-01
Date submitted: 2022-05-24 15:03
Submitted by: Albert, Carlo
Submitted to: SciPost Physics Core
Ontological classification
Academic field: Physics
Specialties:
  • Statistical and Soft Matter Physics
Approach: Computational

Abstract

For stochastic models with intractable likelihood functions, approximate Bayesian computation offers a way of approximating the true posterior through repeated comparisons of observations with simulated model outputs in terms of a small set of summary statistics. These statistics need to retain the information that is relevant for constraining the parameters but cancel out the noise. They can thus be seen as thermodynamic state variables, for general stochastic models. For many scientific applications, we need strictly more summary statistics than model parameters to reach a satisfactory approximation of the posterior. Therefore, we propose to use the inner dimension of deep neural network based Autoencoders as summary statistics. To create an incentive for the encoder to encode all the parameter-related information but not the noise, we give the decoder access to explicit or implicit information on the noise that has been used to generate the training data. We validate the approach empirically on two types of stochastic models.

Author comments upon resubmission

Based on the reviewer comments, we have substantially reworked our manuscript. We have replied to all the issues raised by the two reviewers using the reply option on the submission page.

List of changes

1) We have modified our definition of the concentration property. In the previous version, we had used a definition that worked for discretized variables only.

2) We have removed a minor inconsistency between the manuscript and the code. The nonlinear function we are using in Eq.9 is f(x) = x(1-x), not f(x) = x^2exp(-x) as previously written in the manuscript.

3) We added a distinguishing feature of our approach to the Introduction:

“What distinguishes our approach from other information-theoretic machine learning algorithms such as the ones recently presented in \cite{cvitkovic_2019_MinSuffStatML} and \cite{chen2020MLsufficientStats}, is the possibility to use {\em explicit} noise information. This makes our approach also applicable in situations where only noise-, but no parameter-information is available, as could be the case in observed rather than simulated data. As an example, we might want to use it to remove rain-features (rain playing the role of the noise) from hydrological runoff time-series in order to distill runoff-features that stem from the catchments themselves. We will examine this application in future publications.”

4) At the end of Section 2, we further extended the analogy between summary statistics learning and thermodynamics. In particular, we stress the importance of considering the entropy of summary statistics, when doing inference with high-dimensional correlated data.

5) We added a new figure (Fig. 5), showing the reconstruction capabilities of the decoder for the first model.

6) We added new inference results, for the case when only 2 summary statistics are learned, for the first model: Figs. 5 and 6 (left panel).

7) In Appendix 6.2, we are much more explicit about how the Autoencoders are trained. We also added a short discussion on the scalability of our methods.

Published as SciPost Phys. Core 5, 043 (2022)


Reports on this Submission

Report #2 by Anonymous (Referee 1) on 2022-6-11 (Invited Report)

Report

The others answered to the question I raised and in particular clarify points that were originally missing in the manuscript. As such, I recommend publication.

  • validity: -
  • significance: -
  • originality: -
  • clarity: -
  • formatting: -
  • grammar: -

Report #1 by Anonymous (Referee 2) on 2022-6-7 (Invited Report)

Strengths

1. The authors improved their manuscript given the referee's remark. I feel that the article is clearer now.
2. They add some other results confirming their previous results.

Report

The authors have answered positively to the referee's comment. On my side, as not expert on the field, I see no particular reason not to accept the manuscript.

  • validity: good
  • significance: ok
  • originality: good
  • clarity: good
  • formatting: good
  • grammar: good

Login to report or comment