A simple guide from Machine Learning outputs to statistical criteria

The paper offers a very discussion of the way uncertainties could be included in analyses involving neural networks. While I am not sure the topic of the solutions are completely new, I think the paper can be very useful for a phenomenological audience. I especially like the way it discusses the use of anomaly searches, a topic which is not widely discussed in the available literature. In addition, the paper is written for an audience which is different from the usual experimental papers on the topic, so it will be useful for quite a few readers.

There are not really groundbreaking new ideas in the paper, but there are very nice examples. While for SciPost Physics this lack of new ideas might be a problem, I am fine with publishing the paper in Core. Also, we should keep in mind that lots of model building papers or experimental analyses or ML-studies include no new ideas or in any way unexpected results.

I added whole lot of comments on the presentation in the attached pdf file. They include possible clarifications and also critical comments. The problem is that I cannot upload the file, it is just a Mac-annotated version of the original pdf file, so I am not sure what to do. I will send it to the authors directly for now, to avoid further delays.

Please go through those comments and consider them, in cases where I did not get it, please feel free to ignore them and tell me off...

My apologizes for the very late report!

This paper describes how to take the output of a neural network and use it for hypothesis testing in the context of collider physics. While the manuscript is mostly well-written, I am not sure what this paper adds to the literature. There is nothing special about neural networks - the paper describes how one takes a fixed observable and then uses simulations to create histograms of the observable to compute p-values. This is common knowledge and so I am unsure what is new in this paper. From the title, I was expecting to see some discussion of how one can use network outputs directly as test statistics instead of indirectly by first making histograms to compute likelihood ratios. This subject is also well-documented in other papers, but there could be room for additional innovation, such as in the area of uncertainty quantification (which is also not really discussed in detail since the entire topic of nuisance parameters are relegated to references). I am sorry if I have misunderstood the paper, but in the present form, I cannot recommend publication in SciPost Physics.