Resonance Searches with Machine Learned Likelihood Ratios

1. The use of highdimensional event information for the construction of likelihood ratios clearly outperforms the histgram based approach.

2. Further developpment of neural network based likelihood ratio estimates for resonances, after it was established for effective field theories.

1. The paper lacks a discussion of the limitations and liability of their approach.

The present paper « Resonance Searches with Machine Learned Likelihood Ratios «  demonstrates how to increase the performance of resonance seaches by using neural networks for the estimation of likelihood ratios. The use of neural networks to learn the likelihood ratio has been demonstrated previously for EFTs. Since the matrix element can in those cases be expressed in terms of a polynomial in the Wilson coefficients, the Z’ benchmark considered in this paper represents a more complex model due to the appearance of a resonance.

The paper is well written and clearly structured. With the growing importance of machine learning methods in high energy physics, the presented performance evaluation is relevant for publication. In this context I suggest a more detailed analysis of the obtained results especially with respect to limitations of the neural networks. In particular since uncertainty estimations for neural networks are not yet established, such a discussion is necessary to estimate under which conditions the results are reliable.

1. Fig. 2 illustrates the differences between the different approaches. The paper would profit from a discussion of the observed effects.

I would naively expect the dashed and the solid blue line to be the same when the distribution of events over the non displayed dimensions can not be distinguished for the chosen values of theta. However in that case I would expect those lines to coincide as well with the grey line. Starting at 350 GeV the ML method consistenly favors the $\theta_0$ hypothesis. The discrepancy in this region is larger than the modest improvement of the ML method with respect to the histogram method seen in the bulk region. I assume this is due to lower statistics in the tail, where the network has not sufficiently learned the underlying structure. Please explain the observed curves.

A horizontal line at zero would be helpful to guide the eye.

2. Fig. 5 seems to show a much smoother behaviour compared to Fig. 4 thanks to the interpolation properties of the neural network. Is this correct?

3. A main point of the paper is to include the full information at detector level instead of 1 dimensional information in the histogram. In the following the authors demonstrated that the 2 dimensional plane of $m_{jj}$ and $\Delta y_{jj}$ fully covers already all information contained in the two 4-dimensional jets.
The authors should consider to include an extension where they could choose a more complex model to profit from the highdimensional information.

4. The authors state that "by evaluating the squared matrix element of an event at a handful of “benchmark” points in the parameter space, one is able to infer the squared matrix element for arbitrary theory parameters."
While this is true in theory, in practice the inference of the polynomial parameters - in particular of all interference terms - after showering and detector simulation requires much more than a handful of benchmark points.

5. page 4: The sentence starting with ‘In this work’ does not seem to be complete.

1- Applies a machine-learning-based generalization of the matrix element method to a resonance search for the first time.

2- The paper clearly describes what was done.

1- There is no discussion of uncertainties. This is not just incorporating the usual nuisance parameters; for simulation-based inference, it is essential that the data actually live in the simulation manifold. A more specific suggestion is in the report.

2 - This analysis in practice uses data-driven background estimates. I did not see how this simulation-based approached meshes with a data-driven background estimate.

This paper is a nice application of simulation-based inference to a resonance search in collider-based particle physics. The presentation is clear and the results adds a useful contribution to the literature. Below are some comments and suggestions that would be good to implement before this paper is published.

Major

- You say "The full event information is given by the set of four-momenta of every particle in the final state. Summarizing this relatively high dimensional information with invariant mass plausibly results in a significant amount of information loss.", but then you say "For each event, our observables x consist of the four-vector of each reconstructed jet". So you are still only using 8 dimensions of the high-dimensional space. Can you comment on the scalability of your algorithm?

- "We also calculate the likelihood ratio from histograms" -> do you apply any other event selection? It seems like dijet searches ofen apply some asymmetry requirements on pT and/or rapidity to select dijet s-channel events.

- "A possible critique of this interpretation is that the histogram based approach does not perform as well as ALICE because it requires use of a binned PDF..." -> this is a nice test!

- Thinking a little about uncertainties, it would be helpful if you could add something about the folloing:

-> What if p(x|\theta) is not correct (=nature) for any \theta?
-> What if p_\theta given in Eq. (3.1) is not correct? (if r is wrong, then your procedure may be suboptimal for precision, but this would also make your p-values wrong for accuracy)

- Related: the dijet analysis is always done using a data-driven background estimation technique. How does that interplay with your analysis?

Minor

- "Typical searches for BSM resonances involve fitting signal and background spectra in the invariant mass of the final state particles..." -> I know what you mean, but this is not technically correct. For example, most SUSY searches do not do bump hunts, but they are looking for BSM resonances.

- "However, relying soley on the invariant mass..." -> 14]-[17] use more than just the invariant mass?

- "...we can easily produce a set of events..." -> often it is rather computationally expensive to sample so maybe drop "easily"? Also, I would add the word "simulated" before "events".

- Eq. (2.1): On the one hand, the law of total propability lets you write p(x) = \int dz p(x|z) p(z) for any z, but on the other hand, "partons" are unphysical and factorization theorems for the LHC are not exact (except for Drell-Yan) so p(x|z) and p(z) are not physical quantities. Is this relevant for your ratio in Eq. (2.2) - is there an implicit assumption that the unphysical schemes are the same for the top and bottom? You say that "z" is not well-defined in an experiment - I would say that "z" is not well-defined period (it is not observable in an experiment).

- Your notation is a bit confusing. X is usually a random variable, but here, X is a set and x is a random variable. However, x is also used interchangeably as a realization of the random variable x. I would suggest that X be the random variable, x is the realization, and a new symbol be used for a set of events.

- Eq. (2.9): I was a little confused when I saw this, because I thought you were saying that this is p(\theta)/p(\theta_0) because the lefthandside doesn't depend on X. Perhaps writing p(X,\theta,\theta_0) would make this clearer?

- [37] belongs to the sentence before. It obviously doesn't matter, but why did you pick R = 0.5? No one has used that radius parameter for years now.

Please see the above bulleted list. Additionally, the labels in Fig. 3-5 are impossible to see - please make them relatively bigger.