SciPost Submission Page
Resonance Searches with Machine Learned Likelihood Ratios
by Jacob Hollingsworth and Daniel Whiteson
This is not the latest submitted version.
Submission summary
| Authors (as registered SciPost users): | Jacob Hollingsworth |
| Submission information | |
|---|---|
| Preprint Link: | scipost_202003_00050v2 (pdf) |
| Date submitted: | 2020-12-09 01:29 |
| Submitted by: | Hollingsworth, Jacob |
| Submitted to: | SciPost Physics |
| Ontological classification | |
|---|---|
| Academic field: | Physics |
| Specialties: |
|
| Approach: | Phenomenological |
Abstract
We demonstrate the power of machine-learned likelihood ratios for resonance searches in a benchmark model featuring a heavy Z' boson. The likelihood ratio is expressed as a function of multivariate detector level observables, but rather than being calculated explicitly as in matrix-element-based approaches, it is learned from a joint likelihood ratio which depends on latent information from simulated samples. We show that bounds drawn using the machine learned likelihood ratio are tighter than those drawn using a likelihood ratio calculated from histograms.
Author comments upon resubmission
Reviewer 1 raised some concerns over the unconverged behavior in the tails of the invariant mass distribution in Figure 2, specifically remarking that the discrepancies in this region are larger than the improvement in the bulk region. We believe it is worth mentioning that, by normalizing each bin to the number of data points within each bin, the figure exaggerates the relevance of this effect in determining the likelihood ratio of a set of events. Because significantly more events will lie in the bulk region than the tails, the improvements in this region will contribute significantly more to the likelihood ratio than the noise in the tails when summing over a set of events.
Reviewer 1 asked about the smooth behavior seen in Figure 5, compared to the relatively noisy behavior in Figure 4. As the reviewer suggested, this smoother behavior is because the neural network interpolates in theta-space, whereas the histogram approach is uninformed by the behavior at neighboring grid points.
Reviewer 1 also suggests applying this to a more complex model where we would be able to leverage the full high dimensional information. We have considered alternative, more complex models for the subject of future work. It is worth noting that our claim that all information is contained in a two dimensional subset of features can only be made as a result of the higher dimensional analysis using the full jet 4-momenta. In this way, the analysis performed in the manuscript fully leverages the available high dimensional information.
Reviewer 1 raised concerns over the sentence "By evaluating the squared matrix element...". We agree with the reviewer here that the "inverse" problem of inferring theory parameters or Wilson coefficients requires a large number of data points in spite of the morphing structure due to showering and detector simulation. Our statement here comments on the relative ease of the "forward" problem of calculating squared matrix elements (or equivalently, joint likelihood ratios) which takes place at the parton level. For a given event z, the morphing structure allows one to only evaluate the squared matrix element at some number of benchmark points in theta-space in order to infer the value at any point in theta-space. Since resonance searches do not benefit from a morphing structure due to the nonlinear dependence on mass, we must evaluate the squared matrix element of each event on a grid in theta-space in order to reproduce a similar degree of coverage.
Reviewer 2 inquired about additional event selection criterion that may have been applied to the data set. We did not apply any further cuts or selection criteria to the data set beyond those mentioned in the manuscript.
Reviewer 2 asked how we may incorporate data-driven background estimates into the analysis. One way this may be achieved is to use data-driven normalization of Monte Carlo samples to construct a partially data-driven background estimation. Joint likelihood ratios for background events, which would not possess a theta dependence, are given by a constant related to the overall cross-sections for theta and theta_0.
Reviewer 2 brings up insightful comments about Equations (2.1) and (2.2). We do not believe that the quantities p(x|z) and p(z) must necessarily be well-defined in experiment for our approach to be valuable. In Eq. (2.2), we do assume that the probability distributions p(x|z) are the same for the numerator and the denominator. However, this distribution is defined by the programs that simulate event showering and detection. These simulations will always converge to the same distribution for a given event, and so these distributions are well defined in simulation. There is of course an additional assumption that the simulation suite accurately models event generation and detection that occurs in experiment. This is a fundamental assumption that cannot be forgone, though is common in the literature and can be aided by profiling over nuisance parameters.
Reviewer 2 also brought up remarks on the notation used to denote sets of events, random variables, and events that are an instantiation of the random variable. We have changed the notation for sets of events to \Chi so that our notation is not at odds with the convention in statistics that X is a random variable and x it's realization. The notation used in our manuscript was borrowed from previous publications on these methods (specifically [23-26]). In order to not deviate from this precedence, we have added a statement clarifying that x will be used interchangeably to mean a random variable and its realization, and its meaning should be clear from context.
Reviewer 2 asked about our motivations for choosing R=.5. This choice was made so that we could check for consistency of our data with older datasets.
We would like to once again thank the reviewers for their time and consideration, and hope that we have adequately addressed their concerns and questions.
List of changes
We have added a paragraph to the end of Section II, beginning "There are a handful...", which addresses both reviewer's concerns about uncertainty estimates and mismodelling of the neural networks.
We have added a horizontal line at log r = 0 in Figure 2 in order to guide the eye.
We have also added a sentence in the caption of this figure that highlights the unconverged behavior that is seen in the tail of the distribution. These changes were made in response to reviewer 1's feedback on this figure.
We have changed the sentence on page 4, which previously read "In this work, we focus initially on a qualitative assessment of the application of machine learned likelihood ratios to resonance searches." to the new sentence "We focus on a qualitative assessment of the application of machine learned likelihood ratios to resonance searches."
We have added a new sentence to the conclusion (beginning "Since these methods scale well...") which addresses the scalability of the algorithm. We have also added a citation to a high dimensional application in a different field, which uses high-dimensional images as the input data. These changes were made in response to reviewer 2's request that we comment on the scalability of the algorithm.
We have changed "Typical" to "Many" in the sentence that read "Typical searches for BSM resonances..." in the second paragraph of the introduction. This change was made in response to reviewer 2's critique of this sentence.
We have moved the citation block [14-17] in response to reviewer 2's comment from the sentence "However, relying soley on the invariant mass..." to the next paragraph, after the sentence "Certain methods have been developed to overcome this information loss."
In the sentence on page 3 which formerly read "Using a standard suite of programs, we can easily produce a set of events..." we have removed the word "easily" and included the word "simulated", which were changes suggested by reviewer 2.
We have introduced a new symbol, \Chi, to represent a set of events, which was previously represented with X.
We have changed the labels and color bars in Figures 3-5 to make them more readable.
Current status:
Reports on this Submission
Report
Thank you for taking into account my feedback! I think the paper is basically ready for publication. However, I would like to briefly follow up on two points:
- Uncertainty quantification. I am glad to see the new paragraph at the end of Sec. 2. However, I think it is important to state clearly that you are assuming that the data can be described by p(x|\theta) where \theta may include a parameter of interest (like a mass) as well as any nuisance parameters quantifying systematic uncertainties. This is true for all simulation-based inference problems, but it is very important and thus worth stating in this context. Furthermore, in the last part of the paragraph, you talk about how you would check if your likelihood ratio estimator was not exactly correct. Would you need to add an uncertainty to the corresponding p-values? How would you do that?
- Background estimation. While a data-driven approach where simulation shapes are normalized in sidebands is common, it is not used for dijet searches. Those searches basically all use parameters fits directly to data, with no simulation. I don't think it is trivial to integrate your approach into that context, but I also appreciate that it is not really relevant for this paper where you are illustrating a new method. Can you please add a remark about this somewhere in the paper?