Unbinning global LHC analyses

Dear authors

Thank you for your efforts and the clearly detailed draft. However, I would appreciate more explanations. Please consider this review as coming from someone who is not an expert in the method and is trying to understand it based on your paper in order to apply it to a realistic analysis.

There are several SBI application papers referenced, ranging from other phenotypic studies to experimental applications. What is the specific innovation of this paper concerning the method? From the title and introduction, I understand that the focus of this paper is not on the practical application presented, but rather on the technique itself. However, in the conclusion, I read that the innovation lies in combining a few channels. I confess that I did not understood the message.

I believe the paper would benefit from a simpler perspective regarding the formulas for the proposed loss function and their interpretation. It should be accessible for someone like a student analyst. For example, in the context of EFT analysis at the LHC, one of the most complex aspects of optimization and statistical analysis is that modifications to the Wilson coefficients alter the signal shape. Keeping this in mind, I found a few concepts unclear and would appreciate further clarification.

• What kind of signal MC do you add to your training? An SM-like one only? Some mixture of shapes with alternative Wilson coefficients and some sort of interpolation?
• How are the changes in signal topology taken into account in the training? Can some distributions be shown for specific examples, for example, one of the variables that figures in the histogram-based fit showing that signal kinematics changes with the fitted Wilson coefficients?
• What do you actually fit in the case of the SBI? Can some distributions be drawn in signal and MC for a few parameter values?

I understand that for each value of the Wilson coefficients in the scans, there will be different distributions for the signal and background when trying to extract the signal. This is the reason you are calculating upper limits. How would your method work if experimentalists wanted to compute likelihood scans for the Wilson coefficients? Is this a limitation of the technique?

I believe that comparing the SBI with the histogram-based method may not be entirely fair. The histogram-based approach, as you've pointed out, is one-dimensional, while there are multi-dimensional methods available that do not involve SBI, do not specifically target Wilson coefficients. A more equitable comparison regarding the benefits of your loss function introduction would involve examining the results of neural network training with and without it. In the scenario without the loss function, you could focus on the Standard Model-like signal without considering loss functions that depend on Wilson coefficients.

In conclusion, the many factors overlooked on this work are acceptable for a phenomenological-qualitative assessment in the context of SBI application in the context of SMEFT in dibosons. If there is an improvement in all individual channels from using the technique, it per se indicates that the statistical combination is going to be better. That shall be enough for the argument for the usage of the method in a realistic setup.

When attempting to quantify the gain from the combination using measures such as the increase in sensitivity on the combination based on the amount of additional data that would be nescessary in using other method—I believe that approach is somewhat misguided. To make such claims one needs to at least have an idea if the sensitivity arrived with histogram based in this work is close to what is found on the experiment.

1 - Be more clear with how SBI results are evaluated on data analysis, how signal shape changes with the Wilson coefficients are modeled (or ignored), add few distributions
2 - Understand a more fair comparison point with SBI, or at least compare minimally the sensitivity of the histogram based results with the realistic analyses

Ask for major revision

1. Recent years SMEFT has risen as one of the most prominent framework to interpret the wealth of data from LHC. However, the many parameters that SMEFT offers and its intricate dependencies create a challenging analysis task. SBI methodology––although can be quite complicated––manages to tackles this challenge, however, is also unclear whether the improvements are truly worth the effort. The paper shows a good proof-of-concept example comparing how SBI method is superior against recent WW and WZ conventional SMEFT analyses approach and the widely agreed upon STXS approach in the Higgs group, thereby providing strong motivation for experimentalists to seriously consider the SBI approach in their main approach. The paper shows that with SBI method, one can gain factor of 2 to 4 in luminosity gain, which is a significant improvement, especially considering the fact that LHC would have operated for nearly 30 years! This paper can serve as a good reference study that people can point to in the future to show as a proof-of-concept of why the SBI approach is worth the trouble for years to come.

2. The paper also achieves this by considering multi-sector EFT fits. Often there is a question when combining multiple sector whether the sectors can be easily disentangled by final state, and thereby not much gain to be made other than providing orthogonal constraints to whichever operator the process is sensitive to. However, the correlation effects on Figure 9 shows interesting features, and the SBI method truly shining in its performance.

1. The study was done without systematics and thereby the total rate measurement sensitivity maybe slightly overestimated. However, I do not believe that that is a major concern for the overall result of the paper.

2. The study aims to compare histogram approach to SBI, but the choice of the variables and binning may have been suboptimal, and the gap in the improvements in some cases may be bridged by some simple optimization in the histogram case. However, even the cases for a few analyses that do use the "optimized" variable still show significant improvements (30%) which compared to luminosity gain, is still quite significant (1.8 more lumi). This was noted and therefore, I believe that it is also not a major concern for the overall result of the paper. In general, the variable choice is a bit tricky. For example, the comparison to STXS binning is both fair and an unfair one. It is fair in the sense that this is the currently widely accepted binning choice, but also an unfair one as we know that it won't be the most optimal choice for the SMEFT. The mTWZ and the pTl1 choice for WZ and WW are chosen by what is used in the cited paper, therefore to some level it is somewhat fair. However, for WW case, it seems the cited reference uses some multivariate approach. So it begs the question how fair this variable choice of pTl1 is.

1. It does connect between Machine Learning and the SMEFT approach.

2. This serves as a good study that will lead to more SMEFT + SBI approach in the future.

3. and 4. are not met

1.​ “first precision-hadron collider in history”; I believe that Tevatron’s precision program on various electroweak measurements including W mass would be considered as a precision study as well. So I think calling this “first” would be a bit of a hyperbole. I suggest searching a different wording or phrase.

1. In general, loss function eq. 5 is not necessarily have to be exactly that if I understand correctly. So if this particular form—which is shown in reference [6]—perhaps, it is worth expanding and noting it can be done with other loss function, but this is one of many forms that work.

2. “we do not consider WZ backgrounds for our analysis.”; I think what is meant here is “backgrounds for WZ production”. I think it has a danger of sound like WZ is not considered, which is the signal in this case.

3. ttbb background modeling is often difficult and depending on how the ttbar samples were created and showered, ttbb can easily be double counted. I do not think for the main conclusion of the paper this caveat is a major factor of concern, but a few words on this would strengthen the clarity.

4. In figure 2, and similar ones, “small horizontal lines” I think is confusing, since the horizontal lines are not necessarily smaller, but rather it’s the smaller interval that is in vertical direction. For clarity I suggest a different wording.

5. Figure 5, the most striking feature is shown in the cphiD vs. cphiW, where the paper identifies as pTW not capturing the differences. It’s true that this clearly shows that STXS is insufficient in binning strategy, but I wonder if there is another variable that the author can identify that would recover some of the differences, or whether truly one has to consider complex structure of multi-variate distributions to be able to recover the gain. If it is the latter, and if the paper could expand upon the discussion on it I think it would really make a strong case for SBI as the paper is concluding here.

6. Figure 7. The two ellipses and the why the center was not centered around zero when there are more backgrounds did not seem clearly explained to the reader.

7. Figure 7. I think the “in between” the two solid ellipses are the allowed region by 1 sigma. If this is somehow made a bit more clearer either through shading or if not shading to be consistent with the other plots, a bit more words in the caption than what is currently there would make it more clearer.

8. Figure 9. Especially when it comes to cphiW vs. cphiWB it seems that when multi-sector fits are being done the SBI truly shines in making individual measurements to work together. If only the histo approach was used, ZH would not have added much to the constraint but with SBI it does. I think this is quite interesting and if the author agrees I think would be worth pointing out.

Publish (easily meets expectations and criteria for this Journal; among top 50%)

1- Interesting first application of extending SBI to global LHC analyses. 2 - Clear flow/narrative 3 - Good use of benchmarks (traditional methods) to show improvements with SBI procedure.

See attached PDF document.

Ask for minor revision