Signal region combination with full and simplified likelihoods in MadAnalysis 5

We thank the referee for the careful assessment of our work. To the questions regarding section 3 (the validation of individual analyses), we reply as follows:

*) ATLAS-SUSY-2018-31, purpose of showing LO analysis: it is not uncommon that (exploratory) phenomenological studies use LO cross sections, as computing higher-order ones is CPU-time consuming and, depending on the model, not always straightforward (e.g., when the available calculations are for certain limits only). Also, in the case of the MSSM, the reference cross sections provided by the LHC SUSY Cross Sections WG are for the particular simplified-model assumptions and do not apply to more general MSSM scenarios. We think it is useful to illustrate the effect in one example. Showing LO results for all analyses discussed in section 3 would clearly be an unwarranted proliferation of plots, but one example can be useful to our mind.

*) ATLAS-SUSY-2018-04, and why we can reproduce the expected limit but fail with the observed one: in fact the agreement of the expected limit curves is somewhat accidental. As explained in the paper, the acceptance\times efficiency values from the MadAnalysis5 implementation of this analysis are roughly 30% higher than those from ATLAS. Therefore, the excluded cross section is smaller, and the observed limit on the stau mass stronger than in the official ATLAS result. For the expected limit, however, this effect is compensated by the fact that we report "apriori" (pre-fit) expected values, while ATLAS seemingly reports "aposteriori" (post-fit) expected values. The difference between pre-fit and post-fit is discussed in the last paragraph of section 2.1. Incidentally, for this particular analysis, the difference between pre-fit and post-fit background numbers is just of the right size to bring the ATLAS and MA5 expected limit curves in agreement. Had we used post-fit expected values, we would see roughly the same over-exclusion as for the observed limit. For better illustration, the attached "supplement.pdf" contains a concrete example.
We have added a remark in the paper (at the end of the last paragraph discussing ATLAS-SUSY-2018-04) to make this clear.

*) CMS-SUS-16-039: why in the 3-body decay region the limit derived with MA5 doesn't match the CMS one. In fact we had not explicitly simulated the 3-body decays with off-shell W and Z bosons. We thought this was clear from the figure caption and plot title, but agree that it is confusing that the limit curves don't match. We have now updated Figure 5 properly including also the off-shell region. The CMS exclusion contour is now reproduced well in both the high mass and the 3-body decay regions.

*) CMS-SUS-19-006, over-exclusion in particular in the expected limit: In reply to the referee's concern, we modified the last sentence of the relevant paragraph (see resubmission letter)

The requested plots showing the ratio of MA5/CMS excluded cross sections, for observed and expected limits, are attached. We agree that a detailed comparison of upper limits on a cross section obtained in a recast and with those provided officially is a very good way to assess the quality of an implementation of any specific recasting program. However, this kind of validation lies outside the scope of the present paper, which only focuses on the statistical modelling. We therefore decided not to include such plots and the related discussion in our manuscript.

As an aside, we note the information provided by CMS is sometimes a mix of pre-fit and post-fit background numbers (e.g., background numbers of events being pre-fit, but the covariance matrix being post-fit, or vice-versa) and the distinction is not always made clear. We are discussing this problem with the CMS collaboration, but do not want to enter into this discussion in the present paper.

Attachment:

supplement.pdf