pyBumpHunter: A model independent bump hunting tool in Python for High Energy Physics analyses

Dear Mohamed Rameez,

Many thanks for reading our paper in detail and for all your comments.
They have been very useful to improve the quality of our paper and we took most of them into account.
Please see below our answers to your questions and comments.
We uploaded a revised version of the paper on arXiv featuring all the mentioned modification (https://arxiv.org/pdf/2208.14760v3.pdf).


1.
We have found a paper that proposes a Bayesian view of the Look Elsewhere Effect.
https://iopscience.iop.org/article/10.1088/1475-7516/2020/10/009/meta https://arxiv.org/abs/2007.13821
This reference has been added in the discussion about the principle of the BumpHunter algorithm without going into the details. Please see the footnote at the bottom of page 3:
*“The method proposed by the BumpHunter algorithm relies purely on frequentist considerations, although there also exists different methods relying on a Bayesian approach [9].”*

However, we couldn’t find a relevant reference on the effect of going from a one dimensional bump hunt to a multidimensional one. Therefore, as you do not seem to insist too much on this, we would leave it as is.

2.
The choice of the negative sign for the significance of a deficit is indeed a convention.
In the case of a global p-value greater than 0.5, the sign of the significance changes and should become opposite to what is expected according to the convention.
We added the following sentence in the paragraph in order to clarify the behavior in the case of a search for a deficit (1st paragraph of page 4):
*“However, when considering a search for a deficit, the sign of the significance is flipped, so it will be negative in case the global p-value is smaller than 0.5 and positive otherwise.”*

3.
Our point in this paragraph is that the global p-value depends on the count of pseudo-data histograms that passes a given criteria (test statistic greater than a threshold).
If this count becomes very low, the behavior of the global p-value becomes inaccurate.
This problem is illustrated in another study that we have made recently and that is also available on arXiv: https://arxiv.org/abs/2211.07446
In the case when the count becomes exactly 0, the global significance goes to infinity.
This is a problem of available statistics, the higher the global significance, the more pseudo-data histograms are required.
We decided to rephrase the paragraph and add the paper mentioned above in the references.
Here is the part that has been modified (bottom of page 4):
*“This way of computing the global p-value is efficient to account for the look elsewhere effect, but has one limitation: the minimum non-null global p-value that can be computed depends on the number of pseudo-data distributions that were generated. When the count of pseudo-data distributions for which the test statistic passes the threshold ($N_{>data}$) becomes small, the obtained global p-value becomes inaccurate. In the extreme case when $N_{>data}$ becomes exactly 0, it causes the global significance to become infinite. [...]. Another approach to address this problem is discussed in [10].”*

4.
Using a mean+-68% interval (1 sigma) would indeed make more sense and this feature will be added to future releases of pyBumpHunter.
However we decided to keep this description in the paper since it corresponds to what the current stable release of pyBumpHunter does.

5.
Here we propose a new way to normalize the reference background histogram to the observed data in the case when the true normalization factor of the background is not known a priori (as for a floating normalization).
We made it more explicit in the text and also added a new figure (figure 8) in order to better illustrate the normalization procedure.
Here is the sentences we added to the text (page 10)
*“The figure 8 illustrates the principle of the side-band normalization procedure. Side-bands are used in the original proposal of the algorithm but do not treat the problem of the floating normalization of the reference background, as we propose.”*

Best regards,

Louis Vaslin, for the authors