SciPost Submission Page
Spey: smooth inference for reinterpretation studies
by Jack Y. Araz
This is not the latest submitted version.
This Submission thread is now published as
Submission summary
| Authors (as registered SciPost users): | Jack Araz |
| Submission information | |
|---|---|
| Preprint Link: | https://arxiv.org/abs/2307.06996v2 (pdf) |
| Code repository: | https://github.com/SpeysideHEP/spey |
| Date submitted: | 2023-07-21 15:04 |
| Submitted by: | Araz, Jack |
| Submitted to: | SciPost Physics |
| Ontological classification | |
|---|---|
| Academic field: | Physics |
| Specialties: |
|
| Approaches: | Theoretical, Computational, Phenomenological |
Abstract
Analysing statistical models is at the heart of any empirical study for hypothesis testing. We present a new cross-platform Python-based package which employs different likelihood prescriptions through a plug-in system, enabling the statistical inference of hypotheses. This framework empowers users to propose, examine, and publish new likelihood prescriptions without the need for developing a new inference system. Within this package, we propose a new simplified likelihood prescription which surpasses the approximation accuracy of its predecessors by incorporating asymmetric uncertainties. Furthermore, our package facilitates the integration of various likelihood combination routines, thereby broadening the scope of independent studies through a meta-analysis. By remaining agnostic to the source of the likelihood prescription and the signal hypothesis generator, our platform allows for the seamless implementation of packages with different likelihood prescriptions, fostering compatibility and interoperability.
Current status:
Reports on this Submission
Report #2 by Anonymous (Referee 1) on 2023-10-29 (Invited Report)
- Cite as: Anonymous, Report on arXiv:2307.06996v2, delivered 2023-10-29, doi: 10.21468/SciPost.Report.8013
Strengths
Presents a tested and accessible solution to a common and important set of problems in the field.
Is well written and clear and comes with links to well-documented open-source code.
Weaknesses
Focusses too much on exclusion limits, at least in the introduction. More emphasis early on on potential discovery, and on fitting parameters (as shown in the later neutrino example) would improve the paper.
Report
Reproducible interpretation and re-interpretation of statistical results based on experimental data is a topic of growing and crucial importance. If sucessfully adopted and supported, this package could facilitate significantly better communication and rigour in this area.
Requested changes
1. See my comment on the introduction above.
2. Eq 2 and 3 are a bit confusing in terms of the subscript/superscript on n. The index i moves from subscript to superscript, and seems to maybe sum over s and b at one point and over bins at the next. Also, lambda is said to be a function of n_s and n_b but is shown as a function of mu and theta. I realise this is pretty standard stuff but please check for clarity.
3 page 5 please clarify what "units determined by the user" means. How are these units used and reported by Spey? Does Spey even know or care what they are?
Report #1 by Anonymous (Referee 2) on 2023-10-19 (Invited Report)
- Cite as: Anonymous, Report on arXiv:2307.06996v2, delivered 2023-10-19, doi: 10.21468/SciPost.Report.7969
Report
In this paper, the author presents a new software tool designed for statistical inference in reinterpretation studies. The tool's versatility enables users to create various likelihood approximations, integrate pre-existing packages, and combine these likelihoods under multiple assumptions. Additionally, the author introduces a new likelihood approximation technique to improve the precision of previously proposed simplified approaches. All techniques are rigorously tested throughout the paper, and their respective limitations are clearly delineated. The improvements and techniques provided in this paper have been showcased to enhance the limits of an MSSM scenario, demonstrating that even with existing analyses, one can extend the boundaries of current searches.
Furthermore, the author allows users to extend the coverage of the package in terms of the versatility of likelihood proposals and adds the means to cite their independent extensions through the package. This feature enhances the tool's utility and promotes collaboration and the sharing of new statistical methodologies. Hence, I believe the manuscript is valuable for HEP and any empirical sciences.
I have a minor structural observation concerning the paper. While the conclusion is well-structured, providing a concise summary and outlining future steps, a similar organisation is missing in the introduction. I recommend that the author revise the introduction to underscore the current challenges within the field and elucidate how this tool offers a viable solution to these issues.
I thank the referee for their analysis of the paper; I have updated the introduction in accordance with the referee's request, which can be found in Arxiv v3. I hope this version will satisfy the referee for the publication of the manuscript.
Thanks
Best regards
Author: Jack Araz on 2023-12-04 [id 4167]
(in reply to Report 2 on 2023-10-29)I thank the referee for the detailed analysis of the paper. I have updated the manuscript in accordance with their comments, which can be found in Arxiv v3.
1) I have updated the introduction and included an additional example in section 2 to emphasize these. 2) All indices have been moved to superscript, and the definition of lambda has been extended. Additionally, I added a footnote stating that Spey does not need this particular definition, i.e. lambda can be a function of anything since this is defined by the user. Still, current implementations are only accepting a number of yields so far. 3) This has been clarified in footnote 8; indeed, Spey does not care/know the unit of the cross-section. This implementation originated from a request from recasting packages to compute the upper limit on cross-section instead of only on POI (which is essentially the upper limit on POI times cross-section value, hence can easily be computed externally using the
poi_upper_limit()function).I hope these corrections and clarifications will satisfy the referee for the publication of the manuscript.
Thanks Best regards