SciPost logo

SciPost Submission Page

Spey: smooth inference for reinterpretation studies

by Jack Y. Araz

This is not the latest submitted version.

This Submission thread is now published as

Submission summary

Authors (as registered SciPost users): Jack Araz
Submission information
Preprint Link: https://arxiv.org/abs/2307.06996v2  (pdf)
Code repository: https://github.com/SpeysideHEP/spey
Date submitted: 2023-07-21 15:04
Submitted by: Araz, Jack
Submitted to: SciPost Physics
Ontological classification
Academic field: Physics
Specialties:
  • High-Energy Physics - Phenomenology
Approaches: Theoretical, Computational, Phenomenological

Abstract

Analysing statistical models is at the heart of any empirical study for hypothesis testing. We present a new cross-platform Python-based package which employs different likelihood prescriptions through a plug-in system, enabling the statistical inference of hypotheses. This framework empowers users to propose, examine, and publish new likelihood prescriptions without the need for developing a new inference system. Within this package, we propose a new simplified likelihood prescription which surpasses the approximation accuracy of its predecessors by incorporating asymmetric uncertainties. Furthermore, our package facilitates the integration of various likelihood combination routines, thereby broadening the scope of independent studies through a meta-analysis. By remaining agnostic to the source of the likelihood prescription and the signal hypothesis generator, our platform allows for the seamless implementation of packages with different likelihood prescriptions, fostering compatibility and interoperability.

Current status:
Has been resubmitted

Reports on this Submission

Report #2 by Anonymous (Referee 2) on 2023-10-29 (Invited Report)

  • Cite as: Anonymous, Report on arXiv:2307.06996v2, delivered 2023-10-29, doi: 10.21468/SciPost.Report.8013

Strengths

Presents a tested and accessible solution to a common and important set of problems in the field.

Is well written and clear and comes with links to well-documented open-source code.

Weaknesses

Focusses too much on exclusion limits, at least in the introduction. More emphasis early on on potential discovery, and on fitting parameters (as shown in the later neutrino example) would improve the paper.

Report

Reproducible interpretation and re-interpretation of statistical results based on experimental data is a topic of growing and crucial importance. If sucessfully adopted and supported, this package could facilitate significantly better communication and rigour in this area.

Requested changes

1. See my comment on the introduction above.
2. Eq 2 and 3 are a bit confusing in terms of the subscript/superscript on n. The index i moves from subscript to superscript, and seems to maybe sum over s and b at one point and over bins at the next. Also, lambda is said to be a function of n_s and n_b but is shown as a function of mu and theta. I realise this is pretty standard stuff but please check for clarity.
3 page 5 please clarify what "units determined by the user" means. How are these units used and reported by Spey? Does Spey even know or care what they are?

  • validity: high
  • significance: high
  • originality: good
  • clarity: high
  • formatting: excellent
  • grammar: good

Author:  Jack Araz  on 2023-12-04  [id 4167]

(in reply to Report 2 on 2023-10-29)

I thank the referee for the detailed analysis of the paper. I have updated the manuscript in accordance with their comments, which can be found in Arxiv v3.

1) I have updated the introduction and included an additional example in section 2 to emphasize these. 2) All indices have been moved to superscript, and the definition of lambda has been extended. Additionally, I added a footnote stating that Spey does not need this particular definition, i.e. lambda can be a function of anything since this is defined by the user. Still, current implementations are only accepting a number of yields so far. 3) This has been clarified in footnote 8; indeed, Spey does not care/know the unit of the cross-section. This implementation originated from a request from recasting packages to compute the upper limit on cross-section instead of only on POI (which is essentially the upper limit on POI times cross-section value, hence can easily be computed externally using the poi_upper_limit() function).

I hope these corrections and clarifications will satisfy the referee for the publication of the manuscript.

Thanks Best regards

Report #1 by Anonymous (Referee 1) on 2023-10-19 (Invited Report)

  • Cite as: Anonymous, Report on arXiv:2307.06996v2, delivered 2023-10-19, doi: 10.21468/SciPost.Report.7969

Report

In this paper, the author presents a new software tool designed for statistical inference in reinterpretation studies. The tool's versatility enables users to create various likelihood approximations, integrate pre-existing packages, and combine these likelihoods under multiple assumptions. Additionally, the author introduces a new likelihood approximation technique to improve the precision of previously proposed simplified approaches. All techniques are rigorously tested throughout the paper, and their respective limitations are clearly delineated. The improvements and techniques provided in this paper have been showcased to enhance the limits of an MSSM scenario, demonstrating that even with existing analyses, one can extend the boundaries of current searches.

Furthermore, the author allows users to extend the coverage of the package in terms of the versatility of likelihood proposals and adds the means to cite their independent extensions through the package. This feature enhances the tool's utility and promotes collaboration and the sharing of new statistical methodologies. Hence, I believe the manuscript is valuable for HEP and any empirical sciences.

I have a minor structural observation concerning the paper. While the conclusion is well-structured, providing a concise summary and outlining future steps, a similar organisation is missing in the introduction. I recommend that the author revise the introduction to underscore the current challenges within the field and elucidate how this tool offers a viable solution to these issues.

  • validity: -
  • significance: -
  • originality: -
  • clarity: -
  • formatting: -
  • grammar: -

Author:  Jack Araz  on 2023-12-04  [id 4168]

(in reply to Report 1 on 2023-10-19)

I thank the referee for their analysis of the paper; I have updated the introduction in accordance with the referee's request, which can be found in Arxiv v3. I hope this version will satisfy the referee for the publication of the manuscript.

Thanks
Best regards

Login to report or comment