SciPost Submission Page
Rare and Different: Anomaly Scores from a combination of likelihood and out-of-distribution models to detect new physics at the LHC
by Sascha Caron, Luc Hendriks, Rob Verheyen
This is not the latest submitted version.
This Submission thread is now published as
Submission summary
| Authors (as registered SciPost users): | Luc Hendriks · Rob Verheyen |
| Submission information | |
|---|---|
| Preprint Link: | https://arxiv.org/abs/2106.10164v4 (pdf) |
| Code repository: | https://github.com/l-hendriks/combined-anomaly-detection-code |
| Data repository: | https://zenodo.org/record/3961917 |
| Date submitted: | 2021-07-21 07:04 |
| Submitted by: | Hendriks, Luc |
| Submitted to: | SciPost Physics |
| Ontological classification | |
|---|---|
| Academic field: | Physics |
| Specialties: |
|
| Approaches: | Computational, Phenomenological |
Abstract
We propose a new method to define anomaly scores and apply this to particle physics collider events. Anomalies can be either rare, meaning that these events are a minority in the normal dataset, or different, meaning they have values that are not inside the dataset. We quantify these two properties using an ensemble of One-Class Deep Support Vector Data Description models, which quantifies differentness, and an autoregressive flow model, which quantifies rareness. These two parameters are then combined into a single anomaly score using different combination algorithms. We train the models using a dataset containing only simulated collisions from the Standard Model of particle physics and test it using various hypothetical signals in four different channels and a secret dataset where the signals are unknown to us. The anomaly detection method described here has been evaluated in a summary paper [1] where it performed very well compared to a large number of other methods. The method is simple to implement and is applicable to other datasets in other fields as well.
Current status:
Reports on this Submission
Report #1 by Anonymous (Referee 2) on 2021-9-20 (Invited Report)
- Cite as: Anonymous, Report on arXiv:2106.10164v4, delivered 2021-09-20, doi: 10.21468/SciPost.Report.3548
Report
The authors study new approaches to anomaly detection in the context of the DarkMachines Anomaly Score challenge. They examine different combinations of One-Class Deep Support Vector Data Description models (for detecting out of sample anomalies) and autoregressive flows (for detecting rare in-sample anomalies). They quantify the performance of these combinations using the DarkMachines Anomaly Score datasets, using versions of the significance improvement (SI) score. Since the DarkMachines datasets contain many different signals, they consider the median, max and min significance improvements across all the different signals.
This work has some novel and interesting aspects and advances our understanding of a very important problem (model independent searches for new physics at the LHC). However, before it can be published, there are several questions about the methodology that I believe must be addressed:
- The Deep SVDD model seems like it could trivially learn the constant function, and then it would have no anomaly detection power. What prevents it from just mapping all inputs (regardless of signal or background) trivially to a constant? The authors should explain why this apparent, obvious failure mode does not happen for their Deep SVDD model.
- The models considered in this work were trained on a background-only sample, and then evaluated on background and various signals. If I understand correctly, for both training and evaluation the backgrounds were drawn from the same distribution (i.e. produced with the same generator and detector simulation). In that case, the anomaly scores being computed here could be very misleading. In particular, if the SM background in the data is not sufficiently well modeled by simulation, the sensitivity to new physics could be significantly worsened (it might flag the entire dataset as anomalous). The authors should discuss this issue at length and as quantitatively as possible. Why do they expect the SI metrics they computed here to be at all relevant if the background in the data is mismodeled by simulation?
- Anomaly scores are only one component of a successful new physics search strategy. Obviously, background estimation is another, equally important component. As far as I could tell, there was no mention of background estimation anywhere in this paper. The authors should include a discussion about how they imagine they could combine their anomaly score with an accurate method of background estimation.