The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider

Aarrestad, Thea; van Beekveld, Melissa; Bona, Marcella; Boveia, Antonio; Caron, Sascha; Davies, Joe; de Simone, Andrea; Doglioni, Caterina; Duarte, Javier; Farbin, Amir; Gupta, Honey; Hendriks, Luc; Heinrich, Lukas A.; Howarth, James; Jawahar, Pratik; Jueid, Adil; Lastow, Jessica; Leinweber, Adam; Mamuzic, Judita; Merényi, Erzsébet; Morandini, Alessandro; Moskvitina, Polina; Nellist, Clara; Ngadiuba, Jennifer; Ostdiek, Bryan; Pierini, Maurizio; Ravina, Baptiste; Ruiz de Austri, Roberto; Sekmen, Sezen; Touranakou, Mary; Vaškeviciute, Marija; Vilalta, Ricardo; Vlimant, Jean-Roch; Verheyen, Rob; White, Martin; Wulff, Eric; Wallin, Erik; Wozniak, Kinga A.; Zhang, Zhongyi

doi:10.21468/SciPostPhys.12.1.043

SciPost Physics

The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider

T. Aarrestad, M. van Beekveld, M. Bona, A. Boveia, S. Caron, J. Davies, A. De Simone, C. Doglioni, J. M. Duarte, A. Farbin, H. Gupta, L. Hendriks, L. Heinrich, J. Howarth, P. Jawahar, A. Jueid, J. Lastow, A. Leinweber, J. Mamuzic, E. Merényi, A. Morandini, P. Moskvitina, C. Nellist, J. Ngadiuba, B. Ostdiek, M. Pierini, B. Ravina, R. Ruiz de Austri, S. Sekmen, M. Touranakou, M. Vaškevičiūte, R. Vilalta, J. R. Vlimant, R. Verheyen, M. White, E. Wulff, E. Wallin, K. A. Wozniak, Z. Zhang

SciPost Phys. 12, 043 (2022) · published 28 January 2022

doi: 10.21468/SciPostPhys.12.1.043
pdf
Submissions/Reports

Abstract

We describe the outcome of a data challenge conducted as part of the Dark Machines Initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenged aims at detecting signals of new physics at the LHC using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of >1 Billion simulated LHC events corresponding to $10~\rm{fb}^{-1}$ of proton-proton collisions at a center-of-mass energy of 13 TeV. We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC, and provide our benchmark dataset for future studies at https://www.phenoMLdata.org. Code to reproduce the analysis is provided at https://github.com/bostdiek/DarkMachines-UnsupervisedChallenge.

TY  - JOUR
PB  - SciPost Foundation
DO  - 10.21468/SciPostPhys.12.1.043
TI  - The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider
PY  - 2022/01/28
UR  - https://scipost.org/SciPostPhys.12.1.043
JF  - SciPost Physics
JA  - SciPost Phys.
VL  - 12
IS  - 1
SP  - 043
A1  - Aarrestad, Thea
AU  - van Beekveld, Melissa
AU  - Bona, Marcella
AU  - Boveia, Antonio
AU  - Caron, Sascha
AU  - Davies, Joe
AU  - de Simone, Andrea
AU  - Doglioni, Caterina
AU  - Duarte, Javier
AU  - Farbin, Amir
AU  - Gupta, Honey
AU  - Hendriks, Luc
AU  - Heinrich, Lukas A.
AU  - Howarth, James
AU  - Jawahar, Pratik
AU  - Jueid, Adil
AU  - Lastow, Jessica
AU  - Leinweber, Adam
AU  - Mamuzic, Judita
AU  - Merényi, Erzsébet
AU  - Morandini, Alessandro
AU  - Moskvitina, Polina
AU  - Nellist, Clara
AU  - Ngadiuba, Jennifer
AU  - Ostdiek, Bryan
AU  - Pierini, Maurizio
AU  - Ravina, Baptiste
AU  - Ruiz de Austri, Roberto
AU  - Sekmen, Sezen
AU  - Touranakou, Mary
AU  - Vaškeviciute, Marija
AU  - Vilalta, Ricardo
AU  - Vlimant, Jean-Roch
AU  - Verheyen, Rob
AU  - White, Martin
AU  - Wulff, Eric
AU  - Wallin, Erik
AU  - Wozniak, Kinga A.
AU  - Zhang, Zhongyi
AB  - We describe the outcome of a data challenge conducted as part of the Dark Machines Initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenged aims at detecting signals of new physics at the LHC using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of >1 Billion simulated LHC events corresponding to $10~\rm{fb}^{-1}$ of proton-proton collisions at a center-of-mass energy of 13 TeV. We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC, and provide our benchmark dataset for future studies at https://www.phenoMLdata.org. Code to reproduce the analysis is provided at https://github.com/bostdiek/DarkMachines-UnsupervisedChallenge.
ER  -

@Article{10.21468/SciPostPhys.12.1.043,
	title={{The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider}},
	author={T. Aarrestad and M. van Beekveld and M. Bona and A. Boveia and S. Caron and J. Davies and A. De Simone and C. Doglioni and J. M. Duarte and A. Farbin and H. Gupta and L. Hendriks and L. Heinrich and J. Howarth and P. Jawahar and A. Jueid and J. Lastow and A. Leinweber and J. Mamuzic and E. Merényi and A. Morandini and P. Moskvitina and C. Nellist and J. Ngadiuba and B. Ostdiek and M. Pierini and B. Ravina and R. Ruiz de Austri and S. Sekmen and M. Touranakou and M. Vaškevičiūte and R. Vilalta and J. R. Vlimant and R. Verheyen and M. White and E. Wulff and E. Wallin and K. A. Wozniak and Z. Zhang},
	journal={SciPost Phys.},
	volume={12},
	pages={043},
	year={2022},
	publisher={SciPost},
	doi={10.21468/SciPostPhys.12.1.043},
	url={https://scipost.org/10.21468/SciPostPhys.12.1.043},
}

Cited by 60

De Vita et al., Deep Learning to improve Experimental Sensitivity and Generative Models for Monte Carlo simulations for searching for New Physics in LHC experiments
EPJ Web of Conf. 295, 09009 (2024) [Crossref]
Roche et al., Nanosecond anomaly detection with decision trees and real-time application to exotic Higgs decays
Nat Commun 15, 3527 (2024) [Crossref]
Kasieczka et al., Anomaly detection under coordinate transformations
Phys. Rev. D 107, 015009 (2023) [Crossref]
Belis et al., Quantum anomaly detection in the latent space of proton collision events at the LHC
Commun Phys 7, 334 (2024) [Crossref]
Metodiev et al., Anomaly detection in collider physics via factorized observables
Phys. Rev. D 110, 055012 (2024) [Crossref]
Dillon et al., Self-supervised anomaly detection for new physics
Phys. Rev. D 106, 056005 (2022) [Crossref]
Fraser et al., Challenges for unsupervised anomaly detection in particle physics
J. High Energ. Phys. 2022, 66 (2022) [Crossref]
Aad et al., Anomaly detection search for new resonances decaying into a Higgs boson and a generic new particle X in hadronic final states using s=13 TeV pp collisions with the ATLAS detector
Phys. Rev. D 108, 052009 (2023) [Crossref]
Dillon et al., A normalized autoencoder for LHC triggers
SciPost Phys. Core 6, 074 (2023) [Crossref]
Verheyen, Event Generation and Density Estimation with Surjective Normalizing Flows
SciPost Phys. 13, 047 (2022) [Crossref]
Finke et al., Tree-based algorithms for weakly supervised anomaly detection
Phys. Rev. D 109, 034033 (2024) [Crossref]
Chen et al., Resonant anomaly detection with multiple reference datasets
J. High Energ. Phys. 2023, 188 (2023) [Crossref]
Larkoski, QCD masterclass lectures on jet physics and machine learning
Eur. Phys. J. C 84, 1117 (2024) [Crossref]
Chen et al., Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks
Electronics 12, 786 (2023) [Crossref]
Gonski et al., High-dimensional anomaly detection with radiative return in e+e− collisions
J. High Energ. Phys. 2022, 156 (2022) [Crossref]
Algren et al., Decorrelation using optimal transport
Eur. Phys. J. C 84, 579 (2024) [Crossref]
Letizia et al., Learning new physics efficiently with nonparametric methods
Eur. Phys. J. C 82, 879 (2022) [Crossref]
Mikuni et al., High-dimensional and permutation invariant anomaly detection
SciPost Phys. 16, 062 (2024) [Crossref]
Bhardwaj et al., Foundations of automatic feature extraction at LHC–point clouds and graphs
Eur. Phys. J. Spec. Top. 233, 2619 (2024) [Crossref]
Buhmann et al., Full phase space resonant anomaly detection
Phys. Rev. D 109, 055015 (2024) [Crossref]
Hallin et al., Resonant anomaly detection without background sculpting
Phys. Rev. D 107, 114012 (2023) [Crossref]
Alvarez et al., Unsupervised Quark/Gluon Jet Tagging With Poissonian Mixture Models
Front. Artif. Intell. 5, 852970 (2022) [Crossref]
Sengupta et al., Improving new physics searches with diffusion models for event observables and jet constituents
J. High Energ. Phys. 2024, 109 (2024) [Crossref]
Hashemi et al., Deep generative models for detector signature simulation: A taxonomic review
Reviews in Physics 12, 100092 100092 (2024) [Crossref]
Canelli et al., Autoencoders for semivisible jet detection
J. High Energ. Phys. 2022, 74 (2022) [Crossref]
Freytsis et al., Anomaly detection in the presence of irrelevant features
J. High Energ. Phys. 2024, 220 (2024) [Crossref]
Buss et al., What's anomalous in LHC jets?
SciPost Phys. 15, 168 (2023) [Crossref]
Mikuni et al., Online-compatible unsupervised nonresonant anomaly detection
Phys. Rev. D 105, 055006 (2022) [Crossref]
Benato et al., Shared Data and Algorithms for Deep Learning in Fundamental Physics
Comput Softw Big Sci 6, 9 (2022) [Crossref]
Dillon et al., Learning Latent Jet Structure
Symmetry 13, 1167 (2021) [Crossref]
Aad et al., Search for New Phenomena in Two-Body Invariant Mass Distributions Using Unsupervised Machine Learning for Anomaly Detection at s=13 TeV with the ATLAS Detector
Phys. Rev. Lett. 132, 081801 (2024) [Crossref]
Alvi et al., Quantum anomaly detection for collider physics
J. High Energ. Phys. 2023, 220 (2023) [Crossref]
Krzyzanska et al., Simulation-based anomaly detection for multileptons at the LHC
J. High Energ. Phys. 2023, 61 (2023) [Crossref]
Golling et al., Flow-enhanced transportation for anomaly detection
Phys. Rev. D 107, 096025 (2023) [Crossref]
Sengupta et al., CURTAINs flows for flows: Constructing unobserved regions with maximum likelihood estimation
SciPost Phys. 17, 046 (2024) [Crossref]
Kösters et al., Benchmarking energy consumption and latency for neuromorphic computing in condensed matter and particle physics
1, 016101 (2023) [Crossref]
Govorkova et al., Autoencoders on field-programmable gate arrays for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider
Nat Mach Intell 4, 154 (2022) [Crossref]
Hallin et al., Classifying anomalies through outer density estimation
Phys. Rev. D 106, 055006 (2022) [Crossref]
Caron et al., Mixture-of-Theories training: can we find new physics and anomalies better by mixing physical theories?
J. High Energ. Phys. 2023, 4 (2023) [Crossref]
Dillon et al., Anomalies, representations, and self-supervision
SciPost Phys. Core 7, 056 (2024) [Crossref]
Ostdiek, Deep Set Auto Encoders for Anomaly Detection in Particle Physics
SciPost Phys. 12, 045 (2022) [Crossref]
Bermot et al.,
, 331 (2023) [Crossref]
Chekanov et al., Enhancing the hunt for new phenomena in dijet final states using anomaly detection filters at the high-luminosity large Hadron Collider
Eur. Phys. J. Plus 139, 237 (2024) [Crossref]
Birman et al., Data-directed search for new physics based on symmetries of the SM
Eur. Phys. J. C 82, 508 (2022) [Crossref]
Belis et al., Machine learning for anomaly detection in particle physics
Reviews in Physics 12, 100091 100091 (2024) [Crossref]
Bradshaw et al., Creating simple, interpretable anomaly detectors for new physics in jet substructure
Phys. Rev. D 106, 035014 (2022) [Crossref]
Chekanov et al., Event-Based Anomaly Detection for Searches for New Physics
Universe 8, 494 (2022) [Crossref]
Golling et al., The interplay of machine learning-based resonant anomaly detection methods
Eur. Phys. J. C 84, 241 (2024) [Crossref]
Bonilla et al., Jets and Jet Substructure at Future Colliders
Front. Phys. 10, 897719 (2022) [Crossref]
Finke et al., Boosting mono-jet searches with model-agnostic machine learning
J. High Energ. Phys. 2022, 15 (2022) [Crossref]
Caron et al., Rare and Different: Anomaly Scores from a combination of likelihood and out-of-distribution models to detect new physics at the LHC
SciPost Phys. 12, 077 (2022) [Crossref]
Leigh et al., PC-JeDi: Diffusion for particle cloud generation in high energy physics
SciPost Phys. 16, 018 (2024) [Crossref]
Bai et al., Non-resonant anomaly detection with background extrapolation
J. High Energ. Phys. 2024, 59 (2024) [Crossref]
De et al., Deep learning techniques for imaging air Cherenkov telescopes
Phys. Rev. D 107, 083026 (2023) [Crossref]
Krause et al., Anomaly detection with flow-based fast calorimeter simulators
Phys. Rev. D 110, 035036 (2024) [Crossref]
Ngairangbam et al., Anomaly detection in high-energy physics using a quantum autoencoder
Phys. Rev. D 105, 095004 (2022) [Crossref]
Bickendorf et al., Combining resonant and tail-based anomaly detection
Phys. Rev. D 109, 096031 (2024) [Crossref]
Heneka et al.,
472, 251 (2024) [Crossref]
Schuhmacher et al., Unravelling physics beyond the standard model with classical and quantum anomaly detection
Mach. Learn.: Sci. Technol. 4, 045031 (2023) [Crossref]
Oleksiyuk et al., Cluster Scanning: a novel approach to resonance searches
J. High Energ. Phys. 2024, 163 (2024) [Crossref]

Authors / Affiliations: mappings to Contributors and Organizations

See all Organizations.

¹ Thea Aarrestad,
² Melissa van Beekveld,
³ Marcella Bona,
⁴ Antonio Boveia,
⁵ Sascha Caron,
³ Joe Davies,
⁶ ⁷ Andrea de Simone,
⁸ Caterina Doglioni,
⁹ Javier Duarte,
¹⁰ Amir Farbin,
¹¹ Honey Gupta,
⁵ Luc Hendriks,
¹ Lukas A. Heinrich,
¹² James Howarth,
¹ ¹³ Pratik Jawahar,
¹⁴ Adil Jueid,
⁸ Jessica Lastow,
¹⁵ Adam Leinweber,
¹⁶ Judita Mamuzic,
¹⁷ Erzsébet Merényi,
¹⁸ Alessandro Morandini,
⁵ Polina Moskvitina,
⁵ Clara Nellist,
¹⁹ ²⁰ Jennifer Ngadiuba,
²¹ ²² Bryan Ostdiek,
¹ Maurizio Pierini,
¹² Baptiste Ravina,
¹⁶ Roberto Ruiz de Austri,
²³ Sezen Sekmen,
¹ ²⁴ Mary Touranakou,
¹² Marija Vaškeviciute,
²⁵ Ricardo Vilalta,
¹⁹ Jean-Roch Vlimant,
²⁶ Rob Verheyen,
¹⁵ Martin White,
⁸ Eric Wulff,
⁸ Erik Wallin,
¹ ²⁷ Kinga A. Wozniak,
⁵ Zhongyi Zhang

Funders for the research work leading to this publication