Testing new physics models with global comparisons to collider measurements: the Contur toolkit

Measurements at particle collider experiments, even if primarily aimed at understanding Standard Model processes, can have a high degree of model independence, and implicitly contain information about potential contributions from physics beyond the Standard Model. The Contur package allows users to benefit from the hundreds of measurements preserved in the Rivet library to test new models against the bank of LHC measurements to date. This method has proven to be very effective in several recent publications from the Contur team, but ultimately, for this approach to be successful, the authors believe that the Contur tool needs to be accessible to the wider high energy physics community. As such, this manual accompanies the first user-facing version: Contur v2. It describes the design choices that have been made, as well as detailing pitfalls and common issues to avoid. The authors hope that with the help of this documentation, external groups will be able to run their own Contur studies, for example when proposing a new model, or pitching a new search.

Submitting Contur scans to a HPC systems using contur-batch 39 C. 6 Iteratively refining the scanned parameter space with contur-zoom 39

Introduction
The discovery of the Higgs boson was the capstone of decades of research, and cemented the validity of the Standard Model (SM) as our best understanding so far of the building blocks of the universe. The SM boasts a predictive track record worthy of its position as one of the triumphs of modern science. It led to the discovery of the vector bosons W and Z, the top quark, and the Higgs boson, and SM cross-section predictions -ranging across ten orders of magnitude from the inclusive jet cross-section at O (10 11 ) pb, to electroweak V V jj processes at O(10 −3 ) pb -have been found to agree with experimental data through decades of scrutiny, with no significant deviations. Despite this monumental achievement, the SM is ostensibly an approximation. Qualitative phenomena such as the cosmological matter-antimatter asymmetry, and astrophysical observations consistent with dark-matter and dark-energy contributions to cosmic structure and dynamics suggest directly that the SM is not the whole story. These indications are reinforced by technical issues within the SM such as the "unnatural" need for fine-tuning of its key parameters, and its formal incompatibility with relativistic gravity.
With the absence so far of evidence for electroweak-scale supersymmetry, or of obvious new resonances in measured spectra, the field of collider physics finds itself at a crossroads. For the first time in fifty years, there is no single guiding theory to motivate discoveries. On the other hand, the LHC has delivered the largest dataset ever collected in particle physics, with the promise of a dataset an order of magnitude larger to be delivered by the high-luminosity (HL) LHC in the coming years. A transition from a top-down, theory-driven approach to a bottom-up, data-driven one is needed if we are to use these data to achieve the widest possible coverage of possible extensions to the SM. The problem is that the field of particle physics does not currently work efficiently in data-driven mode. Searches may take years to produce and concentrate only on certain signatures of a handful of models at a time. These models may even already be excluded, since the new particles and interactions which they feature would have modified well-understood and measured SM spectra. What if we could harness the power of the hundreds of existing LHC measurements preserved in Rivet [1], to rapidly tell whether a model is already excluded? A more comprehensive approach to ruling out models could liberate person-power and resources to focus on the trickiest signatures. This is the purpose of Constraints On New Theories Using Rivet (Contur), a project first described in Ref. [2].
The Contur method has proven an effective and complementary approach to ruling out new physics models in a series of case studies [3][4][5][6], as well as in providing a "due diligence" check for newly proposed models [7,8]. Running a Contur-like scan of any newly proposed new-physics model, or when a new search is being designed, should be routine in experimental particle physics, and would potentially liberate search teams to focus on models which have not already been ruled out. This shortcut around models which -no matter how theoretically elegant -are already incompatible with model-independent observations will accelerate the feedback loop between theorists and experimentalists, and bring us more efficiently to the long-sought understanding of what lies beyond the SM.
The Contur code is now mature enough to turn this vision into a reality, and this manual is intended to accompany the first major user-facing release of the Contur code (Contur v2, tagged on Zenodo as Ref. [9]), so that theorists and experimentalists who are not Contur developers can use this technology to test new models themselves. The Contur homepage [10] provides links to source code as well as up-to-date installation and setup instructions.

Overview
This document is structured as follows: this section gives a general introduction to the Contur workflow and design philosophy. Section 3 deals with the relationship between Rivet and Contur, and how Rivet analyses in the Contur database are classified into orthogonal pools, with advice on adding new analyses. Section 4 runs through setting up and running Contur scans over a set of parameter points in a given model. Section 5 explains how Contur builds a likelihood function to perform the statistical analysis of the results, and how exclusion values are calculated and analysed. Section 6 takes the user through the various plotting and visualisation tools which come with Contur, to help validate and digest results of a scan. Finally, Section 7 concludes the manual. Some of the explanations and figures in these sections have been adapted from a PhD thesis partially focused on the development of Contur [11].
Several appendices are provided to give further detail on some functionality, as well as detailed examples. Appendix A provides a detailed flowchart which covers almost all aspects of the Contur package described in this manual. Appendix B provides the user with a complete didactic example of the analysis of a beyond-the-SM (BSM) model with Contur, using the Herwig [12] event generator. Appendix C provides detailed descriptions of the various helper executables and other utilities which are provided in the Contur package, including details about Contur Docker containers. Appendix D gives further details about the UFO [13] format, which is used to encapsulate the details of BSM models, while Appendix E details Contur compatibility with the SLHA [14,15] format. Appendix F documents how model parameter values can be provided to Contur via pandas DataFrame [16,17] objects. Appendix G provides further details about how to use generators other than Herwig with Contur. Finally, Appendix H provides further detail about the various databases and classifications which are used in the Contur workflow.

The Contur workflow
The basic premise of Contur is that modifications to the SM Lagrangian typically introduce changes to already well-understood and measured differential cross-sections. Therefore, if adding a beyond-SM component to the Lagrangian, i.e. a new interaction involving either SM or new BSM fields, would change a measured distribution beyond its experimental uncertainties, then, in simple terms "we'd already have seen it". This can be quantified more precisely in terms of statistical limit-setting, but the upshot is that if one can predict how a given BSM model would modify the hundreds of observables measured in existing LHC measurements, then it is already possible to exclude regions of its parameter space without the need for a dedicated search.
This perspective turns the immediate model-testing challenge from an experimental one into a computational and book-keeping one. Can we design a workflow to take a BSM model with a set of parameter values, generate simulated events from it, quickly infer the effect of those events in each bin of the LHC measurements to date, and compute the p-value (and hence exclusion status at some confidence level) for that model point? Can one then efficiently repeat that procedure over a range of parameter points, to determine the regions of parameter space which are excluded? Contur is a tool that implements such a process. It builds on several existing data formats, conventions and packages to achieve this goal, and automatically handles the steering of model parameters and associated book-keeping on the user's behalf. The basic workflow is illustrated schematically in Figure 1, and in much more detail in Figure 5 of Appendix A The first requirement is that the BSM model be implemented in a Monte Carlo event generator (MCEG) such that its parameters can be set, and simulated events generated for analysis. Historically this required manual coding, and hence focused on BSM models such as supersymmetry, technicolor, and new quarks and vector bosons, which were considered leading candidates for new physics before LHC operation. The Super-symmetric (SUSY) Les Houches Accord (SLHA [14,15]) format was developed as a MCEG-independent way of specifying the mass and decay spectra of such models, and is understood by many MCEGs. As the "obvious" BSM models waned and gave way to a much wider spectrum of possibilities, a complementary format -the Universal FeynRules Output [13] (UFO) -was developed to transmit not just parameter choices but the entire model, built up from a Python-based encoding of the BSM Lagrangian. The combination of UFO and SLHA files provides an industry-standard way to package the details of any BSM model, such that most MCEGs can interpret it without needing model-specific code. Its ubiquity means that theorists routinely publish UFO files Figure 1: A simplified schematic of the Contur workflow. The dotted box denotes the portion of the workflow that makes extensive use of external packages, affording multiple options, such as the choice of MCEG. These two steps are described together in Section 4, 'Sampling model parameters'. The next stage, taking physics observables as inputs to a statistical analysis, 'Evaluating the likelihood for the model' is described in Section 5. Finally some of the tools to visualise the output of the likelihood analysis, 'Visualisation of parameter space', are covered in Section 6. A much more detailed diagram is available in Figure 5 of Appendix A.
when proposing a new model, making them easy to study and test. Details on how to use a new UFO file as an input to Contur can be found in Appendix D, and use of SLHA-driven configurations in Appendix E.
MCEGs use the specified BSM model and parameters to simulate new-physics events in high energy collisions. In the default Contur workflow, the Herwig [12] event generator is used (see Appendix B for an example), but other event generators, such as MadGraph5 aMC@NLO [18] and Powheg [19] are also supported. Additionally, if events are already generated and parameter steering is therefore not required, Rivet and thus Contur can analyse events stored in HepMC [20,21] format. More details on support for various event generators in Contur are given in Appendix G.
The generated events are fed into Rivet (see Section 3), the output of which then corresponds to the extra BSM contribution which would have been present in any of the hundreds of spectra measured at the LHC so far, if the generated model existed in nature. The BSM component can then be compared to the size of the uncertainty for the measurement, and optionally to the SM expectation. Measurements are grouped into orthogonal pools (see Section 3.1), and Contur uses the best constraint from each pool to form a global exclusion measure for a given model at a given set of parameter values. The details of the statistical treatment can be found in Section 5.1.
This whole process typically takes under an hour for a single point on a single compute node. Repeated for a grid of parameter values, and running in parallel on a compute farm, Contur can determine in a few hours whether wide regions of a model's parameter space are still potentially viable, or already excluded by existing LHC measurements. The Contur package comes with plotting and visualisation tools to present and digest the results of a scan. These are discussed in Section 6 and Appendix C.

The Contur philosophy
Contur is designed to efficiently address the question "How compatible is a proposed physics model with published LHC results?" This question needs to be asked each time a new model is proposed. The ability to answer it depends on a number of factors.
Firstly one must define what is meant by "LHC results". Collider physics experiments produce a variety of different types of results, which can be broadly classified as follows.
1. Extraction of fundamental parameters of the SM, such as the W mass, the Weinberg angle, etc... Such results give experimental constraints on SM parameters which are calculable analytically in perturbative field theory.
2. Extraction of so-called inclusive quantities, such as (for example) the total production cross-section tt, or WW . This usually involves theory input to extrapolate into regions outside the acceptance of the measurement.
3. Measurements of fiducial particle-level observables. In other words, observables corrected for detector effects or "unfolded", but not extrapolated beyond detector acceptance.
Comparing predictions to such measurements requires the generation of simulated events, making use of perturbative field theory but also non-perturbative models, and numerical MC techniques, so that fiducial phase space selections may be applied to final-state particles.

Build detector level quantities
Detector level experimental observables (4) or derived model exclusions (5)

Contur
T y p ic a l e x p e r im e n t r e c a s t w o r k fl o w

Analytic perturbative calculation
Combine measurement data (3) and (4)* * Caveats exist on inclusion of (4) Figure 2: A schematic illustrating the levels at which data and theory may be compared in LHC physics. The vertical (downward) arrows show the direction of increasing complexity of the theoretical prediction; in the reverse direction, increasingly complex corrections must be applied to the data. Horizontal arrows show the comparison data available at each level.
4. Measurement of detector-level distributions. This is the most common type of result used in searches by ATLAS and CMS. They are faster to produce than unfolded results, since the step of validating the model-independence of the unfolding (but not of calibration) can be skipped. However they cannot be compared to theory without an additional detector simulation step.
5. Exclusion regions from searches, usually derived from the detector-level distributions mentioned above. These can sometimes be reinterpreted in terms of new models, but may have significant implicit model-dependence.
This categorisation is shown schematically in Figure 2. The direction of the arrow indicates increasing calculational complexity required of the theory to compare the result to SM predictions. At first, just an analytical calculation of the SM parameter is needed. Then, MC simulation at parton and particle-level are required. Finally, the effects of the detector must be modelled. The level of model assumption built into the experimental data increases in the opposite direction.
All interpretations, or re-interpretations, of results involve compromises and approximations. The Contur philosophy is to strive for speed and coverage of new models, at the expense of some precision and sensitivity. To do this, we focus primarily on fiducial, particle-level measurements, as a compromise between model dependence and detector independence: that is, minimal theory extrapolation in the measurement, and minimal detector dependence in the BSM predictions. This means using results of type 3, and in some circumstances 4, from the list above. A general discussion of reinterpretation tools and requirements is given in Ref. [22].
In addition to making use of particle-level measurements to help exclude new physics models, another pillar of the Contur philosophy is to use inclusive event generation instead of exclusively generating individual processes. Inclusive event generation has the advantage of covering all allowed final states which would be affected if that BSM model were realised. Generating events in this way, Contur can paint a more comprehensive picture of the exclusion across all manner of final states, rather than focusing on the most spectacular signatures of a new model. Indeed, there are several cases in recent Contur papers where exclusion power for a model in some region of parameter space has come from an unexpected signature, which might not have been tested if the user had to actively switch on individual processes. By contrast, determining which processes are most important in different regions of model parameter space is not trivial if one is not an expert in the phenomenology of a particular BSM model. Herwig is an event generator which features an inclusive mode which generates all 2 → 2 processes featuring a BSM particle in the propagator or an outgoing leg. For this reason, Herwig is the default event generator used in Contur, as can be seen in the example in Appendix B. Nevertheless, Contur retains the possibility to study individual processes, for instance, when only a particular process is of interest, or to check how much of a contribution would come from processes which are more complex than 2 → 2.

Limitations of Contur
It is important to note that Contur is not at present a 'discovery' tool. It will not identify regions of BSM parameter space which are more favoured than the SM; such regions will show up as 'allowed', but the test statistic is one-sided and gives no more positive information about a BSM scenario than that. In any case, Contur only uses data which have been shown to agree with the SM.
Most of the other limitations of the Contur method at present stem from incomplete information published by the experiments. Three common issues arise: the SM prediction of a measurement is not published, the bin-to-bin correlation information for the systematic uncertainties is not made available, or the public information contains hidden, model-dependent assumptions. These items are discussed in more detail in the following.
The fact that most entries in the HepData library (described in Section 3) currently do not include a SM prediction means that assumptions must be made with respect to the null hypothesis in the Contur method. In particular, if the SM-prediction information is not available, Contur assumes the data are identically equal to the SM. This is an assumption that is reasonable for distributions where the uncertainties on the SM prediction are not larger than the uncertainties on the data; it is also the assumption made in the control regions of many searches, where the background evaluation is "data-driven". When used in this mode, Contur would be blind to a signal arising as the cumulative effect of a number of statistically insignificant deviations across a range of experimental measurements d . To extract such a signal properly requires evaluation of the theoretical uncertainties on the SM predictions for each channel. These predictions and uncertainties are gradually being added to Contur and can be tried out using a command-line option (see Ref. [6] for a first demonstration). For these reasons, limits derived by Contur where the theory predictions are not used directly are best described as expected limits, delineating regions where the measurements are sensitive and deviations are disfavoured. In regions where the confidence level is high, they do represent a real exclusion.
A further limitation comes from a lack of information about correlations between bins in some published measurements. For measurements which are not statistically limited, systematic correlations between bins may be important. Without knowing the size of the correlations between bins, Contur must only use the single most sensitive bin in a given distribution to avoid double-counting correlated excesses across multiple bins. This limits the sensitivity of Contur when a BSM signal is spread over several bins in a distribution. However, in an increasing number of cases, a breakdown of the size of each major source of correlated uncertainty in each bin is provided by the experiments, and in these cases Contur is able to make use of them.
More fundamentally, some measurements are defined in ways which make their use in Contur limited or impossible. This usually occurs because SM assumptions have been built into the measurement (for example extrapolations to parton level, or into significant unmeasured phase space regions), important selection cuts (a common example being jet vetoes) have not been implemented in the fiducial phase space definition, or large data-driven background subtractions (for example in H → γγ) have been made. Existing examples, and the conditions under which some such routines may or may not be used, are discussed in Section 3.3.
Finally, and trivially, if a Rivet routine and HepData entry are not available for a measurement, Contur cannot use it.

Code structure and setup
The Contur tool is mostly structured in the standard Python form, with a package directory contur containing the majority of processing logic, and secondary bin and data directories respectively containing executable "user-facing" scripts, and various types of supporting data files. A set of unit tests is implemented in the tests directory, using the pytest framework. An outline of the directory structure can be seen in Listing 1.
The contur Python package is internally divided into modules reflecting the distinct parameter-scanning, MC-run management, and statistical post-processing tasks of Contur operation. The scan module provides helper functions for generating and combining parameterspace scan data, plot implements the standard data-presentation formats, and the run module provides the logical cores of the main user scripts in a form amenable to pytest testing. The statistical machinery central to Contur lives in the main contur namespace, supported by utility functions, data-loading tools, and Contur's analysis-pool data from the util, factories and data modules. These classes are documented in inline documentation using Pydoc, which is linked from the main Contur repository.
The bin directory contains the main user scripts described in this paper, plus the auxiliary ones described in Section C. The data directory contains a mixture of bundled and generated data. Included in the release are: • sets of model files and generation templates created so far, • any modified or new Rivet analysis codes and reference data not bundled with the Rivet release, • theory-based background estimates for analyses where the data need not be assumed to be purely SM. Files generated by the user after installation include: • the compiled Rivet analysis-plugin libraries, • analysis-pool database (see Appendix H), • MC-run template files.
The Contur package relies on a compiled Rivet installation and set of analysis overrides, and requires manually copying template files from data/Models to run MC scans. For these reasons, it is not usually recommended to perform the installation using the Python setuptools scheme e . Installing and using Contur is instead normally done directly from the downloaded project directory, by sourcing a script called setupContur.sh on each shell session, and by running the make command only once upon installation. Sourcing setupContur.sh sets environment variables (such as CONTUR ROOT) that Rivet uses to locate custom analyses and data. Additionally, this script appends the Contur Python module path and the executable bin directory to the system PYTHONPATH and PATH respectively, mirroring the function of Python's standard setuptools. As the Contur package makes use of a compiled database, referencing analysis lists derived from this at run time, setting these environment variables is needed to operate parts of the workflow. Furthermore, a short python script is run when setupContur.sh is sourced which checks the various dependencies and paths.

Rivet analyses
Rivet functions as a library of preserved particle-level measurements from colliders. Each publication has a corresponding Rivet routine: a runnable C++ code snippet which encapsulates at particle-level the definition of the measured cross section. Rivet routines can be thought of as filters that select generated events which would enter the fiducial region, and project their properties into histograms with the same observables and binnings as the measurements. Several hundred measurements are preserved in this way, many of them from LHC experiments. The output of Rivet is a set of statistical analysis objects stored in the native YODA format. YODA files are human-readable text files with structures to encode binnings, statistical moments and other (correlated) uncertainties for the usual statistical analysis objects used in HEP: oneand two-dimensional histograms, one-to three-dimensional scatter plots, profile histograms, and so on...
The HepData [23] repository contains a digitized record of the measured cross-section values and their uncertainties, sometimes also including the best SM theory predictions at the time, and sometimes including a breakdown of uncertainties in each bin or other correlation information. This information about experimental measurements from HepData can also be exported in the YODA format. YODA files are synchronised between Rivet and HepData whenever a new release of Rivet is made, so that a faithful comparison of generator output to measured data and uncertainties can be made.
The measurements present in Rivet and used by Contur in the current version are those in Refs.  although new measurements are continually being added. Contur re-uses these encapsulated analysis routines, but runs with generated BSM events rather than the SM process which was typically the target of the measurement. Rivet is specifically designed to run multiple (or indeed, all) analysis plugins simultaneously for a given beam configuration. It has been optimised to do this quickly and efficiently. Thus BSM events generated by Herwig or another MCEG are filtered through all available plugins, leading to a multitude of histograms showing if, and where, the signal would have appeared in existing LHC measurements. The size of the signal can then be compared to the relevant HepData reference histogram, to decide if the set of BSM parameters in question would have produced a distortion to the SM spectrum beyond measured uncertainties. This simple stacking of BSM contrinutions onto existing measurements needs additional information for certain measurement types. Indeed, normalised histograms are complemented with a fiducial cross-section factor in the analysis database (when this is provided by the experiment), which allows rescaling to the differential cross-section for addition, then re-normalisation. Ratio plots have a similar special treatment, and a profile histogram treatment is being developed in the same way.

Categorisation of Rivet routines into orthogonal pools
If injection of BSM signal leads to an excess in a measured distribution, there may also be excesses in measurements of similar final states produced from partially overlapping datasets, leading to correlations. Since correlations between the measurements cannot be accounted for, this could lead to an overestimate in the sensitivity. To avoid such double counting, Contur classifies Rivet histograms into orthogonal pools based on centre-of-mass energy of the LHC beam, the experiment which performed the measurement, and the final state which was probed. For analyses which measured several final states which are implemented as different options within the Rivet plugin, it is possible to sort histograms from the same analysis into different pools. If there are orthogonal phase space regions measured within the same analysis (for example different rapidity regions in a jet cross section) it is possible to combine the non-overlapping histograms into a "subpool", in which case the combined exclusion of the subpool will be evaluated, and treated as though it came from a single histogram.
The results from each pool can then be combined without risk of over-stating the sensitivity to a given signal. Analysis pools are named as Experiment Center-of-mass energy Final state , where: • Experiment can be ATLAS, LHCB or CMS at present; • Center-of-mass energy can be 7, 8 or 13 TeV; • Final state is a short string which loosely describes the final state, with details given in Table 1 ; These pools, and other information, are stored in a database, described in Appendix H. Although Contur presently only uses LHC results, measurements from non-LHC experiments, such as LEP or HERA, could also in principle be included provided they were made in a model-independent way, and preserved in an appropriate format with a Rivet routine and HepData entry. All that would be needed is to add additional beam modes to simulate collisions of the appropriate particles at appropriate energies.

Adding user-provided or modified Rivet analyses
To make a local modification to an existing Rivet routine or include one not yet in the Rivet release, the new or modified analysis plugin can be copied into the contur/data/Rivet directory along with any updated reference files. Further, if a new theory calculation becomes available this can be added to the contur/data/Theory directory. The new routine can be compiled with a simple make call, followed by setupContur.sh. This will over-ride the default (un-modified) version of the replaced analysis for the next run. The new analysis should also be added to the analysis.sql file, documented in Appendix H.

Rivet routine special cases and common pitfalls
Most analyses preserved in Rivet are particle-level measurements, meaning the measurement is to a large extent defined in terms of an observable final state, and the effects of the detector have already been corrected for during the unfolding procedure, within some fiducial region. As a result, predictions and measurements can be compared directly, without the need for smearing or detector simulation. Some exceptions and caveats exist however, limiting the applicability of some analyses. The current known special cases are discussed below, and their categorisation in the Contur database structure is discussed in Appendix H.2.
Ratio plots The current most powerful particle-level measurement of missing energy is in the form of the measurement of a ratio of plus jets to missing energy plus jets [92]. The cancellations involved bring greater precision, but the SM leptonic process is hard-coded as the denominator, so the results are not reliable for models that would change this -for example, enhanced Z production will contribute to both the numerator and the denominator. For models where this is expected to be an issue, the analysis may be excluded by setting the --xr flag at Contur runtime.
H → γγ These fiducial measurements [119] are very powerful for models which enhance SM Higgs production. However, they rely on a fit to the γγ mass continuum to subtract background. Signals from models which enhance non-resonant γγ production would presumably have influenced this fit, and might have been absorbed into it, so looking at their contribution only in the H mass window will overestimate the sensitivity. These analyses may be excluded in such cases by setting the --xhg flag at Contur run time.
Searches Detector-level Rivet routines do exist for some searches, and can be used by Contur [41,74]. In this case Rivet's custom smearing functionality is used, and the SM background from HepData is used for comparison. These searches may be turned on by setting the -s flag at Contur run-time.
H → WW Like H → γγ, these measurements [46,117] could potentially be very important when SM Higgs production is enhanced. However they involve very large data-driven background subtraction (principally for top), and the reliability of this for non-SM production mechanisms (of Higgs, WW , or just dileptons and missing energy) is in general hard to determine. These analyses may be turned on by setting the --whw flag at Contur run time.
ATLAS WZ This analysis [32] may be useful for models which enhance WZ production, but it calculates event kinematics using the flavour of the neutrinos, and so its impact on other missing energy signals is difficult to evaluate. The analysis may be turned on by setting the --awz flag at Contur run time.
b-jet veto Analyses targeting WW production processes generally use b-jet vetos to suppress WW production via tt. In some cases, this kinematic requirement is made only at detector level, and not included in the fiducial cross section definition implemented in the Rivet routine [46,53,71,123]. These analyses are therefore likely to give misleading results when used on non-SM WW production processes.
These exceptions are catalogued in the analysis database (see Appendix H) and should be taken into account when implementing or adding a new analysis to Contur. Some guidelines for designing analyses to minimise their model dependence and maximise their impact in a Contur-like approach are given in Ref. [22]. The most important principle is that theorybased extrapolations should be avoided where possible, both for background subtraction and unmeasured signal regions. This essentially means defining a fiducial measurement region in terms of final state particles which as far as possible faithfully reflects the actual detector-level event selection.

Sampling model parameters
New physics models usually have a number of parameters which are not fixed. Surveying such a model begins with identifying the parameters of interest and sampling points within that parameter space. Contur provides a simple custom tool-set to facilitate this, currently limited to sampling a small number of parameters in a single scan.
The scanning functionality is implemented in the scan module, and user interaction is mostly controlled with the contur-batch executable. This executable requires three core user-defined components governing the behaviour of the scan: • A run information directory containing the required common files such as model definition and analysis lists. Preparing this directory is outlined in Section 4.1; • A parameter card dictating how the parameters of interest should be sampled. The structure of this file is explained in Section 4.2; • A template generator steering file. This depends on the MCEG being used, and is discussed in Appendices B.5 and G.
Alternatively to constructing scans of model parameters, specific parameter choices can be manually sampled. By calculating observables for a chosen set of parameters (demonstrated using Herwig in Appendix B.3), a file containing YODA analysis objects can be fed directly to the likelihood machinery described in Section 5. This allows manual sampling of a parameter space using the Contur likelihood machinery.

Initial grid setup
The list of observables to calculate is dependent on the available list of compiled Rivet analyses. As discussed in Section 3, this list can be augmented by the user and is subject to change dependent on the Rivet version used. The contur-mkana command line utility is called to generate some static lists of available analyses to feed into the MCEG codes. Specifically, Herwig-style template files are created in a series of .ana files, and a shell script to set environment variables containing lists of analyses useful for command-line-steered MCEGs is also written. After contur-mkana is invoked, re-sourcing the setupContur.sh script defines the necessary environment variables. The Herwig analysis list files will also now exist in data/share. Local model files and, if using Herwig, the analysis list files, should be copied to a subdirectory of the local run area (default name RunInfo) f . This subdirectory is then supplied to the main contur-batch executable via the grid command-line argument.

Parameter card setup
The parameter card is supplied to the contur-batch executable via the --param file command-line argument. The structure of this file is based on the input/output structure defined by the Python configObj package. Entries delineated by square braces define dictionaries named as the contained string. Double square braces are a dictionary within the parent dictionary. The two main dictionaries to steer the parameter sampler are Run and Parameters. An example parameter card with three model parameters is given in Listing 2. Two additional dictionaries are implemented, which allow the user to make processing more efficient by skipping certain points (using a block named SkippedPoints) g and scaling the number of events generated at each point (using a block named NEventScalings), since some points may need to be probed with more precision than others. The NEventScalings dictionary is only applied if the grid is submitted using the --variablePrecision option of contur-batch. Both the SkippedPoints and NEventScalings dictionaries can be added automatically to a parameter card using the contur-zoom utility, which is designed to help the user iteratively refine a parameter scan, and which is documented in Appendix C.6.

Parameter card Run arguments
The Run block is intended to control high level steering of the parameter sampler. Two dictionary keys are defined in this block: • generator, path to a shell script that configures the necessary variables to setup the event generator; • contur, path to a shell script that configures the necessary variables to load the Contur package.
Both of these callable scripts are expected to set up the required software stack to execute the calculation of observables on a High Performance Computing (HPC) node.

Parameter card Parameters arguments
Within the Parameters dictionary, a series of sub-dictionaries (in double square braces) define the treatment of each parameter in the model. The string used as the name of this dictionary is the name of the parameter, and must also appear in the MCEG run-card template. The f These files will be copied automatically by contur-batch if not already present. g In future, we intend to make further use of pandas DataFrame compatibility to provide such functionality more elegantly. mode field defines the type of the parameter, and opens additional allowed fields modifying its behaviour. The available values for mode, with the sub-list detailing the unique additional parameters for each, are given below: • CONST, a constant parameter.
value, a float with the value to assume for this parameter.
start/stop, the floats of the boundaries of the target sampled space for this parameter (note: start must be a smaller number than stop).
number, an integer number of values to sample in the range.
• REL, a relative parameter, defined with reference to one or more of the other parameters.
form, any mathematical expression that Python can evaluate using the eval() function form of the standard library, where parameter names wrapped by curly braces, as seen on Listing 2, will be replaced by the value of that parameter before evaluating the expression. The name between braces must match exactly that of the parameter as specified in the Parameter block. For safety and efficiency, it is preferable (and often necessary) to use the DATAFRAME mode if complex mathematical expressions (i.e. anything beyond basic arithmetic operations) are required to generate the desired value for this parameter.
• SINGLE/SCALED. Single string substitution. If the parameter name is "slha file ", provide a path to a single SLHA file as name which will be treated as described in Appendix E.
• DIR, if using the SLHA specification, giving a directory to the SLHA files as name.
Each file in the directory will generate a separate run point with the parameters set accordingly. • DATAFRAME, one can also provide a pandas DataFrame in a pickle file as name, which provides the parameters to vary and their values, one point for each row of the table.
pandas DataFrame support is further documented in Appendix F .
With these tools, many parameters available in the model can be scoped in the Contur parameter sampler. The parameters whose mode is either LOG, LIN or DATAFRAME are the scanned parameters, and the number of such parameters is the dimensionality of the scan. REL or CONST parameters are then ways to correctly set the additional parameters of the model. Any dimension of scan is technically possible but typically only up to two or three parameters have been considered in physics studies using Contur. For high-dimensional scans, contur-export allows exporting results to a CSV file, so that alternative visualisation tools beyond contur-plot can be used (see Appendix C.2).
The Contur scanning machinery could in principle be extended to find the least constrained parameter point: this could be done by making use of the Contur code interface and connecting to a numerical scanner or minimiser. This would require more efficient scanning of multidimensional parameter spaces: this is an area of active research in the Contur team and in the reinterpretation community more widely.

Generator template
To interface an MCEG with the Contur parameter sampler, a template of the generator input has to be provided. This template file is supplied to the contur-batch executable via the --template file command line argument. The parameters that are scoped in Contur as described in Section 4 are then substituted into this file, thus defining the generator run conditions.
Following the example of Listing 2 for a Contur parameter file, a snippet of the matching Herwig input card is shown in Listing 3. Much of the syntax is Herwig-specific, and further discussion is left to the Herwig documentation. Important features to notice are that the parameters are being defined in the Herwig FRModel (short for FeynRules model, the placeholder for a Herwig-parsed UFO model file). The names within curly braces match the parameter dictionary names in the parameter card file, allowing numeric values for each to be substituted in, following the rules defined in Section 4.2. Since this workflow is based upon string parsing and substitution, any event generator configuration that can be steered in a similar way can be substituted. In the example, the mass of the X particle has been scanned with the Contur sampler by varying the defined parameter, x0. An example of the process definition is also included in Listing 3 for this toy model. In this example, the instruction given to the generator is to inclusively generate all 2 → 2 processes with incoming up and anti-up quarks, and an outgoing hypothesised X particle. According to the Feynman rules in the parsed FRModel.model file, all allowed diagrams will be generated. This is the ideal generator running mode for Contur, consistent with its inclusive philosophy. However, not all generators provide this inclusive option; also, in some cases it may be useful to focus on specific processes.
As motivated in Section 2.2, this generator setup should be set to generate signal-only contributions to the relevant observables. The statistical analysis detailed in Section 5 will treat the observables resulting from generator runs as being additive signal contributions to the background model.

Grid structure and HPC support
The execution of observable calculation in Contur is realised in two steps: definition of the event-generation and observable-construction jobs, and execution of those jobs.
First, if .ana files are required and do not already exist locally, they will be automatically copied by contur-batch from $CONTUR ROOT/data/share to the local RunInfo directory. Next, a run directory will be created (named myscan## by default), with a subdirectory for each distinct set of run conditions (currently the three available LHC beam energies). In a dedicated subdirectory of each of these for each point in the parameter grid, the sampler creates all associated generator files, with the required commands to run the generator and the selected Rivet analyses. A shell script containing all the commands to execute the generator run from a fresh login shell is also written. An example scan directory is shown in Listing 4.
Next, the scripts which perform the calculations for each parameter point need to be executed. The contur-batch executable will automatically send each job to a HPC node. Contur supports the PBS, HTCondor and Slurm batch systems, the one in current use being controlled by use of the the --batch command-line argument. The default behaviour is to use PBS submission, where the queue name is controlled by the queue command-line argument. Slurm differs from PBS only in use of the sbatch command-line tool in place of qsub, while the HTCondor system differs from the others in not having queues and having to generate a job description file (JDF) for each scan point's condor batch call.
Alternatively, if the --scan-only command-line option is used, contur-batch will only generate the batch scripts (and JDFs if necessary) but not submit them, leaving detailed run control entirely to the user. In either mode, no batch-system management is performed by Contur once the jobs are running: for this you should use the suite of tools specific to your batch system (qstat, qdel, etc., or their Slurm or HTCondor equivalents).
The contur-batch executable also controls the number of events which are generated for each parameter point (using the --numevents option, defaulting to 30,000) h . During execution, once the generator has reached the requested number of events, the observables calculated by Rivet are stored as filled histograms in the YODA histogram format. Each parameter space point subdirectory in the grid as shown in Listing 4 will have a corresponding YODA file containing the calculated observables.

Evaluating the likelihood for a model
Calculation of the CL s exclusion at a given point in parameter space requires the construction of a likelihood function for that point. The main analysis executable, called simply contur, is responsible for this task. Taking as input a series of calculated observables in YODA format, this can be run either on a single point in parameter space, or on a grid of points generated using contur-batch, in which case a map of the likelihood of the parameter points explored by the parameter sampler is constructed. This section describes the calculation of the likelihood for an individual point in parameter space.
As this is the main analysis component in Contur, the functionality is implemented throughout the package modules. The core analysis classes are implemented in the factories module. The entry-point analysis class is named Depot which should contain the majority of the relevant user-access methods. Several intermediate classes handle various aspects of the data flow, down to the lowest-level class defining the statistical method, Likelihood. The data module implements much of the interaction between Rivet and Contur, defining how to build covariance matrices for example. The run module implements the behaviour of the executable, and interaction of this calculation with the rest of the modules.

Statistical method
A test statistic based on the profiled log likelihood ratio (LLR) can be written, with µ being the parameter of interest (POI) and ν being nuisance parameters. A single hat, e.g.ν, denotes the maximum likelihood estimator for the parameter. A double hat, e.g.ν, denotes the conditional maximum likelihood estimator for the parameter, conditioned on the assumed value of the POI. This test statistic can be used to construct a frequentist confidence interval on the POI. The convention in High Energy Physics (HEP) is to use the h In general this should correspond to an effective luminosity comparable to the luminosity of any statisticallylimited measurements they are to be compared to. For many BSM models the cross sections are small, so this number of events is not enormous. Recent Contur publication have for example typically generated the default 30,000 events per set of parameter values. CL s prescription [127,128], defined as a ratio of p-values, where the CL i values are defined for both i ∈ {b, sb} as where f i (t µ | µ) is the probability density function of the test statistic under an assumed value of the POI µ. The CL values are hence the probabilities of finding t µ values at least as large as that observed, under each hypothesis. The final p-value expression in eq. (2) is asymmetric between the background (b) and signal+background (sb) hypotheses since, cf. eq. (1), large t µ values are more background-like, and small ones more signal-like. The test statistic in the asymptotic limit [129] can be approximated by with σ 2 the variance of the POI and N the data sample size. Likelihoods in HEP are often written as a Poisson distribution composed of three separate counts; the hypothesised signal count (s), the expected background count (b) and the observed count (n). In this situation, the POI µ is defined to be a signal strength parameter, with the resulting likelihood written as In the asymptotic limit, where the Poisson distributions approximate normal distributions, the form of the test statistic becomes where σ 2 is now the variance of the counting test: this is the standard ∆χ 2 construction. This model can be extended by incorporating nuisance parameters on the background model into the likelihood function given in equation (5), and by taking a product of multiple counting tests. A likelihood for a single histogram with i bins and j sources of correlated background nuisance in each bin can be written as with Σ j being the covariance matrix for each correlated source of nuisance and ν j being the corresponding vector for the correlated nuisance parameters across the bins. In this case there are now multiple sources of nuisance common to each counting test, or bin in the histogram.
In this example there would be j different sources of nuisance, so there are j constraint terms. The constraints are now i-dimensional Gaussians to account for the covariance of each nuisance parameter between bins. The sources of nuisance can be profiled by maximising the log likelihood for the hypothesised µ.
The practical implementation of this has relied on the inclusion of the uncertainty breakdown into the YODA reference data files included with Rivet. In the asymptotic regime, maximising the log likelihood is equivalent to minimizing the χ 2 . The minimisation itself, and handling of covariance information, are achieved with the help of the SciPy [130] statistics package along with NumPy [131] for array manipulation. Assuming each individual uncertainty arising from a common named source in the reference data is fully correlated, the correlation matrix for each source of uncertainty can be built. This gives the set of Σ j matrices needed to maximise the likelihood. Minimising the χ 2 for all nuisances simultaneously gives the requisite conditional maximum likelihood estimators,ν i,j , required to form the profile likelihood as given in equation (1). Following similar asymptotic arguments, a CL s confidence interval can be calculated with the full set of nuisances suitably profiled. As an example, the test statistic in the limiting cases leading to equation (6), for two counting tests with one correlated source of nuisance can be written, For more complex cases, the sum of the covariance matrices built from each named uncertainty then gives the full covariance matrix between bins which can be used to calculate the likelihood combining all bins in a histogram. Noting that after using the breakdown of the total uncertainty into its component sources to profile the nuisances, the resulting total covariance (Σ = Σ j ) between bins can be used to construct the test. This test statistic omits the second "reference"μ term seen in equation (6), this term is trivial when running the default mode of generating background models from data, however does need full treatment in a similar manner when extending to non trivial background models (see Section 5.3 for more detailed discussion of background models).
The --correlations flag can be set in the contur executable to enable the calculation to use the correlation information where it is available. The default behaviour of Contur is however not to build the correlations between counting tests, and hence fall back to collecting single bins with the largest individual CL s to represent the histogram. This is because the full use of correlations can make the main Contur run over a large parameter grid quite slow, due to the nuisance parameter minimisation step. Various command-line options are provided to speed up convergence of the fit, or to apply a minimum threshold on the size of considered to error sources, all of which may speed up the process without significantly affecting the result. Neglecting the systematic uncertainty correlations entirely, and falling back on the "single bin" approach for all histograms, is very fast and in most cases gives a result reasonably close to the full exclusion, albeit with more vulnerability to binning effects. The user is encouraged to experiment with these settings, perhaps neglecting correlations for initial scoping scans, and reserving the full correlation treatment for final results.
Currently there is no functionality to correlate named systematics between histograms, which might in principle allow combination of all bins in an entire analysis for example. For most purposes, correlating a given histogram gives the required information. Combining different histograms is then taking a product of the likelihood in equation (7), where the histograms chosen to be combined are deemed to be sufficiently statistically uncorrelated. How these likelihood blocks are chosen to "safely" minimize correlations when combining histograms is described in the following section.

Building a full likelihood
In Section 3.1 the division of available Rivet analyses into pools was shown. As described in Section 3, the data and simulation used for comparison come in the form YODA objects from the relevant HepData entries. Some of these carry information about the correlations between systematic uncertainties. A likelihood of the form given in equation (7) can be used to calculate a representative CL s for each histogram.
There are also overlaps between event samples used in many different measurements, which lead to non-trivial correlations in the statistical uncertainties. To avoid spuriously high exclusion rates due to multiple counting of a single unusual feature against several datasets, an algorithm is used to combine histograms safely. To represent this, a pseudo-code realisation of the three main components of the algorithm is given in Listing 5. Starting with an imagined function, Likelihood, which would build a likelihood function of a form similar to that described in Section 5.1 from input histogram(s), and return a computed CL s value. The stages to combine all the available information into a full likelihood are realised as follows: 1. Calling BuildFullLikelihood loops through the defined pools in Contur, and calls EvaluatePool on each pool.
2. Within each pool, work through all histograms calling EvaluateHistogram on each.
3. Depending on desired behaviour EvaluateHistogram either builds the correlation matrix where possible, returning the CL s of the full correlated histogram or defaults to finding the bin within the histogram with the maximum discrepancy, returning this to EvaluatePool.
4. Now with each histogram evaluated, Concatenate can be called, combining orthogonal counting tests within the pool where allowed. Where a single bin has been used (if no correlation information is requested and or found), the histogram is reduced to this single bin representation. The histogram (or concatenated histogram) with the largest CL s within the pool is returned to BuildFullLikelihood 5. Now the representative histogram from each pool has been appended to a list, this list can also be concatenated. The bins (or bin) extracted from each pool are treated as uncorrelated counting tests with a block diagonal correlation matrix between each pool. The representative CL s forming the full likelihood can then be returned.
While selecting the most significant deviation within each pool sounds intuitively suspect, in this case it is a conservative approach. Operating in the context of limit setting means that discarding the less-significant deviations simply reduces sensitivity.

Theory-driven background models
The formal argument for a test statistic based on the profile likelihood ratio was given in Section 5.1. An alternative to this would be to use a test statistic based on a simple hypothesis likelihood ratio between the signal (µ = 1) and no signal (µ = 0) hypotheses. Such a test statistic could be written, in a similar form to equation (6), as, where the background model, b, can have nuisance parameters included in a similar fashion which can in turn also be profiled. In the case that the modelled background value, b, approaches the observed count, n, then the two forms of the test statistic converge. This is equivalent to the statement that as the most likely signal strength (μ) tends to zero, the 'reference' values in the χ 2 test statistics both tend to zero. In this limiting case, the argument followed that the form of the test statistic omitting these reference values, given in equation (9), was sufficient. The example Rivet plot shown in Figure 3 illustrates a signal model appearing in a region of a generated histogram that would however represent different CL s intervals arising from the two constructions. If the resonance in this spectrum were to appear in one of the regions where the theoretical expectation closely matches the data instead, the two forms would largely coincide. The default mode of running Contur is to generate the background model from the data, and with this the coincidence of the two forms of the test statistic is guaranteed. Typically, state of the art theoretical predictions are not automatically provided alongside measurement data. If such data are provided (as detailed in Section 3.2), then invoking the --theory command line option in the contur executable will load and use this data where appropriate.
As extensive use of theoretically-generated background models has not yet been made in any physics studies, the default implemented behaviour is to report the CL s resulting from a direct hypothesis test, essentially as written in equation (9). When more use is made of theoretically-generated "non-trivial" background models in physics studies, it is intended to report both forms of the test statistic as standard. In cases where the nontrivial background model is known to poorly model the data, such as in Figure 3, it is expected that the two forms of the test statistic would start to significantly diverge. The combination of a full profile likelihood with correlated nuisances will enable sophisticated physics studies, however it is expected that the current standard based on a simple 'direct' hypothesis test will remain useful for a range of fast pragmatic studies.

Running the Contur likelihood analysis
The method described thus far in this section is handled automatically in the contur executable. Either a single YODA file, or a directory containing a structured grid steered by the parameter  Figure 3: A comparison of a generic Axion-like Particle model [132] producing a 50 GeV resonance in a diphoton mass measurement [87]. The black data points represent the observed data from the original measurement. An NNLO QCD diphoton background prediction is shown in green [133] for comparison. The red line represents the sum of the BSM signal: either with the theoretically calculated background model (Figure 3a), or the background model generated from data (Figure 3b). The caption of the legend for the red line shows the index of the most sensitive bin in square brackets followed by the calculated CL s value in each case. sampler (as described in Section 4.4), can be supplied to this executable with the --grid command-line argument.
In the former case, a summary file is written which may then be processed by the contur-mkhtml script to produce a web page summarising the exclusion and display all the tested Rivet histograms, highlighting those which actually contributed to the likelihood.
In the latter case, the grid will be processed point-wise, evaluating the full likelihood at each parameter point that has been sampled. The resultant grid of evaluated likelihoods is written into a .map file, which is a file containing a serialised instance of the Depot class. This is written out using the standard library pickle functionality and can be read and manipulated for further processing. The Pydoc documentation describing the details of this class is linked from the main Contur repository. The executable implements a number of high level control options for vetoing analyses and controlling the statistical treatment.

Visualisation of parameter space
The .map files described in the previous section contain the Contur likelihood analysis for a sampled collection of points. The core plotting tools that interact with these files are described in this section. There are multiple auxiliary tools to aid visual understanding of the .map files, which are detailed in Appendix C. The core plotting library, which is build upon matplotlib [134,135], is implemented in the plot module, and user interaction with this module is driven by the contur-plot executable. This executable requires three arguments; the .map file generated from the main contur executable and the names of two parameters on which to draw the axes. Visualisation is limited to two dimensions, but if more than two dimensions were scanned, then multiple 2D plotting instances can be invoked. The names of the requested parameters should match what they were initially called in the parameter sampler (see Section 4.2). The main default visualisation of the likelihood space is demonstrated in Section 6.1. Some methods to interface additional information (such as exclusion contours from other tools) into the default visualisation tools are reviewed in Section 6.2.

Grid visualisation
The sensitivities calculated by Contur for each grid point can be expressed as 2D heatmaps, for the overall sensitivity or for each pool separately. The heatmaps indicate where the considered signal model can be excluded due to existing LHC measurements available in Rivet and which part of the phase space is still open. The per-pool heatmaps give more detailed insights into where a specific pool contributes, allowing to draw conclusions on the production processes and decay modes involved. An overview of how the individual pools' sensitivities compare to each other is provided by plotting the dominant pools: Contur then shows colour-coded which pool has the highest sensitivity for a given grid point in the same plane as for the heatmaps. Finally, Contur also provides exclusion contours at 68% and 95% confidence level as interpolated from the 2D sensitivity grids. Examples of these types of output plots can be found in Figure 4. Further information about available options can be found in Appendix C.7.

Including additional information in default plots
The default grid visualisation tools described in Section 6.1 provide two methods to supply additional data, allowing creation of additional grids to overlay on the native Contur grid. Both methods use the Python importlib package, defining a series of Python functions to import via command-line arguments to the contur-plot executable. Both methods require user-defined functions that take as an argument a Python dictionary of the parameters, named as specified in the scan (see Section 4). Both methods expect to return a pseudo "exclusion" value specifying the exclusion at the requested point in parameter space. The values are expected to be set such that negative numbers are allowed and positive numbers are excluded (i.e. the contour is drawn at the level set of zero), it is generally advised to use the "distance" of the value from 0 to accurately fit the contour. The two methods, and examples of functions expected for the two formats, are given in the following subsections.

Plotting external grids
The first method of adding additional information to a plot is invoked by supplying a file containing an external grid with the command-line argument -eg (or --externalGrid) NameOfFile. These functions define user supplied grids, allowing arbitrary numbers of points within the space to be considered. These can be read from additional data sources within the supplied file or simply used to calculate analytic constraints at a much higher resolution than in the Contur scan. The function should return a tuple with the first argument being the list of parameter space dictionaries of the new considered points, and the second argument being a list of floats of the pseudo-exclusion values. An example loading in a user supplied grid for visualisation is shown in Listing 6.

Plotting external functions
The second method of adding additional information to a plot is invoked by supplying a file containing a function with the command line argument -ef (or --externalFunction) NameOfFile. This type of input is for curves which can be evaluated internally from the parameter point coordinates. For example, one might use this functionality to indicate the value of an additional model parameter. These functions recycle the existing Contur sample of points to evaluate functions on a grid of the same resolution as the Contur scan. The function should return a float of the pseudo-exclusion value. Internally, Contur will then evaluate this function on the Contur grid. An example function is shown in Listing 7.

Summary
This manual accompanies the release of Contur v2, which is the first public-facing version. Please refer to the Contur homepage [10] for links to the latest instructions and source code.
In this document, the method and the structure of the Contur package were set out, the core functionality of the Contur code was described and the motivations behind key design choices were given. On a more philosophical note, the objective of the Contur package is to allow the HEP community easily to re-use the LHC analyses preserved in Rivet and HepData to derive exclusions on new physics models. These analyses, the bulk of which are particle-level measurements of SM processes, are often highly model-independent, and can be used to rule out models which would have interfered with otherwise well-understood spectra. The fact that models can be tested programmatically, making use of the runnable code snippets in Rivet which encapsulate their fiducial regions, implies that large regions of parameter space can be probed with minimal "hands-on" effort from analysts. The authors believe that this ability to interrogate existing LHC data directly, rather than construct a new search for each new model proposed by theorists, is a key step in the necessary paradigm shift in HEP from "top-down" (theory-driven) to "bottom-up" (data-driven), which is being brought about by the proliferation of candidate new physics models, in the face of increasingly large datasets and corresponding pressure on computing and human resources. The Contur developers are always happy to receive feature requests, and new members of the team are welcome to contribute.

A Detailed schematic of Contur workflow
A detailed diagram summarising the Contur workflow is provided in Figure 5.

B Example Contur study with Herwig
Contur supports various event generators, as documented in Appendix G, but the default choice is Herwig. This is because Herwig features an inclusive event generation mode, where one can very easily generate all 2 → 2 processes which include a BSM particle or coupling. This section shows a complete example of running a parameter scan for a BSM dark matter vector mediator model, encapsulated in the pre-loaded DM vector mediator UFO file.

B.1 Use of Docker
As documented in Appendix C.1, the user may find it convenient to run Contur within a Docker container on a local machine. While this avoids a formal installation of Contur's dependencies, it does also prohibit the user from submitting jobs to a HPC cluster: for this, one needs to do a full installation on the relevant cluster. Nonetheless, one can still generate events, run Contur on individual parameter points, and analyse results of Contur scans which have been performed elsewhere. If the user wishes to use a Docker container to run this example, they can follow the commands in Listing 8 and proceed with the rest of the example.

B.2 Setting up the run area
The model used in this example is one of many pre-loaded example UFO files and associated templates that come with the Contur package. These can be found in the contur/data/Models directory. The first step is to make a work area, and to copy into it a template RunInfo directory as well as the model's UFO files. Once this has been done, one needs to convert the UFO to the Herwig format, and compile it. This will render the model readable by Herwig. Listing 9 shows the steps to setup a run area for a DM vector mediator model.

B.3 Event generation
In Herwig, an EventGenerator object is built to generate events. The configuration for this object is done in a Herwig input file (see Listing 10 for an example), with filename extension .in. In Contur these files are usually called LHC.in, and for each example model Contur provides there is an associated example input file in the model directory. A recommended starting point for Contur studies is to complete a single run of Herwig on the chosen model. The LHC.in file needs to be customized for a particular model, by specifying the values of the parameters in the UFO file one is considering. This might mean setting particle masses, coupling strengths or other model parameters. The input file should therefore contain lines like those in Listing 10, customized to the parameters of a given model. One should also specify which BSM particles should be considered during event generation (either as outgoing or intermediate particles), and add the setting to inclusively generate all processes involving that particle. Finally, one needs to tell Herwig to pipe the generated events into Rivet, so that they can be analysed directly and used in the Contur workflow. For batch runs, Contur's steering code automatically appends these lines. After setting up the input file, one can simply read it and generate events with Herwig. For the case of the DM vector mediator model, a template Herwig configuration file for a single run is provided in the model directory. Listing 11 shows the steps to using this template for event generation. First one must copy the template file into the run area. Next, the Herwig read step reads and builds the event generator from the configuration file LHC.in, and the Herwig run line tells the event generator to generate 200 events. Note that the Herwig run card, LHC.run, is the output of the first line. If successfully run, the output of Listing 11 will produce the file LHC.yoda containing the results of the Herwig run. Note that the commands here will read the analysis listing file 13TeV.ana from the installed area. If you wish instead to read a local version, modify the read command -I argument to point to that instead.

B.4 Running Contur on a single YODA file
Following the steps of the previous section where an LHC.yoda file was produced from the DM vector mediator model, the second command in Listing 12 tells Contur to analyse it. The computed exclusion will be printed to the terminal. Additional options when running Contur are also available, and can be accessed through contur --help.
Once Contur has successfully analysed the YODA file, an ANALYSIS folder is made, and an exclusion for the model at the specified parameter points is printed alongside some other information about the run. Following along the case study for the DM vector mediator model, the printed exclusion corresponds to the parameter values defined in Listing 10. It is often useful to plot the relevant Rivet histograms from the Contur run to get a better idea of the underlying physics in the calculated exclusion. The third line of Listing 12 runs contur-mkhtml on the analyzed YODA file, which generates a contur-plots directory that contains all the histograms, alongside an index.html file to view them in a browser. An example of the output is shown in Figure 6.

This step cannot be run from within a Docker container, as it requires access to a HPC system. Instead, one should use a full installation on a HPC cluster.
The next step is to essentially repeat the procedures described in Appendix B.3 and Appendix B.4 and complete a series of runs at various model parameter points, so that an exclusion for the model in the parameter space can be drawn as a 2D heatmap. This can be done efficiently using Contur's automated batch-job submission functionality. Here we assume that qsub is available on your system; it is the default choice for Contur. Slurm and HTCondor batch systems are also supported, as described in Section 4.4.
To set up Contur for batch-job submission, the user must tell Contur what region of parameter space to sample. To do this, the user should replace the nominal mass of dark matter (Xm), nominal mass of the vector mediator (Y1), and the couplings gYq and gYXm in Listing 10 by arbitrary variables. A steering file, param file.dat, will specify what values to set for each parameter. Template files are available in the data/share directory and can be copied to runarea as per Listing 13.
The newly-copied LHC.in should resemble Listing 14, where the values for the model parameters have been replaced with their respective variables inside curly brackets. Also notice that this version of the Herwig input file is missing the commands that specify beam energy, run the Rivet pipeline, and save the event generator (cf. listing 10). These lines will be added automatically by Contur for each beam energy when the batch-jobs are submitted.
The user should then modify the steering file param file.dat to look like listing Listing 15, replacing the placeholder paths for Herwig and Contur under the Run heading with local paths. The free parameters of our DM vector mediator model are listed under Parameters. The variable name for each parameter must match those in Listing 14 for Contur to recognize the parameter and substitute in the correct value.
For each parameter, the mode for sampling must be specified. For this example, the particle masses of the dark matter candidate and the vector mediator are set to the LIN linear mode. The start and stop options indicate the sampling range in GeV, and number=15 tells Contur to sample 10 points in this range in a linear fashion. The coupling of the vector mediator to dark matter and to quarks are set to the CONST constant mode, with values 1.0 and 0.25 respectively. The user is now ready to submit a batch job over the specified range of parameter points, following the commands of Listing 16 in the runarea directory. This will create a directory called myscan00 which contains three directories corresponding to the beam energies 7, 8, and 13 TeV. Inside each beam energy directory there will be 100 run-point subdirectories that correspond to your specified range in param file.dat. Inspecting a few of these reveals the Herwig input files generated, and the shell scripts (runpoint xxxx.sh). These shell scripts can be submitted manually, or run locally in your terminal for troubleshooting.
Listing 16: Submitting a Contur batch job with 1000 events per point to the mediumc7 batch queue (queue names will of course depend on your local cluster) § ¤ cd runarea contur-batch -n 1000 -q mediumc7 ¦ ¥ Listing 17: Running Contur on a grid § ¤ cd runarea contur -g myscan00 ¦ ¥

B.6 Running Contur on a grid
Once the batch jobs have successfully finished, each runpoint directory in myscan00 should have produced a YODA file. Each YODA file will be named according to the scan point, for example LHC-S101-runpoint 0001.yoda. Some functionalities are provided by contur-gridtool, which performs various manipulations on a completed Contur grid and can be useful for troubleshooting. For example, running contur-gridtool -c myscan00 runs a check to see whether all grid points have valid YODA files. See section C.3 for full details of how to use it. Listing 17 runs Contur with the -g or --grid option, which means statistical analysis is to be performed on the specified grid myscan00. The output of this step is a directory named ANALYSIS, inside which a contur.map file of the corresponding grid is produced. You may wish to create a shell script for this step and submit it to the batch system for larger grids.

B.7 Making Contur heatmaps
The last step in a Contur sensitivity study is the visualisation of the computed limits in the form of 2D sensitivity heatmaps. Once Contur has successfully run and produced a contur.map file, one can run the contur-plot command on it while specifying the variables to plot. In the example in Listing 18, the mass of the dark matter particle mXm is on the x-axis and the mass of the vector mediator mY1 is on the y-axis. The output plots are shown in Figure 4.

C Contur tools and utilities
The Contur package provides several tools and executables to assist the user in the preparation, manipulation and visualisation of Contur results. These tools are documented below.

C.1 Contur Docker containers
Containerisation of software packages with Docker provides a convenient way to bundle a piece of software with all its dependencies, so that one can forego a formal installation and run the software simply by downloading and entering the container. This also allows fast and trouble-free deployment across operating systems. The Contur developers maintain two types of container, which are regularly updated on DockerHub: • hepstore/contur is a container which includes the latest version of Contur along with all its dependencies except for the Herwig MCEG. This is useful for users who do not wish to generate events, but instead analyse the results of existing scans performed elsewhere, or make use of the visualisation tools. It may also be of use for expert users who wish to use the container as a base and install other MCEGs on top.
• hepstore/contur-herwig is a container which includes the latest version of Contur along with all its dependencies including the Herwig MCEG. This is heavier than the hepstore/contur container, but allows the user to generate events.
These containers can be downloaded from the command line using for example docker pull hepstore/contur-herwig:latest, where the tag after the colon can be replaced by another keyword to download a particular version.
A limitation of running Contur via a Docker container is that one does not typically have access to a HPC on a local machine. Furthermore, HPC clusters often do not support jobs running through a Docker container. Therefore, it is usually not possible to submit scans to HPC clusters if using Contur via a container.
If the user wishes to build their own container locally, they can make use of the Dockerfiles which are provided in docker/*/Dockerfile. In addition to the Dockerfiles used for the above-listed containers, a contur-dev Dockerfile is provided, for users wishing to access the development branch of Contur. A detailed example of how to download or build one of the Contur containers is provided in Listing 19.

C.2 Exporting the results of a Contur statistical analysis to a CSV file using contur-export
The map files containing the Contur likelihood analysis for a sampled collection of points described in Section 5.4 can also be exported to a CSV file by using the contur-export command with the -i and -o flags to specify the input and output paths for the map and CSV files respectively. Adding the -dp or --dominant-pools flag appends a column containing the dominant pools for each point.

C.3 Manipulating Contur scan directories with contur-gridtool
Contur provides a compilation of grid tools for managing grids produced with the contur-batch command. These can be accessed with the contur-gridtool command followed by various optional parameters. They allow to merge different signal grids into a single one (--merge), removing files unnecessary for post-processing (--remove-merged) or compressing those that are crucial to reduce disk space (--archive). Other options check for (--check) or resubmit (--submit) failed jobs to the batch system or identify the grid points that are most consistent with a given set of parameters (--findPoint). $ git clone https://gitlab.com/hepcedar/contur.git $ cd docker/contur # or e.g. docker/contur−dev, to directory containing Dockerfile $ docker build -t contur . # it may then take some time to build the container... $ docker run -it -p 80:80 -v path/to/useful/directory:/mydir contur # ... and proceed as above ¦ ¥

C.4 Concatenating the results of Contur statistical analyses
Running the contur command on a scan directory produces a .map file. One may want to concatenate the results of several scans by merging their relevant .map files. This can be achieved using the contur-mapmerge command.

C.5 Submitting Contur scans to a HPC systems using contur-batch
An executable called contur-batch is provided to prepare a parameter space scan and submit batch jobs. It produces a directory for each of the various beam energies (7, 8 and 13 TeV by default, but configurable with the --beams command), containing generator configuration files detailing the parameters used at that run point and a shell script to run the generator that is then submitted to a HPC cluster. The --param file, --template file and --grid options may be used to specify the names of the relevant configuration files if they differ from the defaults. The number of events to generate at each point is controlled by the --numevents option, defaulting to 30, 000. In the simplest use-case, the same number of events will be generated at each point in the scan. However, this may be sub-optimal, since some areas of parameter space may require far more events than others, for example if BSM processes are swamped by SM processes. In such a case, the expert user may wish to generate a different number of events at each point. This behaviour can be enabled using the --variablePrecision flag, which then looks for an additional section of the parameter card entitled NEventScalings indicating how to scale the number of events for each point. See Section 4.2 for more information on the parameter card, and Section C.6 for information on the contur-zoom tool which can be used to automatically add the NEventScalings section to a parameter card. By default, the tool assumes that Herwig is the MCEG, but this can be changed with --mceg. If the user wishes to run their own instances of a MCEG and Rivet, and pipe this information to the jobs, the flag --pipe-hepmc can be used. The MCEG seed can be changed using the --seed option.
One can use the --out dir option to specify where the scan directory should be written. Several options exist to specify the batch system to use (--batch system) and the queue name (--queue) and well as the maximum wall time (--walltime) and maximum memory consumption for jobs (--memory). Finally, the --scan-only flag can be used to do a dry run: prepare the directories without submitting them to the cluster.

C.6 Iteratively refining the scanned parameter space with contur-zoom
The contur-zoom utility is designed to optimise the hyper-parameters of a parameter scan, such as the ranges, granularity of binning, and number of events to generate at each point. The reasoning behind this tool is that not all areas of a parameter space are interesting: indeed, parts of the parameter space which are well below the exclusion level, or well above it, can be ignored, and the more interesting regions to focus on are those where the gradient of CL s values is large. Furthermore, focusing only on interesting regions of parameter space avoids wasting computing resources on points where the result is unambiguous. A user approaching a new model may wish to begin with a coarse, wide-ranging scan of parameter space, and then iteratively "zoom" into the more interesting regions.
contur-zoom automatically determines a new set of hyper-parameters for a parameter card, given the results of a previous coarser scan. It does this by defining a figure of merit for each point in a scan, to approximate the CL s gradient at that point; this is calculated as the average difference in the CL s value of a given point with respect to all adjacent points in the scan. By construction, this figure of merit will always be within 0 and 1, since that is the range of possible CL s values. This figure of merit is implemented for Contur scans of arbitrarily-high dimensionality.
Given a figure of merit for each point, it is possible to change the ranges of a scan to focus on the region with the fastest change in CL s , by specifying a minimum threshold for the figure of merit. The "zoomed" range of parameter values is an n-dimensional parameter space that is obtained by iterating through each dimension, and locating the smallest range of points on that axis that contains all points above the threshold, and using this reduced parameter space for the next iteration. The result is a new rectilinear scan range of the model parameters, containing all points with a figure of merit above the threshold.
This procedure can be applied to a Contur parameter scan using the contur-zoom command, where the threshold can be specified with --thresh, defaulting to 0.25. The .map file for the original scan should be provided using --m path or -m, and the corresponding original parameter file with --o path. One can either choose to replace the original parameter file with the zoomed version (using --replace), or specify where to write the new files to with --n path. If one wanted to restrict the zooming to a single dimension of the n-dimensional parameter space, this can be achieved using the --param option. Finally, one can over-ride the figure of merit for particular points with special CL s values, so that they are included in the new range regardless of the gradient. For example, one may wish to keep all points on the 68% and/or 95% CL contours. This can be achieved using the --vals option, and specifying a space-separated list of CL s values (between 0 and 1). The algorithm will then keep all points with CL s within 0.01 of the specified values.
To avoid wasting computing resources on uninteresting points, one may consider excluding points with a figure of merit below a given threshold from processing. This can be achieved using the --skipPoints option of contur-zoom, with the exclusion threshold specified by --thresh. The indices of all points below that threshold will be added to a new block of the parameter card, labelled SkippedPoints, and these points will not be processed during the Contur scan directory preparation and processing.
Finally, one may want to prepare a variable-precision scan over a parameter space, i.e. one where the number of events generated at each point may change depending on the region of parameter space. One may wish to generate more events near "interesting" regions, according to the figure of merit defined above. This can be achieved using the --nEventScalings option. This option will add a section to the parameter card labelled NEventScalings, which is simply the value of the figure of merit at each point. When using contur-batch with a parameter card which has a NEventScalings section, and using the the --variablePrecision (or simply -v) option to indicate a variable-precision scan, the number of events specified with -n will indicate a maximum number of events, which will be scaled by the value of the figure of merit at that point. Thus, the points with the highest figure of merit (which is always between 0 and 1 by construction) will be processed with a number of events close to the maximum, while less interesting points, where the CL s gradient is smaller, will be generated with fewer events.
Finally, the contur-zoom command allows the user to re-bin the parameter space based on the figure of merit, while maintaining the same ranges. This can be achieved using the --rebin flag. This will create a new parameter card where the number of bins along each axis is doubled. This option can be used in tandem with the --skipPoints and --nEventScalings options, where the figure of merit of the newly-generated bins will be by default set to the same value as their parent bins.
Examples of the effect of contur-zoom commands on an example parameter card can be seen in Listing 20. The suggested approach would be to begin with a broad, coarse scan over a given parameter space, and iteratively update the ranges, number of points and number of bins using contur-zoom.

C.7 Visualising the results of a Contur scan using contur-plot
The contur-plot executable produces visualisations of the results of a Contur grid scan. The tool takes as input a .map file obtained from running contur -g on a parameter scan directory. This executable can handle 2-or 3-dimensional scans. The user should therefore specify a .map file to read and 2 or 3 variables to plot as positional arguments.
In addition to the positional arguments, the user can specify --theory and --data arguments to add additional information to a plot. This is discussed in detail in Section 6.2.
The plot title can be set using the --title option. The x-and/or y-axis labels can be set using --xlabel and --ylabel options, which accept L A T E X formatting but special characters must be escaped with a backslash. Furthermore, the user may choose to display the x-and/or y-axis on a logarithmic scale using --xlog and --ylog flags. In addition to the overall heatmaps, the heatmaps for individual analysis pools can be generated if the --pools flag is turned on, where certain pools can be skipped using the --omit option.
Some other expert-user options exist to control the interpolation between points, for example. The full list can be viewed using --help. A few examples of contur-plot commands are shown in Listing 21.

C.8 Visualising a single parameter point with contur-mkhtml
It is often useful to run Contur on a single parameter point (i.e. a single YODA file) from a scan, to understand which analyses are providing the exclusion power, and view the associated histograms super-imposing the measured data and the generated signal at that point. The contur-mkhtml utility prepares a summary page for the user, which concisely presents the most important histograms which contribute to the CL s exclusion at a given point.
The executable can only be used after the main contur executable has been run on the YODA file of a given point, as in the example in Listing 12. An example screenshot of the summary web page which is produced can be seen in Figure 6. The --reducedPlots flag causes only the most important histograms to be plotted, thus speeding up processing time.

C.9 Herwig-specific cross-section visualisation tools
For each point in a scan of parameters, Herwig produces log files which detail the generated cross-sections and branching ratios for the processes which contribute. This is valuable information, as one can use it to understand how the phenomenology of the model changes for different regions of the parameter space. Two helper Python executables are provided, to parse this information and present it in a digestible format.
First, contur-extract-herwig-xs-br parses the log files for a given point in the parameter scan, and returns an ordered list of processes and their cross-sections to the terminal. This list represents all the processes which contribute some configurable fraction of the total crosssection (option --tolerance, by default 1%) at that point. To aid digestion of results, similar  Figure 6: Example of the summary web page for a single-point contur run, as produced by contur-mkhtml .
processes are grouped together, with their cross-sections summed. The default summation rules are summarised below.
• Differences between leptons (electrons, muons and taus) are ignored. This behaviour can be over-ridden using --splitLeptons, --sl; • Differences between incoming partons are ignored: all flavours of quarks and gluons are merged. This behaviour can be over-ridden using --splitPartons, --sp; • Differences between particles and antiparticles are ignored. This behaviour can be over-ridden using --splitAntiparticles, --sa; • Differences between "light" quarks are ignored: u, d, s, c, b are grouped. This behaviour can be over-ridden using --splitLightQuarks, --sq, or just the b can be split out using --split b, --sb; • Optionally, one can choose to ignore differences between electroweak bosons W, Z, H using --mergeBosons, --mb; The resulting output will show the outgoing particles from the matrix element. It may be that one is interested not in the particles which come out of the hard scatter, but the stable particles one would find in the final state. To help determine this information, contur-extract-herwig-xs-br can recursively apply SM branching fractions of unstable SM particles, and can extract the predicted branching fractions of BSM particles from the log files and apply those recursively too. This behaviour can be activated using the --foldBRs,--br option or --foldBSMBRs,--br bsm for BSM decays only. Some examples of the output of the script can be found in Listing 22. A second executable, contur-scan-herwig-xs-br, can call contur-extract-herwig-xs-br at each point in a parameter scan, and present the cross-section information as a cross-section heatmap for each process. At present, the tool can only handle two-dimensional scans, and the variables to use as the x-and y-axes of the resulting plots should be provided via --xy. This script takes the same options as contur-extract-herwig-xs-br in terms of merging similar processes, and additionally can take a -p or --pools option, which further groups final states into pools of "analysis types", for example grouping together processes which have the same or similar number of leptons, photons, jets (from quarks, or gluons) or b-jets, or missing energy (from neutrinos or stable BSM particles). Examples of outputs of this tool can be found in Figure 7.

C.10 Interactive visualisation
To further aid digestion of Contur results, a web-based visualisation tool, contur-visualiser, is provided. This tools combines the Contur parameter scan, CL s calculation and Herwig log-file parsing, to build an interactive webpage where these results are presented in a combined way. The page can be opened on any browser on the local machine. The result is a heatmap showing the CL s exclusion at each point of a parameter scan, where hovering the cursor over a particular point reveals the cross-section information for that point. Clicking on a given point will cause the evaluation of a single-point Contur run (as described in Section C. 8), and will open a new page showing the summary for that point.
Since the visualiser is I/O intensive, it is recommended to run the contur-visualiser locally rather than via ssh. Users may find it convenient to run this tool via a Docker image   Once inside the container, the contur-visualiser tool should be run from the contur-visualiser directory, and requires a path to a .map file (using -m), the path to the scan directory (using -d), and the names of the variables to plot (using -x and -y).
The visualiser will then run through each point in the parameter scan directory and collect the output of contur-extract-herwig-xs-br, as well as the CL s values at each point. Then, it will create an interactive page which can be accessed by opening http://0.0.0.0:80/ on the local machine (outside the Docker container). An example of a screenshot of such a web page is provided in Figure 8. Hovering over the points on the heatmap reveals the x, y and CL s values at that point, while the side-panel shows the contur-extract-herwig-xs-br output at that point, so that the user can gauge which processes might be contributing to excluded points, for example. To dig further into the details of a given point, the user can click on a point on the heatmap, and this will trigger the terminal in the Docker container to run contur-mkhtml on that point. Once the terminal has finished running that command, a further click will open a new window, displaying the summary plots for that point, similar to those shown in Figure 6. A detailed example of how to run the contur-visualiser tool is provided in Listing 23.

C.11 Other tools
As explained in Section 4.1, the contur-mkana helps the user to generate static lists of available analyses to feed into the MCEG codes. A Herwig-style template file is created in a series of .ana files, and a shell script to set environment variables containing lists of analyses useful for command-line-steered MCEGs is also written. contur-mkthy is designed to help prepare SM theory predictions for particular Rivet analyses, for a more robust statistical treatment. This information is not always provided by the HepData entry of a given measurement, so it sometimes has to be obtained from an alternative source. This script helps the user translate the raw prediction into a format usable by Contur. This tool is not usually intended to be needed by regular users.

D BSM models as UFO files
The Universal FeynRules Object [13] (UFO) format is a Python-based way to encapsulate the Lagrangian of a new model. It contains the basic information about new couplings, particles and parameters which are required to generate BSM events. Since its inception in 2011, this format has become something of an industry standard, which is well-known and commonly used by theorists, and there exists a database of models with such implementations i . Furthermore, as the name implies, it is a format which is compatible with multiple event generators.
To get started with the study of a particular BSM model with Contur, the UFO file should be copied into the local RunInfo directory. The documentation associated with the model should give the adjustable parameter names. The precise next steps will depend upon the generator. A detailed example using Herwig is provided in section Appendix B.

E.1 Scanning over a directory
If a set of parameter points is predetermined, and the SLHA files are available, these can be used simply as the input of a Contur scan. The param file.dat syntax is as given in Listing 24.
To make the parameters from the SLHA file available for plotting in the produced .map file, use the -S parameter to pass a comma-separated list of SLHA block names to contur when running on the grid. The parameter names will have the block name prepended, for example MASS:1000022 would be the mass of the lightest neutralino.

E.2 Modifying SLHA files
If a single SLHA file is available and the user wishes to modify it, for example scanning two of the particle masses over some range, this can be done using the param file.dat syntax given in Listing 25.
This would use the SLHA file C1C1WW 300 50 SLHA.slha1 as a template, and would scan the χ 0 (PID 1000022) and χ ± 1 (PID 1000024) particle masses over the ranges specified by the mode, start, stop parameters in the specific number of steps. The modified parameters (only) would be written to the params.dat files for later use in analysis. A single letter in front of the particle ID integer is required, and should be unique to a given SLHA block, allowing (for example) several properties of the same particle (in different blocks) to be varied at once.
Alternatively, all the parameters in a single block may be scaled by a common factor, as shown in Listing 26, where the couplings in the RVLAMUDD block will also be multiplied by factors from 0.01 to 0.1, in 15 steps.

F.1 Creating pickle files
If a pandas DataFrame with column names corresponding to parameter names in param file.dat is available, data frame.to pickle('path/to/file.pkl') can be used to produce and save a pickle file to load into Contur.

F.2 Loading pickle files
If a pickle file is available, the DATAFRAME block of a parameter file can be used to specify an absolute path, or path relative to the current working directory, under the keyword name, from which to load a pickle file into Contur. Contur only supports one pickle file for each param file.dat, although an arbitrary number of parameters can be extracted from that file. An example is shown in Listing 27.

F.3 Interoperability
The DATAFRAME mode can be used alongside other modes; for modes such as LOG/LIN with more than one parameter value, the scan will occur across each entry in the pandas DataFrame. Events are generated with MadGraph5 aMC@NLO [18] through a steering script, an example for which is given in Listing 28. This is functionally comparable to the LHC.in file for steering Herwig as shown in Listing 10. In the steering script, at first MadGraph-specific variables are being set. If a grid of signal points is to be generated using a batch system, it is important to include the options set run mode 0 and set nb core 1 as by default MadGraph runs on multiple cores which can be problematic on some HPC systems. These two lines configure MadGraph correctly for single core mode and make thus more efficient use of computational resources. The desired UFO can be used by calling import UFO model directory , which -in contrast to usage of Herwig-does not need to be compiled. Afterwards, the model-specific processes are defined j and MadGraph started (launch). Parton showering for the generated events as well as giving a HepMC file as output is taken care of by Pythia [137], initialised by shower=Pythia8. Afterwards, generation-and model-specific parameters are set. Just as for Herwig, parameters should be included in curly brackets if Contur is used to generate a signal grid, otherwise concrete parameter values should be given.
After setting up the steering script, MadGraph generates events when called as $MG DIR/bin/mg5 aMC MG steering script where $MG DIR points to the installation directory of MadGraph. This will give HepMC files as output in mgevents/Events that can be processed subsequently with Rivet to obtain a YODA file. Starting from this, the steps involving Contur are almost identical to those detailed for Herwig in Sections B.4 to B.7. Due to different MC weight nomenclature within MadGraph, when running on a single parameter j In the example, the arbitrary choice of a top quark pair produced in association with the mediator is made. To generate a signal grid with Contur using contur -g, specify MadGraph to be used as the MC generator by giving the option --mceg madgraph.

G.2 Powheg support
Events can be generated in Powheg in the .lhe format using the pwhg main executable together with an input file called powheg.input. These events can then be transformed to the .hepmc format and showered using a full-final-state generator such as Pythia. These .hepmc events can then be passed through Rivet as usual to obtain a YODA file for processing by Contur to get exclusion limits.
Machinery to steer Powheg using Contur has been created based on the PBZpWp Powheg package which produces events at leading and next-to-leading order for electroweak tt hadroproduction in models with flavour non-diagonal Z boson couplings and W bosons [138,139]. Three BSM models are currently implemented, namely the Sequential Standard Model (SSM) [140], the Topcolour (TC) model [141,142], and the Third Family Hypercharge Model (TFHMeg) [143]. In what follows we exemplify this steering chain by explaining how to run jobs on a HPC system to set exclusion limits on the mass of Z in the SSM. Powheg running does not support the UFO format, but the example discussed in this section could be used as an example if one wanted to use other Powheg packages.
To run a batch job one needs three executables (main-pythia, pwhg main, and pbzp input contur.py), two files (param file.dat and powheg.input template), and one directory (RunInfo), all in one run directory. main-pythia is responsible for the creation of the HepMC file and of the parton showering. More details on these are listed below.
• The RunInfo directory contains the needed analysis steering files (.ana) and can be prepared as described in Section 4.4.
• The pbzp input contur.py script is used to create and fill the powheg.input files based on the model choice in param file.dat, it needs powheg.input template in order to do so.
• The param file.dat file defines a parameter space, as with other generators.
In the SSM, there are only two parameters, i.e. the mass (mZp) and the width (GZp) of the Z boson in GeV, but one also needs to include the name of the model (SSM in this example), and the parameters of the other models as dummy k . The param file.dat of the SSM should be formatted as in Listing 30, where setupPBZpWp.sh is a script which sets the environment needed to run pwhg main and setupEnv.sh a script which sets up the run-time environment which the batch jobs will use, as a minimum it will need to contain the lines to execute your rivetenv.sh and yodaenv.sh files. For all the setup files, one should give the full explicit path. The setupPBZpWp.sh and the setupEnv.sh should be always in the same order as shown in this example, i.e. in generator one first gives the full path to setupPBZpWp.sh then the one for setupEnv.sh. In addition, one should check that the parameters defined in params file.dat are also defined in powheg.input template, in other words, removing or adding new parameters should be done in both files. The HPC submission procedure using contur-batch follows the same workflow as for other MCEG options, but specifying --mceg pbzpwp and -t powheg.input template to indicate the correct template. When the batch job is complete there should, in every run point directory, be a runpoint xxxx.yoda file and an output.pbzpwp directory that contains the .lhe file. Creating the heatmap can then be done as explained in Sections B.6 and B.7.
k The angle θ sb (tsb) needed for the TFHMeg, and cot θH (cotH) needed for the TC model, since for now we only include the SSM, the TFHMeg and the TC models. This is done in order to be able to use the same powheg.input template for all the models.

H The analysis database
The categorisation of Rivet analyses into pools, as described in Section 3.1 is implemented in an SQLite database, distributed with Contur. The source code analysis.sql is in the data/DB directory, and after installation the compiled database will be in the same directory, named analysis.db. The database contains the following tables:

H.1 General configuration
beams Short text strings specifying known beam conditions, e.g. 13TeV.
analysis pool Defines the analysis pool names, associates them with a beam, and gives a short text description of the pool.
analysis Lists the known Rivet analyses, assigns them to an analysis pool, and stores the luminosity used, in the units corresponding to those used in the Rivet code.
blacklist Optionally, for a given analysis, defines any histograms (via regular expression matching) which should be ignored.
whitelist Optionally, for a given analysis, defines any histograms (via regular expression matching) which should be used. If an analysis has any whitelist entries, all unmatched histograms will be ignored.
subpool Optionally for a given analysis, list (and name) subsets of histograms which are know to be statistically "orthogonal" in the sense of containing no events in common.
normalization Some measurements are presented as area-normalised histograms (for example when the discussion focuses on shapes). Contur requires the cross section normalisation, so that it knows the weight with which signal events should be injected. For such histograms, this table stores this normalisation factor. For searches, where the measured distribution is often just a number of events per (number of units), this results sometimes in bins with unequal width. In this case, the "number of units" should be given in the nxdiff field. The number of events in each bin will be obtained by multiplying by the bin width and dividing by nx. If the bin width is constant, this can be left as zero, and will not be used.
needtheory Analyses which both require and use the SM prediction.

H.2 Special cases
The remaining tables define various special cases of analyses which may be included or not in a Contur run by setting command-line options at run-time. See Section 3.3 for usage and more discussion on why these special cases are treated differently. metratio Missing energy ratio measurement(s) from ATLAS. Included by default.
searches Search analyses (for which detector smearing is used). Excluded by default.
atlaswz ATLAS WZ analysis. Excluded by default.
bveto Analyses with a b-jet veto which is not implemented in the fiducial phase space. Excluded by default.