SciPost Submission Page
A standard convention for particle-level Monte Carlo event-variation weights
by Enrico Bothmann, Andy Buckley, Christian Gütschow, Stefan Prestel, Marek Schönherr, Peter Skands, Jeppe Andersen, Saptaparna Bhattacharya, Jonathan Butterworth, Gurpreet Singh Chahal, Louie Corpe, Leif Gellersen, Matthew Gignac, Stefan Höche, Deepak Kar, Frank Krauss, Jan Kretzschmar, Leif Lönnblad, Josh McFayden, Andreas Papaefstathiou, Simon Plätzer, Steffen Schumann, Michael Seymour, Frank Siegert, Andrzej Siódmok
This is not the latest submitted version.
|Authors (as Contributors):||Andy Buckley · Louie Corpe · Leif Gellersen · Deepak Kar · Frank Krauss · Leif Lönnblad · Steffen Schumann|
|Arxiv Link:||https://arxiv.org/abs/2203.08230v2 (pdf)|
|Date submitted:||2022-04-05 21:46|
|Submitted by:||Buckley, Andy|
|Submitted to:||SciPost Physics Core|
Streams of event weights in particle-level Monte Carlo event generators are a convenient and immensely CPU-efficient approach to express systematic uncertainties in phenomenology calculations, providing systematic variations on the nominal prediction within a single event sample. But the lack of a common standard for labelling these variation streams across different tools has proven to be a major limitation for event-processing tools and analysers alike. Here we propose a well-defined, extensible community standard for the naming, ordering, and interpretation of weight streams that will serve as the basis for semantically correct parsing and combination of such variations in both theoretical and experimental studies.
Submission & Refereeing History
You are currently on this page
Reports on this Submission
Anonymous Report 2 on 2022-5-2 (Invited Report)
I think that in its previous form, submission is not a physics paper but technical note addressed to MC authors, may be only of MCnet community. I have not much to add to previous referee opinion. I agree with it.
On the other hand , standard for definition of weights used in Monte Carlo programs would be beneficial for the community. Technical side of such definition is present, but essential, physics side is missing. If it is not provided, document will be, in my opinion, nearly useless for the community of Monte Carlo users and another failed attempt for standarisation will take place.
If the authors can provide (and agree among themselves) on the section on ``limitation/(applicability domain) for the weight use'', paper can include necessary physics content and its publication may be then justified. My suggestion follow:
For each program it should be explained what are limitation in weights use, that is what is its applicability domain. Alternatively detailed reference to manuals with page numbers etc. can be given. This is not easy, as it touch approximations used in programs design. In my opinion that is essential for the readers. Finally, template of explanations (what is required from the authors for the weight to be used), would be beneficial for future programs (programs versions) documentation too.
Anonymous Report 1 on 2022-4-29 (Invited Report)
The paper proposes a standard for naming and organizing event metadata
("weights") in Monte-Carlo generator and analysis programs.
The paper is not a physics paper. It is intended to help with
bookkeeping issues which can easily become a significant bottleneck in
maintaining and using HEP software. Therefore, a successful effort to
standardize conventions can be very useful, and a document such as the
present one should be suitable for publication in a major journal
despite the lack of proper physics content.
In the past, there have been various attempts at standards in HEP,
most notably several "Les Houches Accords". Since they were agreed
upon by major players in their respective areas of research, they were
usually adopted by the community and are still in use many years
later. However, it should also be noted that with the evolvement and
shifting focus of the field, formal conventions tend to persist in
environments which have not been anticipated when they were first
designed. Applications of previous "standards" often modify or
plainly contradict the intention of the original setup, causing new
problems for the practitioner. Many of these issues can be avoided if
in the original document, all relevant content is stated as precisely
as possible and in formal terms.
Unfortunately, while the intentions of the submitted document are
evident, its current form falls short of this goal on various levels.
1. The document is rather vague about its intended audience. It has
been submitted by the MCnet community, and it reflects on upcoming
simulation runs for LHC Run 3. It refers to software tools such as
the HepMC event format that are currently employed by some part of the
HEP community. If the intended audience was indeed restricted to this
community and time scale, the proper form of the paper would be a
public MCnet note; in this case I would not see justification for a
Conversely, if the intended audience was larger, the authors should
(1) state in concrete terms which API properties and version of
currently existing software are relevant for the implementation of the
intended standard, and (2) relate to a larger environment of
Monte-Carlo simulation for current and future experiments, e.g., how
it should apply to other existing or foreseen software frameworks,
beyond the LHC, and to expected future developments such as NNLO-level
simulation tools. I think that a standard for indexing MC event
metadata could become very useful for the whole community.
2. Basic terms such as 'weight', 'event', 'particle', 'stream',
'vector', etc. must be defined at the beginning of the document in
self-contained and formal terms as far as they are relevant. The
technical definition must be abstracted from the physical
interpretation. In the current text, they are introduced in a rather
casual way, assuming working knowledge (e.g., with HepMC) that is not
evident for users not familiar with 2022 LHC internals. The terms
should be used consistently throughout the document.
3. The scope of the standard must be stated. I understand that the
concrete syntax only defines the format of key identifiers in a lookup
table. However, there are also various recommendations for the
structure of this table (per event record), and there are remarks
concerning the complete stream of records. This leaves any reader
confused about which data objects are actually covered by the
4. It has to be stated whether and how the standard applies to data
independent of their particular storage structure. At the end of the
document there is a paragraph on optimization (i.e., using numeric
indices and a global lookup table mixed with event-local keys). It is
not obvious how such optimization interacts with the syntax and
semantics introduced at the beginning, or whether it actually
contradicts the normative part.
A data-format standard serves three purposes. A writer (human or
computer) is enabled to construct a conforming data structure, a
parser is enabled to correctly process the data syntax, and an
interpreter is enabled to process the semantics as intended.
To this end, the standard definition must be written such that a human
or computer can unambiguously decide whether a given data set is
standard-conforming or not. The current draft is full of unclear
terms like 'may', or 'should'; I do not see how it actually defines
I recommend that the document clearly separates its formal part,
namely the definitions of scope, terms, syntax and semantics, from
informal additions. The formal part alone allows for writing a
correct programming interface. The informal part may include
examples, context, reasons, use cases, or best-practice
recommendations. Actually, worked-out examples would be rather useful
and could help to straighten the core of the document.
- Sec. 2: The term 'particle-level' is misleading without context --
is the weight pertaining to a single electron? Since the detector
simulation environment is not referred to in the standard, I
recommend to use predefined technical data terms where possible.
- Syntax: it is not stated whether character case is meaningful or
not, i.e., whether "NAME" is different from "Name". It is not
stated whether keywords such as "NOMINAL" must be upper case.
(Actually, at the end of 2.2. there is a remark which suggests
that the syntax might be case-insensitive, but this is difficult to
spot, and it is again just a 'should' clause.) It might be
necessary to specify the character encoding. This depends on
whether the standard refers to abstract names or to their
representation in storage, which is not evident in the present text.
- Syntax: it is not clear without reading the text several times what
the allowed characters for ALPHA actually are. To avoid all
ambiguities, directly define ALPHA as a character set at the point
where it appears. (Same for DIGIT).
- Syntax: the 'usual standard form' of an integer or floating-point
number is unclear. The writer of a parser, for instance, must
exactly know which formats to allow and accept. Regarding
floating-point, it must be stated what the allowed ranges are and
whether numeric or textual reading/comparison is to be applied (cf.
numbers differing in the 31st decimal), whether +0.0 is equal to
-0.0, etc. All of this is relevant if the key strings themselves
- Text: what is a 'usual programming language restriction'?
- Syntax: there are potential ambiguities. E.g., with the character string
"WEIGHT=1", a strict left-right parser would treat "WEIGHT" as
NOMNAME and deduce invalid syntax, while another parser could
treat "WEIGHT" as a key with value and find no NOMNAME. This should
be resolved, e.g., by prohibiting predefined words as key
- Syntax/semantics: are integers and floating point values distinct?
- Comments regarding ROOT: is this part of the standard, or is it an
implementation detail that can be skipped? Is there an
equivalence between strings in either direction of translation,
e.g., if both formats appear as keys in the same weight set?
- Semantics/interpretation: for some applications, events are
certainly unweighted, for others they are not. Rejection sampling
is a method that need not apply to future generators, e.g.,
machine-learning based sampling. I do not see these and similar
scattered comments as relevant for the standard, so I would
recommend to defer them to a separate section which may describe the
current use inside the LHC software framework.
- Semantics: The semantic equivalence between NOMINAL, DEFAULT,
WEIGHT, 0 (or letter O?) and empty string appears arbitrary and
redundant. It is easy to miss that only one should (or must?)
appear, and confusing to any reader. I recommend that the authors
decide on one of those and allow the others, if at all, as
legacy variants that a reader but not a writer may support.
- Usage: While it is clearly stated that the NOMINAL weight is
special, I do not understand why this particular weight name cannot
be annotated by key-value pairs. If all other weight names can be
parsed and interpreted, it might be useful for an application to
know which parameters apply to the nominal weight without having
to look at metadata elsewhere.
- Semantics: e.g., comments regarding IRREG: why not just state for
any element whether it is mandatory or optional. Statements such as
'might not even exist' just confuse the reader.
- Semantics: The text states that a setting such as "KEY=" without
value may be interpreted as boolean true. To me, such a setting
rather looks like an undefined value. Is this semantics actually
- Text: The terms GenEvent/GenRun are not defined. The
statement where they appear is unclear to me, how should
documentation be accessible?
- Semantics: There is a paragraph regarding ordering of weight sets
(vectors). This looks like an unfinished work item. I recommend to
either state that this standard does cover weight sets, not just
individual weights, or not. If yes, define sets either as ordered,
partially ordered or unordered, and define the exact requirements on
ordering so they can be assumed and respected by a software
implementation. Analogous arguments apply to other paragraphs, such
as the following section on weight manipulation.
- Usage: there is a paragraph which hints at optimizations such as
storing the keys separately and using integers instead in the event
record. This may be seen as an implementation detail, but it should
be clarified if and how such optimizations must be supported by a
programming interface to this standard.
- Flexibility: The values of 'weight' entries appear to be restricted
to single floating-point values. If an event format allows other
data types such as tables, complex numbers, etc. as individual
'weight' entries, would the proposed standard be sufficent, open to
extensions, or would it become invalid?
I understand that my remarks my be seen as pedantic and beyond
the intentions of the document. However, all details and formalities
do matter if somebody wants to produce a correctly working
implementation, possibly without direct contact to the original
authors or years later.
I believe that this standard and the ideas behind it are important and
deserve publication and wide recognition. Unfortunately, in its current
version I cannot recommend acceptance without substantial revision.