Differentiable MadNIS-Lite

Very innovative study of using ML-based methods to sample phase space for complicated processes in particle physics. The new ingredient is the decomposability into kinematic subfeatures of phase space and the differentiability of these building blocks, which opens the possibility to learn the dependence of phase space features, matrix elements and PDFs on certain parameters of the problem.

As of yet, this is still a proof-of-concept and a thorough study over the full zoo of QCD and electroweak processes in LHC (and maybe other collider) setups needs to follow. It is not clear that the number of building blocks is sufficient to describe each and every kinematic features in all setups. Potentially the dependence on "commercial" tools like pyTorch or TensorFlow might be considered a risk.

The authors present a very interesting proof-of-concept implementation
of ML-based phase-space sampling, combined with traditional adaptive
MC integration methods and - as a new ingredient - differentiable
building blocks for the components of the learning. I find the results
presented very interesting, and even if clearly that is just a first
step to potentially replace well-established PS sampling algorithms in
the future, the conceptual work in this paper deserves publication in
a journal like SciPost. Clearly, there is a lot of potential in that
framework, which is easily generalizable to more complicated processes
and setups, which to me is the main strength of the paper. The two
examples used in the manuscript are DY + 3/4 jets or ttbar + 2/3 jets,
so processes that are dominated by a single resonance or pair
production with additional jet radiation. It would be highly
interesting to see this setup being applied to multi-heavy-resonance
production (like VVV, V=W,Z) or VBF/VBS-like processes.

One point that to me did not become fully clear is how much of the
multi-channel setup of the processes studied in the paper within the
MadNIS-Lite framework is covered by the multi-channel version of the
VEGAS algorith, and what is covered by the trained spline
parameterization. Particularly (maybe I overlooked that aspect), are
channel weights among the training parameters in MadNIS-Lite or not,
and are their correlations between different channels that influence
the gain in performance during the training?

The authors chose to implement their building blocks into pre-defined
framework like pyTorch or Tensorflow, and I am curious to get to know
whether this poses any form of constraints on the implementation or
also on the maintenance of such a framework for one-and-a-half decades
of HL-LHC to come.

One question on the end of the first part of Sec. 3.2, below Fig. 2 on
8: to me it is not obvious why it is not possible to define a
multi-channel generalizatiojn of the KL-divergence. The authors
mention that this seems intrinsically possible only for the variance
loss, but somehow I stupidly fail to see why.

In the discussion of Fig. 6 in the last part of Sec. 4 the authors
discuss a point I was wondering about when looking up the W+3jet PS
parameterization in App. C: clearly for infrared QCD radiation the
outmost minimum of shat is M_W^2, however, the authors do apply a cut
to cure the IR singularity on k_23^2, so shouldn't the minimum of shat
not rather be given by M_W^2 + k_23,min^2? The authors are mentioning
this fact by the avoidance of the region < 0.2 in the sampling of the
time-like invariant.

On Appendix B, I would have the question how general the authors
consider their findings on the stability of the trainings comparing
variance, Kullbaeck-Leibler and reverse KL loss, especially when going
from analytic toy ensembles to LHC physics processes?

Some small things that I found and believe to be typos:
- In App. C there seems to be a conflict in notation between q_1 in
Eqs. (69) vs. item (ii) above Eq. (74). q_1 should be either
p1 - k5 or p1 - k1.
In Eq. (70) the third integration variable should be k_{1234}^2

Publish (easily meets expectations and criteria for this Journal; among top 50%)

- The manuscript delivers a more in-depth study on the potential of using differential programming for matrix elements, PDF evaluation, and in particular phase-space generation, than has previously been available.

- Studies like these can discover and explore new avenues in the development of efficient event generators for the HL-LHC and other future colliders, which come with a much increased requirement on simulated event sample sizes; here, first steps are shown towards using differential programming as part of the toolbox of future efficient event generation frameworks. Moreover, differential programming is helpful for understanding parameter dependencies and estimate uncertainties and can become an essential tool for simulations frameworks such as event generators.

- The introduction makes too strong statements with respect to the requirements and future development of the techniques as used in the field (see "Requested changes"); in particular, "modern machine learning methods" are implied to be the only way to deliver the required sample sizes, ignoring non-ML developments that already deliver significant speed-ups (such as "old-school" performance improvements as in 2209.00843, or the various projects porting event generation to GPU). If the authors disagree, they would need to back up their statements with references supporting them.

- The contents of Sec. 2 is not always in itself clear without referring to previous publications of the collaboration (see "Requested changes").

- In terms of the figures of merit laid out (standard deviation and unweighting efficiency), a breakthrough over existing techniques is not achieved, but modest factors of 1.5--2 are shown with respect to previous implementations in MadGraph. A comparison with tools beyond MadGraph is missing altogether; a lower computational cost of MadNIS-lite compared to the full MadNIS technique is claimed in the conclusions, but not shown.

- The quality of the baseline VEGAS-optimized results is not specified; i.e. what steps have been made to ensure that the VEGAS optimization is fully converged, in particular with respect to the unweighting efficiency, or to ensure that the number of bins is appropriate etc. These questions are particularly important because Table 3 seems to indicate that at least ten times more points are used to train the new approaches, compared to the number of points used to train the VEGAS-only benchmark.

The authors deliver a timely study of potential advantages of using differentiable programming for event generation and in particular phase-space generation. It goes beyond previous proof-of-principles by studying potential advantages of the technique for optimizing phase-space generation. The study methods and results are explained in a detailed and reproducible way.

Some of the identified weaknesses should be addressed by a minor revision, see "Requested changes".

Otherwise, the journal's acceptance criteria are met, although the author's claim that the manuscript "[details] a groundbreaking theoretical/experimental/computational discovery" is a stretch, given that known methods are applied to a previously identified problem, a proof-of-principle study of applying the method already exists, and given that the results only suggest incremental improvements at this point. The study has great merits, and there is no need to embellish it.

1. Introduction

- 1.1 In the first paragraph, the authors claim that "we will have to rely on modern machine learning (ML) [4, 5] to significantly improve the speed and the precision of these simulations". However, this statement is not supported in [4, 5], nor is it self-evident. Other methods that deliver or potentially can deliver large speed-ups, e.g. profiling and addressing bottlenecks in existing codebases as in arXiv:2209.00843, or modifications/adaptations of the relevant algorithms for optimized use on large HPC clusters or to profit from hardware accelerators such as GPU are not mentioned here, and might themselves be sufficient e.g. for the requirements of the HL-LHC. While I leave it to the authors whether they want to present their work in this wider context, I would ask to at least tone down the statement and just write something like "modern machine learning methods can help us to […]" instead.

- 1.2 Similarly, still in the first paragraph, the claim that "modern neural networks will transform" many of the components of the event generation pipeline is much too strong. I would ask the authors to tone it down. I understand that the purpose of the sentence is mostly to list relevant related work, but the current formulation wrongly implies that a *groundbreaking* paradigm shift induced by using neural networks is going on in the field. While the authors might argue otherwise, a claim like that should clearly not be done en passant, but should be properly argued.

- 1.3 In the second paragraph, the authors write that generative networks are "currently trained on and amplifying simulated training data". This formulation implies that these techniques are actively used for the production pipeline e.g. by the LHC experiments. To my knowledge, they are not. I would ask the authors to remove "currently" to avoid this possible misunderstanding.

- 1.4 In the third paragraph, the authors write "A proof of principle has been delivered for differentiable matrix elements [99], but the same methods are used for differentiable detector design […]" Should the use of the conjunction "but" here imply that only [99] is a proof of principle, and the other studies cited [100-103] are not, i.e. ready for production use? I believe not, given that at least some of these again have signifiers like "towards" or "first steps" in their titles and/or abstracts. Perhaps the authors could refine their formulation here.


2. MadNIS basics

- 2.1 Regarding the two equations in Eq. (8), the authors seem to indicate that these are two possible prior assumptions. Are they? But then the authors should not use the same symbol, since they are not equal (perhaps the authors accidentally used the same superscript?) Also, while I understand that this section is just meant as a quick review, please make clear to introduce all symbols that appear in Eq. (8). Currently, this is not the case.

- 2.2 In Eq. (10), theta seems to appear for the first time, but it is not introduced as a new set of network parameters if I'm not mistaken. I would ask the authors to introduce theta in case I have not overlooked it.


4. Differentiable phase space — MADNIS-Lite

- 4.1 There is a typo in the beginning of Sec. 4.2. "For typical a […]" Swap "typical" and "a".

- I would ask the authors to provide details on the VEGAS baseline. How did the authors ensure that the optimization is fully converged (or trained with a similar computational effort)? As the unweighting efficiency depends on the tail of the weight distribution, further optimisation can still make a large difference even if e.g. the integral itself has already converged nicely. Table 3 seems to indicate that the authors use at least ten more points for training the MadNIS-Lite parameters compared to what the authors use to optimize the baseline VEGAS grids (but perhaps I am misunderstanding the numbers given in the table). This becomes even more puzzling considering that the number of VEGAS parameters is much higher than the number of MadNIS-Lite parameters, as the authors state in the "Implementation" paragraph. More details would help to understand better how the baseline the authors compare against is constituted, which is important to fully appreciate their results.


5. Outlook

- 5.1 In the conclusions on MadNIS-Lite, the authors write that "for a set of benchmark processes, they show great promise and low computational cost." This is a bit vague, in particular given that there is no direct comparison with the full MadNIS results given in the draft, and because there is no quantifiable data on computational cost. I would ask the authors to write more explicitly what the conclusions for MadNIS-Lite are, given the results in the draft. This would also further contextualize the outlook on the MadGraph developments in the last paragraph.

Ask for minor revision