Relative entropy for power-law correlated cluster sequences

Thank you very much for considering the topic interesting, the results sound, and for recommending the acceptance of our manuscript. We deeply appreciate the time you spent to read our work and the requested changes of better clarifying the motivation of our study, whose inclusion in the Introduction of the revised version will certainly add value to the manuscript.

The text below summarizes the motivations for proposing the "relative cluster entropy" $\mathcal{D_{C}}(P \| Q) $ and the main differences with respect to the simple "cluster entropy" $\mathcal{S_C} (P)$.
This text will be included in the Introduction of the revised manuscript.

Answer to question:

$\mathcal{S_C} (P)$ holds the main properties of the Shannon functional, in particular, $ 0 \leq \mathcal{S_C} (P) \leq \ln N $. If $P$ is a distribution concentrated on a single cluster value, $\mathcal{S_C} (P) = 0 $ corresponds to the minimum uncertainty on the outcome of the cluster, the random variable of interest. If $P$ is a fully developed power-law distribution, $\mathcal{S_C} (P)=\ln N$ corresponds to the maximum uncertainty obtained as the power-law distribution spreads over a broad range of cluster values. Thus, according to the Shannon interpretation, $\mathcal{S_C} (P)$ is a measure of randomness of all the possible cluster outcomes or equivalently a measure of the variability associated with the random variable.

In this work, we go beyond the simple "measure of uncertainty of the random variable (the cluster size)" provided by $\mathcal{S_C} (P)$.

We develop an "inference method for hypothesis testing of a general class of models" underlying the stochastic phenomena under investigation. We put forward the $\mathcal{D_{C}}(P \| Q) $ cluster divergence with the first argument $P$ the empirical distribution and the second argument $Q$ a model within a broad class of probability distributions. It is a metric on the space of probability distribution, called a divergence rather than a distance since it does not obey symmetry and triangle inequality.

The asymmetry of the relative entropy reflects the asymmetry between data and models, hence it can be used for inference purposes on the model underlying a given distribution. If $\mathcal{D_{C}}(P \| Q) >> 0 $, the hypothesis likelihood is very low and, unless the quality of the empirical data should be questioned, the model distribution $Q$ must be rejected. The higher $\mathcal{D_{C}}(P \| Q) $, the lower the likelihood of the hypothesis. This property allows to use $\mathcal{D_{C}}(P \| Q) $ for hypothesis test about a model distribution $Q$. If the hypothesis on the model were true, $P$ should fluctuate around its expected value $Q$, with fluctuations of limited amplitude, with occurrence probability greater than the significance level, permitting the acceptance of the hypothesis on the model $Q$.

The ability of the divergence to select an optimal distribution might be relevant in several contexts and in particular complex phenomena obeying power-law distributions. The accuracy of the estimation of the power law exponent is still actively debated by the scientific community (Ref. Aaron Clauset et al, SIAM review 2009).

Thank you very much for the report and for considering the proposed approach interesting.
We deeply appreciate the time spent to read our work.

Thanks a lot for reading the manuscript and acknowledging the importance of the approach from both the fundamental and applied perspectives.

All the four minor corrections requested in your report will be implented in the revised version of the manuscript.

Thank you very much for the fruitful comments. Part of the text below clarifying these comments will be included in the revised manuscript. Here below our response to the questions 1 and 2 follows:

Answer to Question 1:

The cluster entropy $\mathcal{S_{C}}(P)$ and the relative cluster entropy $\mathcal{D_{C}}(P \| Q) $ can be interpreted as an information measure over a partition generated by a coarse-grained mapping of the two-dimensional phase-space spanned by a particle (e.g. a simple Brownian path $\{x_t \}$). According to Gibbs’ original idea at the core of the information entropy concept, a coarse grained description is defined by smoothing out the fine details of the system hence increasing the observer’s ignorance about the exact microstate of the system. As the structure of the system becomes blurry, randomness and entropy increase. Coarse-grained descriptions are obtained by performing a local average over each cell of the partition in phase space: average temperature or free energy are classical examples.

In the information clustering approach proposed in our manuscript, the coarse grained description of the particle path $\{x_t \}$ is obtained by a local average $\widetilde{x}_{t,n}$ over the phase-space cells where $n$ indicates the cell size. The linear regression $ {x_{t}}= \widetilde{x}_{t,n}+\epsilon_{t,n} $ yields the error in the estimate $\epsilon_{t,n} ={x_{t}}- \widetilde{x}_{t,n}$, which quantifies the loss of structure and information of the coarse grained description. The values of $t$ when $\epsilon_{t,n}=0$ corresponds to no loss of information and to the minimum value of the cluster entropy $\mathcal{S_{C}}$ . The errors $\epsilon_{t,n}$ generate a random sequence of macrostates, which ultimately define for each $n$ a finite partition $\mathcal{C}_{n,j}=\left\{C_{n, 1}, C_{n, 2}, \ldots\right\}$ of macrostates of the particle path clustered between the values of $t$ when $\epsilon_{t,n}=0$.
$\mathcal{S_{C}}(P)$ is bounded, involves integrating over cell components, ranges from a minimum value ($0$) at the finest-grained description (corresponding to smallest clusters) towards the maximum value ($log N$) at the coarsest-grained description. In other words, $\mathcal{S_{C}}(P)$ provides a tool to estimate the entropy or randomness increase as the partition becomes coarser and coarser.
$\mathcal{D_{C}}(P \| Q) $ is also bounded, involves integration over cells, ranges between the maximum value given by Eq.(11) and the minimum value which is zero for $P=Q$.

Answer to Question 2:

The notation in Eq. (6) of the submitted manuscript was really unfortunate as we used the symbol $D_V $ to indicate the variance of the relative cluster entropy with respect to the null hypothesis of zero divergence. In this answer we provide the revised notation for Eq.(6) where the variance of $\mathcal{D}_{C}[P|| Q ] $ is indicated by $\sigma^2_{\mathcal{D}_C} $ (thus using the common notation) . We also argue on the reason why the quadratic behaviour of the variance conveniently provides, in a straightforward and accurate way, the minimum value of the relative cluster entropy (and ultimately the corresponding optimal power law exponent of the analysed data) rather than using directly the value of $\mathcal{D}_{C}[P|| Q ] $. For the sake of completeness, we report below the updated text and notation:

"To obtain the best estimate of the probability distribution $P$, the minimum relative entropy principle is implemented non-parametrically on the values plotted in Fig.4 of the submitted manuscript. To this purpose, the variance $\sigma^2_{\mathcal{D}_C} $ of $\mathcal{D}_{C}[P|| Q ] $ around the value $\mathcal{D}_{C}[Q|| Q] =0 $, i.e. the null hypothesis expected for $P=Q$ is estimated:
\begin{equation*}
{\sigma^2_{\mathcal{D}_C}} \equiv \frac{1}{k-1} \sum_{j=1}^k
\left[\mathcal{D}_{C}[P|| Q]- \mathcal{D}_{C}[Q||Q]\right]^2 \equiv \frac{1}{k-1} \sum_{j=1}^k
\left[\mathcal{D}_{C}[P|| Q]\right]^2
\end{equation*}
where the sum runs over the range of cluster indexes obtained by the partition process. The right side of Eq.(6) corresponds to the mean square value of the area of the region between the curve $\mathcal{D}_{C}[P||Q]$ and the horizontal axis. Given the linearity of the relative entropy operator, $\sigma^2_{\mathcal{D}_C} $ is positive defined has a quadratic behaviour characterized by the typical asymmetry of the Kullback-Leibler entropy and thus can be easily used to estimate the minimum of the relative cluster entropy."