Charge order and antiferromagnetism in twisted bilayer graphene from the variational cluster approximation

1- Interesting subject and which is a hot topic of the condensed matter of 2D materials.

2- Advanced numerical method adapted to complexity of the studied system

1- This topic has been the subject of many publications since its appearance in 2018. I understand that the authors can not quote all the articles that have been published on it. However, a state of the art is missing and the comparison of the results with the recent articles dealing with the subject seems necessary.

2- The results need to be a little more detailed.

3- These results are obtained with a particular tight binding model developed for systems without interaction. It is not at all obvious that this model is valid with interactions. Of course this is often the case, which is why it is important to discuss the validity of the model used.

In this manuscript, B. Pahlevanzadeh, P. Sahebsara, D. Sénéchal study the effect of electronic interactions in magic-angle twisted bilayer graphene. They focus on quarter- and half-filling using a tight binding model proposed by Kang and Vafek (Ref. [1]) to simulate the 4 low-energy bands. For these numerical calculations they use the variational cluster approximation (VCA), which allows to include extended interactions.
The quality of the work seems to me good and the method well adapted for this study. The methods are well presented in the manuscript. The main conclusions too, however I find that the results could be a bit more detailed.
This work brings undoubtedly new results in a very active field and that is why I think it could be published in SciPost Physics, however it is to me necessary that the authors answer the different remarks and questions to be able to give a definitive opinion.

1- Update the manuscript by taking into account remarks of section “Weaknesses”.

2- The authors have recently published an article in SciPost Physics with the same model (Ref. [7] in the present manuscript). The 2 articles are not redundant. However, the authors should justify explicitly the need for a new article and explain the overall coherence of their work based on the model proposed by Kang and Vafek (Ref. [1]).

3è The figures 1 and table 1 seem to be exactly the same as the figure 1 and table 1 of the Ref. [7], is it necessary to show them again ?

4- In this work, the authors use a 12-site cluster containing 3 unit cells. Is it possible to justify this choice?

5- The strong-coupling limit is presented in section 2.1. Although this limit is interesting for itself, I do not think it is applicable to the case of magic-angle twisted bilayer graphene. Indeed in the strong coupling limit, a 4 bands model is not sufficient because the interactions will have also a strong effect on many more bands.

6- Page 10, it is written: “This Mott transition is essentially caused by extended interactions”. The authors should elaborate a bit more on this point and complete it.

7- Section 5, antiferromagnetism is shown at half filling for not to strong interaction. Can the authors specify the spatial magnetization state? Is the antiferromagnetism found for the interlayer order, the interlayer order or both?

1) Application of a well-suited technique to study symmetry breaking orders in a model with non-local interactions

1) The cited literature is incomplete
2) The study does not include the simultaneous breaking of spin/charge symmetry via appropriate Weiss fields and limits itself to one type of magnetic and charge order respectively

In this paper, the authors study the extended Hubbard model on a two-layer honeycomb lattice as introduced to describe twisted bilayer graphene (TBG) by means of the variational cluster approximation (VCA). The employed cluster technique is adequate to address the question of symmetry breaking in correlated electron systems and in combination with the dynamical Hartree approximation even non-local interactions can be -to some extend- included.
The VCA is applied to a 12-site cluster on which local and inter-site correlations are taken into account exactly. Extended inter-cluster interactions are decoupled within dynamical Hartree approximation and long-range order is allowed for by cluster Weiss fields whose strength is determined via a variational principle.
The focus is set on the Mott transition at half- and quarter filling as well as on antiferromagnetic and charge density instabilities for the respective fillings. In both cases, the authors find absence of the symmetry-breaking solutions if the extended interactions of the model obey a given set of relations (eq. (5) of the paper).
It is then concluded that for these filling ratios the ground state of the model is of Mott insulating type and not gapped due to symmetry breaking in the spin or charge sector.

The paper is interesting and can constitute an important contribution to help to pinpoint the importance of non-local interactions and correlation effects on the insulating nature of TBG at half- and quarter filling.
However, the current version of the manuscript includes a few weaknesses which need to be corrected before I can recommend it for publication in SciPost Physics.
In particular, the citations and discussion of the relevant literature need to be updated to meet the journal's acceptance criteria.

The following points should be addressed:

1) A proper comparison to the existing literature on TBG both from theory and experiment is missing. In particular, the results need to be compared to other theory papers that investigate the importance of non-local interactions for the ground state properties at half- and quarter filling.
Even more studies exist for the Hubbard model with on-site interactions applied to TBG. By mean-field-decoupling the non-local interaction terms, the authors could introduce an effective U to compare to those papers, too.

2) One of the issues of cluster techniques is always the analysis of finize-size effects. Since the 12-site cluster is close to the limit of numerically exploitable cluster sizes, a full finite-size scaling is not feasible. Still, the comparison to at least one additional cluster size/geometry would be helpful. One such candidate could be a supercluster of 8-site clusters (4 sites x 2 layers), which was already employed within VCA in similar contexts.

3) Another point concerns one of the strenghts of VCA, which is not used to full capacity here, namely the possibility to check for the competition of different symmetry-breaking fields on equal footing. For instance, at quarter filling it would be important to check for breaking of spin- and charge-order, as it has been discussed in literature for TBG at different magic angles.

4) When explaining the Hartree decoupling of the inter-cluster interactions, the authors cite Refs. 16 & 18. However, a reference to the first paper that introduced this type of mean-field decoupling in context of VCA is missing and needs to be added: PRB 70, 235107 (2004).

5) The authors explain how to determine the mean-fields in the dynamical Hartree approximation. In the present case, is there a specific reason why the authors decided to choose the variational determination of these fields over a self-consistent determination?

6) A central question for the applicability of the studied model to TBG concerns the values and structure of the interaction terms. The choice of the interactions (e.g. the relations (5) and the considered values) needs to be discussed properly. In their previous study, Ref. 7, the authors devoted a small paragraph to explaining their choice of values of U. Here, such an analysis is even more important. In particular, it should be discussed in how far the considered interactions V0-V3 agree with ab initio calculations of the screened interactions (cRPA, e.g. Refs. cited in Ref.7, or self-consistent atomistic Hartree theory, PRB 103, 195127 (2021)).

7) How robust is the absence of AF order at half-filling with respect to the choice of U when deviating from the case a=1? Is the 'critical' value of a changing with U?

8) The authors studied AF order at half-filling, but other magnetic orders are discussed in context of TBG for different fillings. Can the authors exclude other magnetic orders (e.g. FM order or on-site AF ordering between the two layers) at half- or quarter filling within their VCA setup?

Finally, some minor points:

9) Throughout the manuscript it should be specified in which units U is measured (in meV and not in units of the largest hopping?).

10) The matrix of differences between the one-body terms of the lattice and the reference system is called V, see e.g. eq(17). Although being standard nomenclature in context of VCA, this naming is slightly inept here since it can be easily confused with the interaction terms V, see e.g. eq(18).

The authors use a cluster approach treating short-range interactions within the cluster exactly.

The description of the model is insufficient.

The references are insufficient.

Not all possible charge and spin orders are analyzed. A combination of charge and spin order is not analyzed.

Because of using an effective model, order inside one unit cell of the Moire lattice is not considered. This could lead to the possibility of a local antiferromagnetic order inside the unit cell and a ferromagnetic order between different unit cells. Such possibilities are not analyzed.

The authors analyze correlated insulating states in an effective model for twisted bilayer Graphene using variational cluster approximation. They include nonlocal density-density interactions arising from the extended nature of the Wannier orbitals. To include these nonlocal interactions, they use the Hartree approximation within the VCA. They find that the insulating state at quarter filling for large enough interaction strength is a Mott state without long-range charge order. Furthermore, they find that an antiferromagnetic state only arises at half-filling if the nonlocal interactions are sufficiently weak compared to the local interaction. Otherwise, the insulating state at half-filling is a Mott state without long-range order.

Using cluster methods to analyze correlations for an effective model of twisted bilayer Graphene, the authors draw some interesting conclusions for this material.
While I would generally recommend publication, there are several issues that must be changed before publication.

(1) Although there are many publications about correlation effects in twisted bilayer Graphene in experiment and theory, the authors cite only 18 publications. This must be expanded.
A short and not extensive list would be
PhysRevX.8.031089
PhysRevB.98.081102
J. Phys. Commun. 3 035024
PhysRevLett.124.097601
PhysRevB.102.035136
PhysRevB.102.045107
PhysRevB.102.085109
SciPostPhys.11.4.083
PhysRevB.100.155145

As several of these papers also discuss Mott states and antiferromagnetic states, the current results should be compared to these previous results.


(2) During the explanation of the model, I am missing the spin degrees of freedom, which suddenly appear in equation 4. This should be expanded. Furthermore, the condition on the spin degrees should be stated in the strong-coupling section and the calculations for quarter-filling.

(3) Maybe I misunderstood something in the calculation of the strong-coupling limit. When the largest eigenvalue corresponds to the uniform state, then the lowest eigenvalues correspond to nonuniform (charge-ordered) states. Why is the ground state in the strong-coupling limit (when neglecting the kinetic energy) not long-range ordered?

(4) In the section about VCA, the details about how the finite cluster has been solved are missing. This should be expanded.

(5) The authors analyze at quarter-filling only charge order and at half-filling only antiferromagnetism. Why is there no analysis of combinations of charge and spin order? Furthermore, other long-range spin orders other than antiferromagnetism should be discussed.
If possible, the authors should include combined spin and charge order calculations in this manuscript.

(6) Before equation 28, the authors write D=-2. Do they mean D=-2U? Furthermore, it would be better to explicitly name the long-range order as m3 and m4.

(7) (Minor) This might be only my feeling, but my first thought when reading equation 18 in a section called DYNAMICAL Hartree approximation was that t is the time. However, t seems to be the hopping. It would be good to clarify this.