Topological multiferroic order in twisted transition metal dichalcogenide bilayers

1- A major and important topic of condensed matter in 2D materials.

2- The numerical method used is adapted and seems to me correctly applied.

3- A thorough, detailed and well explained study.

1- Is the model used (Wannier Hamiltonian) relevant to study magnetism of twisted transition metal dichalcogenide (TMD) bilayers ? In particular, this approach does not directly take into account the specific roles of the d-orbitals (Transition Metal) and p-orbitals (Chalcogenides) which do not have the same characteristics in terms of magnetization.

2- This study only considers magnetism in the z direction, but the magnetic moment may be in favor of the xy plane rather than the z direction, especially for a system that is not half filled.

This paper presents a detailed numerical study of the magnetism of low angle twisted bilayer TMD, using a Wannier Hamiltonian. It is shown how a topological multiferroic order can emerge in this system at quarter-filling. The competitive effects of on-site and first-neighbor interaction terms are studied in detail. As a result, a multiferroic order displaying simultaneously ferroelectric and ferrimagnetic orders can be obtained. Spin-orbit interactions induce a topologically non-trivial multiferroic order, and topological excitations can be control by and external electric fields. The author have also studied the impact of an underlying substrate in the moiré system by considering super-moiré cell, and thus show the emergence of topological excitations. This theoretical work is very rich and brings advances in a complex and new subject the physics of the twisted bilayer semiconductor. It is well presented and I think it deserves to be published in SciPost Physics.

To reinforce the relevance of this work it seems important to me to discuss the following points in more detail.

1- Wannier Hamiltonian model is widely used in the literature to study the electronic properties of the low-energy states of twisted bilayer. However, its relevance to the study of magnetism is not obvious. The low energy eigenstates, that are well simulated by the Wannier Hamiltonian (without interaction), are complex and localized on a large number of atomic orbitals (mainly d-obitals of metal). When the interaction terms are on, the local magnetic order (at the atomic scale) can be complex. Is it well simulated by Wannier states? It would be interesting for the authors to discuss the consequences of their results for the magnetization of the d-orbitals of metal.

2- If I am not mistaken, the xy plane magnetization is not include in the calculations. The authors should justify that point, especially for a calculation that is not at half-filling.

3- The authors focused on the quarter-filling case. It would be nice to justify this choice a little more. Is it also possible to discuss, at least qualitatively, the cases of other fillings?

4- The SciPost latex style should be used to produce the manuscript.

See attached report

See attached report

See attached report

See attached report

The authors study topological multiferroic order in twisted transition dichalcogenides bilayers (TMDs) in the presence of spin-orbit interactions and applied electrical field. Starting from a model Hamiltonian that captures nearest-neighbor hopping between the AB/BA sites forming an effective honeycomb lattice, the authors analyze spin-/charge excitations (SDW/CDW) that emerge due to on-site and nearest-neighbor interactions within a self-consistent mean-field theory. They demonstrate that spin-orbit interactions render this model a topological multiferroic with non-vanishing Chern Number whose magneto-electrical coupling can further be tuned by the application of electrical fields. The topological character manifests in the existence of interface modes between either ferrimagnetic (FM), ferroelectric (FE) or multiferroic (FEM) domains. These findings are argued to exist in super-moire potentials arising from twisted TMDs encapsulated in substrates, e.g. hBN, with slightly different lattice constant. The authors demonstrate that the additional moiré potential leads to the formation of different topological regions of the multiferroic and hence naturally to the existence of topological interface modes.

The realization of topological states in twisted TMDs with strong intrinsic spin-orbit coupling represents an uprising field of research beyond the physics of twisted multilayer graphene and aligns with recent theoretical and experimental efforts. The discussion of topological multiferroic order is novel and adds to the toolbox of correlated topological phases that can be realized via moiré engineering. The connection to the super-moiré potential as possible candidate to host topological interface modes displays an interesting avenue, in particular because such effects have been observed experimentally in twisted trilayer graphene (TTG) to have major impact on correlated quantum states. The manuscript is soundly written and easy to follow for the reader. I therefore recommend publication as long as the following (minor) points are addressed properly:

Some technical aspects could be described in more detail to provide further clarification or to eventually allow for easier reproduction of the results, especially since there is no Appendix/Supplementary Material. Some aspects I came across are the following:

1.) Concerning the TMD model Hamiltonian: The nearest-neighbor honeycomb model has been widely studied in the literature within, e.g. mean-field approximation, functional renormalization group studies, quantum Monte Carlo etc. Most of these studies focus to half-filling/filling in the vicinity to the van-Hove singularity. Could the authors clarify how quarter-filling in their manuscript is related to the position of the vHS? This would make it easier to compare with existing results.

2.) The authors write: „The interacting model is solved using a self-consistent mean-field procedure including all the Wick contractions, including magnetic symmetry breaking, hopping renormalization, and charge order.“ I would prefer to avoid the statement „all Wick contractions“ as e.g. superconducting order is not accounted for as another possible Wick contraction or do the authors also account for this kind of instability, even though it may be irrelevant for purely repulsive interactions? Furthermore, what are the technical details of the mean-field analysis? How many moiré unit cells were taken into account (or what momentum resolution was used to sample the Brillouin zone) for the self-consistent procedure? Were the calculations performed in real- or momentum-space? Did the initial guess for the self-consistent procedure already include multiferroic order or was the initial state chosen randomly?

3.) Concerning the results of the mean-field analysis: It is interesting that for U/t<4 and up to V/t<2 no charge-density wave is present, but the latter only emerges for U/t>4 in the multiferroic phase. Is there an intuitive picture to understand why the formation of CDW order is diminished at U/t=0?

4.) The authors write: „Since the origin of the interface modes is topological, small reconstructions at the interface will not impact the boundary modes“. For simple graphene, the edge termination (zig-zag/armchair) is crucial for the existence of edge states. Have the authors tried different edge terminations for the L/R domains and found invariant results?

5.) How is the momentum dependence of the interface spectral function A(w, k_x) related to the sketch in Fig. 5 a) and what is the length of the domains in numerical simulations? From the sketch, I would intuitively expect that the system is periodic in y-direction and the interface separates the junction into the left/right domain along x. Then the momentum k_y would describe the bulk dependence. Maybe it would be helpful to add a small coordinate system in Fig. 5a) or to add a more detailed instruction why/how the spectral function A(w, k_x) is computed.

6.) Concerning the super-moiré potential in the last section: Can the authors give the functional form of the moiré potential and how it couples to the Wannier moiré orbitals? The potential looks to preserve the D_6h symmetry of the original honeycomb lattice, though the LDOS in the multiferroic state only seems to preserve D_3h. Is that due to the inversion-breaking Rashba SOC term? Furthermore, how are the Chern Numbers calculated for different regions of the super-moiré? Was the set of parameters for the mean-field Hamiltonian (m, Delta_z, lambda_R) the same as in the section before?

I hope the authors appreciate the comments given above, which should not distract from the high quality of the manuscript.

Typos:

1.) Introduction: „which may drive a non-trivial topological character in a the moire system[16].“
2.) Manuscript: „moire“. In the literature (and for all cited References) „moiré“ seems to be the usual term, but I leave this point up to the authors.
3.) Page 5: Up to this point, we have been based all the analyses on a full selfconsistent solution of the interacting Hamiltonian of eq. (1).
4.) Page 6: and acts as an independent topological source.