RKKY interaction on surfaces of topological insulators with superconducting proximity effect

We consider the Ruderman-Kittel-Kasuya-Yosida (RKKY) interaction between magnetic impurities on the surface of a three-dimensional topological insulator with proximity induced superconductivity. A superconductor placed on the top of the topological insulator induces a gap in the surface electron states and gives rise to a long-range in-plane antiferromagnetic RKKY interaction. This interaction is frustrated due to strong spin-orbit coupling, inversely proportional to the distance between magnetic impurities if the distance between magnetic impurities is smaller than the superconducting coherence length, and dominates over the ferromagnetic and Dzialoshinskii-Moriya type interactions if the distance between magnetic impurities is larger than the superconducting coherence length. The condition for the subgap states that are bound to the magnetic impurities is found.