Enhanced Learning-Based Hybrid Optimization Framework for RSMA-Aided Underlay LEO Communication with Non-Collaborative Terrestrial Primary Network

Low Earth orbiting (LEO) satellite-assisted wireless communication is increasingly vital for future communication networks due to the significant spectrum scarcity in radio frequency channels, presenting a critical bottleneck. Thus, optimizing the utilization of available radio frequency spectrum has become imperative. Advanced techniques like underlay communication and Rate Split Multiple Access (RSMA) have proven effective in enhancing spectrum utilization. When LEO satellites are applied to tasks such as agricultural assistance, search and rescue operations, and military defense, LEO-to-ground communication can leverage underlay fashion using RSMA to transmit messages to multiple users simultaneously on the same channel. However, conventional underlay communication setups necessitate transmitter cooperation to manage system interference. Enabling non-cooperative systems to communicate in an underlay fashion unlocks the untapped potential of these advanced transmission techniques. This study addresses the challenge of maximizing the RSMA rate of the LEO-to-ground communication system (secondary system) operating in an underlay mode without cooperation with the ground-to-ground communication system (primary system), where the primary network operates in a time-division multiple-access fashion. We propose a dueling-based double deep Q-learning solution to optimize the allowed transmission power at the LEO satellite, ensuring no outage in the primary system. Additionally, we introduce an optimal solution framework to distribute the allowed transmission power among all signals of the secondary devices, maximizing the RSMA rate while meeting the rate requirements of all underlay secondary devices. Simulation results demonstrate that this hybrid solution framework provides excellent performance while ensuring no outage at the primary network.