Outage, Capacity, and Error Performance of Downlink RSMA-Based Systems: Analysis and Resource Optimization

This paper comprehensively investigates the performance of downlink multi-user rate-splitting multiple access (RSMA) networks under Nakagami-m fading channels. We first develop the mathematical outage probability (OP) and ergodic capacity (EC) frameworks, deriving exact expressions for both, along with asymptotic analysis in high signal-to-noise ratio (SNR) and low-rate regions, which serve as the foundation for deducing the approximate and maximal energy-reliability and energy-spectral formulas. To enhance system performance, we tackle the non-convex problems of jointly optimizing common and private power allocation (PA) coefficients to minimize the maximal OP performance. Moreover, we also delve into optimizing PA coefficients and rate-splitting factors concurrently to maximize the ergodic sum capacity (ESC). Addressing the influence of modulation schemes on user error rates, we introduce mathematical frameworks for symbol error rate (SER) considering four combined modulation schemes based on binary phase-shift keying (BPSK) and quadrature-phase shift keying (QPSK), the whole cases are quantified in terms of exact and asymptotic manners. Furthermore, we present a straightforward approach to optimize the PA coefficients to minimize the maximal SER performance. Finally, Monte-Carlo simulations are presented to validate our developed frameworks and optimization solutions.