Enhanced Hybrid Beamforming and Signal Detection of OTFS in Presence of Hardware Impairments

Orthogonal time frequency space (OTFS) modulation is a promising approach to improve the performance of millimeter wave (mmWave) communication systems at high mobility while leveraging the wide available bandwidth. However, the high mobility and frequencies in the mmWave regime increase the sensitivity of transceivers to hardware impairments (HIs) such as in-phase and quadrature (IQ) imbalance and direct current (DC) offset, degrading the OTFS performance. We develop an unsupervised deep learning (DL)-based approach to learn a hybrid precoder for a mmWave multi-user (MU) multiple-input and multiple-output (MIMO)-OTFS system, referred to as hybrid beamforming MIMO OTFS (HM-OTFS). In addition, a convolutional neural network (CNN)-based signal detector is proposed for the HM-OTFS system to mitigate the impact of HIs. Our results show that the proposed DL-based beamforming (DLBF) outperforms conventional hybrid beamforming (HBF) schemes aided with estimation and compensation of HIs, providing a performance improvement of more than 2 dB. Furthermore, the proposed CNN-based detector provides a huge performance improvement, compared to conventional minimum mean square error (MMSE) and message passing algorithm (MPA) based detectors, even in the presence of imperfect channel state information (CSI). Extensive simulations establish the bit error rate (BER) performance of the proposed schemes in the presence of HIs, with variations in parameters such as number of users, user's mobility, HIs characteristics, and MIMO configuration.