Enhanced Human Activity Recognition Using Mixture of Swin Transformers with Recurrence Plots of Triaxial Sensor Data

Human activity recognition (HAR) is an innovative technology that leverages sensors and computer systems to monitor and analyze human activities across various environments and workspaces. Despite extensive research addressing the HAR challenges, significant hurdles persist in efficiently processing multimodal triaxial sensors embedded in smartphones. In this paper, we propose a novel approach utilizing a mixture of modified swin transformers to effectively manage the complexity inherent in multimodal sensor data and boost recognition performance. Our approach first preprocesses the triaxial data from various sensors through axis-wise averaging and stacking to generate 2D Recurrence Plots across three channels. Then, the 3-channel images are used to extract features by fine-tuning pretrained transformer models, which are subsequently relayed to an appropriate expert transformer for different activity recognition. The proposed model is tested on three publicly available datasets and an ablation study is conducted with varying number of experts. The experimental results and comparative analysis show that the proposed model outperforms the baseline swin models as well as other deep learning and machine learning based approaches, achieving the highest ROC-AUC and F1 score of 100% and 98%, respectively on the WISDM dataset. The model's robustness and generalization are also evaluated using the Leave-One-Subject-Out (LOSO) strategy, along with assessment of model complexity and energy efficiency. Among the other advantages of the proposed model is that it can be deployed in a federated learning environment where experts are running on different smartphone devices, protecting the user privacy.