Enhanced Human Activity Recognition Using Mixture of Swin Transformers With Recurrence Plots of Triaxial Sensor Data

Human activity recognition (HAR) is an innovative technology that leverages sensors and computer systems to monitor and analyze human activities across various environments and workspaces. Despite extensive research addressing HAR challenges, significant hurdles persist in efficiently processing multimodal triaxial sensors embedded in smartphones. In this paper, we propose a novel approach utilizing a mixture of modified Swin Transformers to effectively manage the complexity inherent in multimodal sensor data and enhance recognition performance. Our approach first preprocesses the triaxial data from various sensors through axis-wise averaging and stacking operations to generate 2D Recurrence Plots across three channels. Then, these images are used to extract features by fine-tuning pretrained transformer models, which are subsequently relayed to a mixture of expert transformers to classify different activities. The proposed model is tested on three publicly available datasets and an ablation study is conducted by varying the number of experts. The experimental results and comparative analysis show that the proposed model outperforms the baseline Swin models as well as other deep-learning and machine-learning approaches, achieving the highest AUC and F1 scores of 100% and 98%, respectively on the WISDM dataset. The model’s robustness and generalization are also evaluated using the Leave-One-Subject-Out Cross-Validation (LOSOCV) strategy, along with an evaluation of the model’s complexity and energy efficiency. Among the other advantages of the proposed model is that it can be deployed in a federated learning environment where experts are running on different smartphone devices to protect the user’s data privacy.