Fusing Transformers in a Tuning Fork Structure for Hyperspectral Image Classification Across Disjoint Samples

The 3-D swin transformer (3DST) and spatial-spectral transformer (SST) each excel in capturing distinct aspects of image information: the 3DST with hierarchical attention and window-based processing, and the SST with self-attention mechanisms for long-range dependencies. However, applying them independently reveals the following limitations: the 3DST struggles with spectral information, while the SST lacks in capturing fine spatial details. In this article, we propose a novel tuning fork fusion approach to overcome these shortcomings, integrating the 3DST and SST to enhance the hyperspectral image (HSI) classification (HSIC). Our method integrates the hierarchical attention mechanism from the 3DST with the long-range dependence modeling of the SST. This combination refines spatial and spectral information representation and merges insights from both transformers at a fine-grained level. By emphasizing the fusion of attention mechanisms from both architectures, our approach significantly enhances the model's ability to capture complex spatial-spectral relationships, resulting in improved HSIC accuracy. In addition, we highlight the importance of disjoint training, validation, and test samples to enhance model generalization. Experimentation on benchmark HSI datasets demonstrates the superiority of our fusion approach over other state-of-the-art methods and standalone transformers. The source code has been developed from scratch and will be made public upon acceptance.