Unsupervised Domain Adaption for Multi-Modal Remote Sensing Image Semantic Segmentation via Asymmetric Boosting Fusion Network

Digital surface model (DSM) can assist optical images in semantic segmentation with complementary information and exhibit small domain shift between source and target domains for domain adaptation of remote sensing images. However, existing methods neglect the respective superiority of optical and DSM images, and diverse domain-independent features which play a critical role in domain adaptation for remote sensing image semantic segmentation. We propose a novel Asymmetric Boosting Fusion Network with a multi-head divergence discriminator, called ABFNet, which pursues the full utilization of both complementary benefits from DSM and optical images while capturing diverse domain-independent features. In particular, to capture effective class-discrimination features, we design an asymmetric mutual booster to enhance each modality according to the properties of another modality. It consists of a DSM-induced geometry booster to enhance the object position and boundary of the optical features, and an optical-induced content booster to enhance the semantic details of the DSM features. Then, we propose a multi-modal non-local fusion module to fuse the boosted features of both optical and DSM modalities. Moreover, to capture diverse domain-independent features in unsupervised domain adaptation, we design a multi-head divergence discriminator with two domain classification heads, and a parameter divergence loss constraining the inconsistency between the classifiers.