Object-based multi-modal convolution neural networks for building extraction using panchromatic and multispectral imagery

Building extraction is one of the important tasks for urbanization monitoring, city planning, and urban change detection. It is not an easy task due to spectral heterogeneity and structural diversity of the complex backgrounds. In this paper, an object-based multi-modal convolution neural networks (OMM-CNN) is proposed for building extraction using panchromatic and multispectral imagery. Specifically, a multi-modal deep CNN (the multispectral CNN and the panchromatic CNN) architecture is designed which can mine multiscale spectral-spatial contextual information. In order to fully explore the spatial-spectral information embedded in panchromatic and multispectral images, the complex convolution and complex self-adaption pooling layer are developed. Furthermore, to improve the building extraction accuracy and efficiency, a simple linear iterative clustering (SLIC) algorithm is used to segment the panchromatic and multispectral remote sensing imagery simultaneous. Results demonstrated that the proposed method can extract different types of buildings, and the result is more accurate and effective than that of the recent building extraction methods.