Enhancing skin cancer detection using attention-based CapsNet with transfer learning from pre-trained CNN

Skin cancer is nowadays the most common and threatening kind of cancer. It is increasing at an alarming rate, causing threats to public health. Deep learning (DL) approaches, particularly Convolutional Neural Networks (CNNs), have made tremendous progress in various image classification tasks. However, the strong dependency of CNNs on predefined anchor boxes often leads to suboptimal localization, particularly for small or low-contrast lesions that are common in skin cancer images. In this study, we proposed an attention-based Capsule Network (CapsNet) for skin image classification. It makes use of capsules, which are collections of neurons that encode data about a feature's existence and position in an image. The proposed work contributes three key steps; the first step leverages a pre-trained VGG16 model to extract rich low- and mid-level image features, reducing the need for large labeled datasets. The second step is the utilization of an attention-based CapsNet to extract spatial relationships within skin lesions, often ignored by traditional CNNs. The attention approach concentrates on the most relevant parts, increasing the potential to distinguish between subtle variations in texture, color, and structure of the images. The third step involves updating the design and training of the attention-based CapsNet to address the challenges occurred in skin lesions, including variations in size, shape, and structure. To validate the model's performance, two benchmark datasets, such as International Skin Imaging Collaboration (ISIC) and the Human Against Machine with 10,000 training images (HAM10000), were used, achieving 95.13 % and 94.73 % accuracy, respectively.