RecognAVSE-V2: An Improved Cross-Attention-Based Audio-Visual Speech Enhancement Approach

Audio-visual speech enhancement (AVSE) is a sub-field of machine learning that aims to improve speech quality from noisy audio signals, leveraging visual cues to guide information flow and enhance learning. In this context, the COG-MHEAR program supports an early competition, namely the AVSE Challenge (AVSEC), to advance the development of new solutions and AVSE methods while fostering the community. This paper introduces the RecognAVSE-V2, an improved and more efficient version of RecognAVSE proposed at AVSEC 2024, that implements a Time-Synced Cross-Attention Mechanism that models the temporal dependencies between both modalities. Its architecture consists of a video encoder for spatiotemporal feature learning, an audio encoder operating on STFT representations, and a time-synchronized cross-Attention module that aligns audio and video features. Experimental results conducted over AVSE Challenge and CHiME3 datasets show RecognAVSE-V2 is capable of outperforming the baseline and its prior version in some cases while consisting of a model with reduced complexity, 20 times smaller, whose inference is 26% faster, and training demands a quarter of the total time.