A Comparative Study on Face/Body-Based Lie Detection Using Convolutional Neural Networks

Recognizing human emotional and physical states using physiological measurements has several potential interdisciplinary applications and use cases across multiple domains, including detecting deceptive behaviors, employee screening, safety and security, mental health assessment, stress monitoring, and identifying learning difficulties and attention disorder. This paper presents and evaluates a novel solution approach for lie detection using multi-frame video models and convolutional neural networks for video processing. Different architectures have been evaluated and compared for detecting facial and bodily clues to enhance the detection accuracy. The video stream is preprocessed by selecting key frames and applying human face and body extraction techniques as inputs to the feature learning and detection model. In an ablation study, we also evaluated various ways to combine the trained face and body models under four scenarios: combining outputs of the same model architecture for both face and body, combining different model architectures for faces, combining different model architectures for body, and combining all model architectures for face and body. Evaluations using the RealLife Trials dataset demonstrate the effectiveness of MobileNetV2 trained on full-body images, outperforming other test methods with more than 94% accuracy while significantly reducing computational costs and model size. Moreover, combining facial and bodily cues enhances model accuracy compared to using each modality in isolation.