Detecting Deepfake Audio and Video Manipulation with Multimodal Deep Learning

Abstract

The increase in fake media production in the digital age and the rapid development of deepfake technology pose serious threats in areas such as violation of individual privacy and digital media integrity. Detecting these forgeries is becoming increasingly difficult, especially for single-mode systems that analyze only visual or auditory data. This study presents a multi-modal deep learning approach that examines both audio and video data simultaneously. While an EfficientNetB0-based feature extractor is used for image data, audio data is converted into Mel spectrograms and analyzed with a convolutional neural network (CNN). The features obtained from both modalities are integrated by the fusion method. The proposed system is trained on the DFDC (Deepfake Detection Challenge) dataset with a two-stage training process (transfer learning and fine-tuning). This study shows that the combined use of image and audio data can increase the success of deep learning-based forgery detection systems. © 2025 IEEE.