Detecting Deepfake-Enabled Financial Fraud Through Cross-Modal Consistency and Reliability-Aware Fusion

Abstract

Remote onboarding, video-based Know-Your-Customer (Video-KYC) checks and voice-authorized payments now let a fraudster win a customer’s or an executive’s identity without ever being physically present. Synthetic faces and cloned voices are cheap to produce, yet most defenses still look at a single signal: deepfake detectors judge the media in isolation, and fraud engines judge only the transaction. This paper presents DeepShield-Fin, a cross-modal framework that reasons jointly over video, audio, interaction behavior, session context and transaction attributes during a live financial session. A Cross-Modal Consistency Analysis (CMCA) module turns five explicit agreement tests (lip–speech synchrony, face–voice identity, facial–vocal affect, behavior–context plausibility and transaction deviation) into an interpretable evidence vector. An Adaptive Gated Cross-Attention Fusion (AGCAF) module then weights each modality by an estimated reliability, so that a noisy microphone or a dropped video frame does not dominate the decision. A calibrated risk head drives a three-tier allow/step-up/block policy, and an Evidence Explanation Engine produces analyst-readable justifications. We specify DeepShield-FinDB, a 5,000-session benchmark with an explicit label-leakage audit, and a five-protocol validation plan. Preliminary validation uses the implemented audio branch and prototype: on 22,622 benchmark clips the audio branch reaches 95.2% accuracy (95% CI 94.9–95.5%), an MCC of 0.904 and a 4.8% EER, against 7.9–8.5% EER for HuBERT, wav2vec 2.0 and RawNet2, and the prototype completes an analysis window in 1.84 s. An analytical deployment study shows why single-channel detection is not enough: at a 1% fraud rate even this detector would raise about five false alarms per detected fraud, which motivates the cross-modal evidence that DeepShield-Fin adds.