Decoding Complex Emotions Through Fusion Of Brain Computer Interface Data and Multilingual Natural Language Models

Abstract

Human‍‌‍‍‌‍‌‍‍‌ emotions are so complex and varied that it is almost impossible to decode them accurate. The data from the Brain Computer Interface (BCI), which is the main source of information for the changes in the brain is still very sensitive to small differences and ‍‌‍‍‌‍‌‍‍‌noise. In opposite multilingual natural language models are excellent at inferring verbal sentiment but they are unable to identify implicit brain signals. In this paper proposed a novel framework that combines neural signals learned through brain computer interfaces with multilingual natural language models to increase the accuracy of emotion recognition. Many modalities approach seeks to overcome the shortcomings of unimodal systems and provide accurate language independent real time emotional state recognition. The proposed work a novel method that blends multilingual natural language models with BCI data to comprehend complex emotions. Advanced preprocessing approaches like Electromagnetic Decomposition Continuous Wavelet Transform (ECD-CT) are applied to minimize noise and extract features from EEG data that were acquired during emotional stimulation. For multilingual textual data, tokenization and embedding generation are feasible. To capture round out emotional cues a Dynamic Attention Joint Embedding (DAJE) model blends the two modalities. The fusion model is refined using the Robust Cross Modal Optimization (RCMO) technique for improved accuracy and generalization. The Adaptive Real time Emotion Recognition System (ARERS) enables robust, low latency, language independent recognition for real time emotion detection. Compared to unimodal EEG (75%) and text (79%) models, the results make it obvious that the combination of EEG and multilingual text embedding considerably boosts emotion detection accuracy, reaching an accuracy of 89.7%. The F1-score of 0.88 , RCMO algorithm in conjunction with the DAJE approach adds to performance across emotions. 89.7% accuracy in real time emotion identification is achieved using low latency inference (average of 180 ms). Future research can deliberate on humanizing model adaptation for numerous real world scenarios in disturbing computing and mental health monitoring, as well as extending cross-cultural and cross linguistic submissions.