Stacked Transformer Ensemble Framework with XGBoost Meta-Learner for Suicide Risk Detection in Social Media Text

Abstract

Suicide has been identified as a major global public health issue. It is reported that there are many uses of  mental health-related disclosures via social media platforms. There have also been reports indicating that  there may be a means to provide timely interventions using automated systems to detect suicidal ideations  within user-created texts. However, previous models have demonstrated limitations in detecting suicidal  ideations due to class imbalance issues, poor context understanding capabilities and the failure to generalize  well when trained using other data sources. The objective of this paper is to develop and evaluate a novel  Stacked Transformer Ensemble Framework (STEF), which utilizes three different transformer-based  architectures: BERTweet, RoBERTa and DistilBERT at the probability level and employs a probabilistic  XGBoost classifier as its final decision-making component. These architectures were selected because they  represent complementary domains of knowledge and have demonstrated success when used individually  for NLP tasks. The STEF was tested using the Generalized Suicide Detection Dataset, which consists of  57,306 social media posts with a large degree of class imbalance (approximately 92% suicidal) and  employed a class weighted cross-entropy loss function to address the uneven distribution of classes when  training the transformers. Results indicate that the STEF outperformed each individual baseline architecture  for all metrics including precision (98.26%), recall (99.05%), ROC-AUC (99.65%) and MCC (88.44%).  An additional evaluation consisting of a tenfold cross validation assessment across seven different  configuration options further supported the STEFs ability to perform consistently regardless of the data  used for testing. These results support the concept that combining multiple domain-specific transformer based architectures using meta learning provides a substantial improvement over individual transformer based models for identifying individuals who are at a higher risk for suicide. Therefore, this model  demonstrates potential as a viable solution for both clinical and social media based mental health  monitoring.