Study Finds AI Models May Help Accurately Assess PTSD in Women After a Traumatic Childbirth

A study by researchers at Massachusetts General Hospital (MGH) found that a generative artificial intelligence (AI) model can accurately screen for post-traumatic stress disorder following recent childbirth (CB-PTSD). The study explored the capabilities—as well as weaknesses—of different models from OpenAI, including ChatGPT, to identify a version that can offer insights into maternal health following a traumatic childbirth.¹

Image credit: Gary | stock.adobe.com

For this study, 1295 women who gave birth to a live baby in the last 6 months and were at least 18 years of age were enrolled and provided information on their mental health and childbirth experience through an anonymous online survey. The enrolled participants were able to recount their childbirth stories at the end of the survey, and these narratives were collected from .02 to 8.34 months following childbirth (average: 2.73 ± 1.82 months). Enrollment took place from November 2016 to April 2017, and from April 2020 to December 2020, and participants were recruited through hospital announcements, social media, as well as professional organizations.²

“Evaluation of PTSD related to traumatic birth currently relies on extensive clinician evaluation, which fails to meet the urgent need for a rapid, low-cost assessment strategy,” says Sharon Dekel, PhD, director of MGH’s Postpartum Traumatic Stress Disorders Research Program, and senior author of the study. “The use of brief patient narratives of childbirth analyzed by AI’s text-based computational methods could become an efficient, low-cost, and patient-friendly strategy for detecting CB-PTSD after a traumatic birth and with more research this tool may potentially aid in identifying women at risk for CB-PTSD before the condition fully develops.”¹

The investigators assessed PTSD symptoms with childbirth using a 20-item self-report measure that measures the presence and severity of DSM-5 PTSD symptoms after a designated traumatic event over the preceding month. Further, the study evaluated 3 different models: a zero-shot learning model, which was given no previous examples (Model 1); a few-shot learning model, which analyzed 2 narratives (1 from a women with CB-PTSD, and 1 without) to classify an expected output for a subsequent narrative (Model 2); and training a machine learning classifier, which converted narratives to a numerical using the text-embedding-ada-002 model by OpenAI (Model 3). The format is then analyzed by a trained machine learning algorithm that was developed by the team.^1,2

The findings demonstrated that Model 3 of the OpenAI text-embeddings-ada-002 model, which can convert narrative data from the personal experiences of the enrolled women with and without probable CB-PTSD to a numerical format, had a superior performance in identifying PTSD compared to the other large language models. The other models are usually trained on larger volumes of data that allows them to understand, analyze, and interpret natural language. Model 3’s performance had also surpassed the basic implementations of ChatGPT and other pre-trained large language models that are trained in both clinical and mental health spaces, supporting its potential to provide a deeper understanding into maternal mental health following a traumatic childbirth.^1,2

“The reliance of the machine learning model using childbirth narrative input from the OpenAI model as its exclusive data source presents an efficient mechanism for data collection during the vulnerable postpartum period, demonstrating 85% sensitivity and 75% specificity in identifying CB-PTSD cases,” said Dekel in the press release. “Moreover, the model we developed could potentially improve accessibility to CB-PTSD screening and diagnosis by fitting seamlessly into routine obstetric care and providing a foundation for commercial product development and mainstream adoption.”¹

The authors suggest that data from other sources, such as patient self-reports and medical records, should be incorporated to better improve the accuracy of the models used. Additionally, the authors note that the enrolled population only included women who gave birth prior to and during the COVID-19 pandemic, which may impact the heterogeneity of the study findings. Additionally, although the study utilized validated self-reports to assess CB-PTSD, evaluations by clinicians or specialists were not conducted. This should be addressed in future research, according to the authors. Further, the investigators emphasize that the content generated by AI models may not always be accurate, therefore, external validation is necessary.²

“Early intervention is essential to prevent the progression of this disorder to chronic stages, which can seriously complicate treatment,” said Dekel in the press release. “Our unique approach could introduce an innovative and cost-effective screening strategy for identifying high-risk women and facilitating timely treatment. It may also holds promise for assessing other mental health disorders, and consequently improving patient outcomes.”¹

References

1. Massachusetts General Hospital. Mass General Study Identifies an AI Model That Can Accurately Assess PTSD in Postpartum Women. News release. April 11, 2024. Accessed May 13, 2024. https://www.massgeneral.org/news/press-release/ai-method-identifying-risk-childbirth-related-ptsd

2. Bartal, A., Jagodnik, K.M., Chan, S.J. et al. AI and narrative embeddings detect PTSD following childbirth via birth stories. Sci Rep 14, 8336 (2024) doi:10.1038/s41598-024-54242-2