Clin Pharmacol Ther. 2024 Jun;115(6):1391-1399. doi: 10.1002/cpt.3226. Epub 2024 Mar 8.

Anna L Silverman ^# 1 2, Madhumita Sushil ^# 3, Balu Bhasuran ^# 3, Dana Ludwig ³, James Buchanan ³, Rebecca Racz ⁴, Mahalakshmi Parakala ⁵, Samer El-Kamary ⁴, Ohenewaa Ahima ⁴, Artur Belov ⁴, Lauren Choi ⁴, Monisha Billings ⁴, Yan Li ⁴, Nadia Habal ⁴, Qi Liu ⁴, Jawahar Tiwari ⁴, Atul J Butte ^3 6, Vivek A Rudrapatna ^3 7

Author information

¹Division of Gastroenterology and Hepatology, Department of Medicine, Mayo Clinic, Phoenix, Arizona, USA.

²Department of Medicine, University of California, San Diego, La Jolla, California, USA.

³Bakar Computational Health Sciences Institute, San Francisco, California, USA.

⁴United States Food and Drug Administration, Silver Spring, Maryland, USA.

⁵Department of Public Health, University of California, Berkeley, Berkeley, California, USA.

⁶Center for Data-Driven Insights and Innovation, University of California Health, Oakland, California, USA.

⁷Division of Gastroenterology and Hepatology, Department of Medicine, University of California, San Francisco, San Francisco, California, USA.

^#Contributed equally.

Abstract

Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California - San Francisco (UCSF)-BERT, to identify serious AEs (SAEs) occurring after treatment with a non-steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE-associated hospitalizations occurring after treatment with a non-steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF-BERT achieved the highest numerical performance on identifying drug-SAE pairs from this corpus (accuracy 88-92%, macro F1 61-68%), with 5-10% greater accuracy than previously published models. UCSF-BERT was significantly superior at identifying hospitalization events emergent to medication use (P < 0.01). LLMs like UCSF-BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre-trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.