J Crohns Colitis. 2024 Aug 14;18(8):1215-1221. doi: 10.1093/ecco-jcc/jjae040.

Martina Sciberras ¹, Yvette Farrugia ¹, Hannah Gordon ^2 3, Federica Furfaro ⁴, Mariangela Allocca ⁴, Joana Torres ^5 6 7, Naila Arebi ^8 9, Gionata Fiorino ^4 10, Marietta Iacucci ¹¹, Bram Verstockt ^12 13, Fernando Magro ¹⁴, Kostas Katsanos ¹⁵, Josef Busuttil ¹⁶, Katya De Giovanni ¹⁶, Valerie Anne Fenech ¹, Stefania Chetcuti Zammit ¹, Pierre Ellul ¹

Author information

¹Department of Medicine, Division of Gastroenterology, Mater Dei Hospital, Msida, Malta.

²Department of Gastroenterology, Barts Health NHS Trust, London, UK.

³Translational Gastroenterology and Liver Unit, John Radcliffe Hospital, University of Oxford, Oxford, UK.

⁴IRCCS OSPEDALE San Raffaele, Gastroenterology and Endoscopy, IBD Center, Milan, Italy.

⁵Division of Gastroenterology, Hospital da Luz, Lisbon, Portugal.

⁶Division of Gastroenterology, Hospital Beatriz Ângelo, Loures, Portugal.

⁷Faculdade de Medicina, Universidade de Lisboa, Lisbon, Portugal.

⁸Department of Inflammatory Bowel Disease, St Mark's National Bowel Hospital, London, UK.

⁹Department of Metabolism, Digestion and Reproduction, Imperial College London, London, UK.

¹⁰IBD Unit, San Camillo-Forlanini Hospital, Rome, Italy.

¹¹APC Microbiome Ireland, College of Medicine and Health, University College of Cork, Cork, Ireland.

¹²Department of Gastroenterology and Hepatology, University Hospitals Leuven, KU Leuven, Leuven, Belgium.

¹³Department of Chronic Diseases and Metabolism, KU Leuven, Leuven, Belgium.

¹⁴CINTESIS@RISE, Faculty of Medicine of the University of Porto, Porto, Portugal.

¹⁵Division of Gastroenterology, Department of Internal Medicine, Faculty of Medicine, University of Ioannina School of Health Sciences, Ioannina, Greece.

¹⁶Association for Crohn`s and Colitis, Malta.

Abstract

Background: As acceptance of artificial intelligence [AI] platforms increases, more patients will consider these tools as sources of information. The ChatGPT architecture utilizes a neural network to process natural language, thus generating responses based on the context of input text. The accuracy and completeness of ChatGPT3.5 in the context of inflammatory bowel disease [IBD] remains unclear.

Methods: In this prospective study, 38 questions worded by IBD patients were inputted into ChatGPT3.5. The following topics were covered: [1] Crohn's disease [CD], ulcerative colitis [UC], and malignancy; [2] maternal medicine; [3] infection and vaccination; and [4] complementary medicine. Responses given by ChatGPT were assessed for accuracy [1-completely incorrect to 5-completely correct] and completeness [3-point Likert scale; range 1-incomplete to 3-complete] by 14 expert gastroenterologists, in comparison with relevant ECCO guidelines.

Results: In terms of accuracy, most replies [84.2%] had a median score of ≥4 (interquartile range [IQR]: 2) and a mean score of 3.87 [SD: ±0.6]. For completeness, 34.2% of the replies had a median score of 3 and 55.3% had a median score of between 2 and <3. Overall, the mean rating was 2.24 [SD: ±0.4, median: 2, IQR: 1]. Though groups 3 and 4 had a higher mean for both accuracy and completeness, there was no significant scoring variation between the four question groups [Kruskal-Wallis test p > 0.05]. However, statistical analysis for the different individual questions revealed a significant difference for both accuracy [p < 0.001] and completeness [p < 0.001]. The questions which rated the highest for both accuracy and completeness were related to smoking, while the lowest rating was related to screening for malignancy and vaccinations especially in the context of immunosuppression and family planning.

Conclusion: This is the first study to demonstrate the capability of an AI-based system to provide accurate and comprehensive answers to real-world patient queries in IBD. AI systems may serve as a useful adjunct for patients, in addition to standard of care in clinics and validated patient information resources. However, responses in specialist areas may deviate from evidence-based guidance and the replies need to give more firm advice.