DoctorBOT: An AI-powered Chatbot for Evidence-Based Obstetric Hemorrhage Management in Low-Resource Settings

Abstract

<title>Abstract</title> <bold>Background:</bold> Postpartum hemorrhage (PPH) remains a leading cause of maternal mortality, especially in low-resource settings. Despite the existence of high-quality clinical practice guidelines (CPGs), their implementation in rural areas is limited due to lack of training and access. Artificial intelligence (AI)-based tools may bridge this gap by supporting real-time clinical decision-making. <bold>Objective:</bold> This study presents the development and evaluation of DoctorBOT, an AI-powered chatbot using Retrieval-Augmented Generation (RAG) to deliver evidence-based recommendations for PPH management. <bold>Methods:</bold> DoctorBOT was trained using vectorized high-quality obstetric CPGs. Clinical questions were collected from 26 rural healthcare workers and evaluated by large language models (LLMs) ChatGPT-3.5 and GPT-4. Performance was assessed through expert evaluation (Likert scale across five domains) and automated semantic comparison using BERTScore. <bold>Results:</bold> GPT-3.5 outperformed GPT-4 in human evaluations for accuracy and clinical utility. Conversely, GPT-4 achieved a higher BERTScore (0.83 vs. 0.79). LLaMA2 achieved the highest score (0.92) after additional fine-tuning. In a user survey, 85% of rural healthcare providers rated DoctorBOT as “very satisfactory.” <bold>Conclusion:</bold> DoctorBOT demonstrates strong potential to support evidence-based clinical decision-making in obstetric emergencies. Its integration could reduce maternal mortality and improve health equity in underserved areas.