How Chatbots Can Really Help Doctors

A recently published study exposed that artificial intelligence chatbots outperform doctors when diagnosing a medical condition and explaining the reasoning behind it. This highlights the importance of training medical practitioners to use AI to their advantage.

“Only a fraction of doctors actually saw the surprisingly smart and comprehensive answers the chatbot was capable of producing,” explains Dr. Jonathan H. Chen, a physician and computer scientist at Stanford University.

The experiment involved 50 doctors – residents and attending physicians alike – who were given six case histories that were intentionally never published. Thus, doctors could be tested on them without any foreknowledge, and ChatGPT could not have been trained on them. Participants needed to give three possible diagnoses with supporting evidence for each and provide a final diagnosis plus three additional steps they would take in their diagnostic process. ChatGPT-4, a chatbot from the company OpenAI, scored an average of 90%, while doctors randomly assigned to use the chatbot got an average of 76%. As for those randomly assigned not to use it, they scored 74%. Doctors are exposed to AI tools for their work, but few know how to exploit the abilities of chatbots. “Only a fraction of the doctors realized they could literally copy-paste the entire case history into the chatbot and just ask it to give a comprehensive answer to the entire question.”