ChatGPT clears US Medical Licensing Exam

Associated Press

Los Angeles: ChatGPT could score at or around the approximately 60% passing threshold for the United States Medical Licensing Exam (USMLE), with responses that made coherent, internal sense and contained frequent insights, according to a new study.

Tiffany Kung and colleagues at AnsibleHealth, California, US, tested ChatGPT's performance on the USMLE, a highly standardized and regulated series of three exams, including Steps 1, 2CK, and 3, required for medical licensure in the US, the study said. Taken by medical students and physicians-in-training, the USMLE assesses knowledge spanning most medical disciplines, ranging from biochemistry, to diagnostic reasoning, to bioethics.

The authors found that after indeterminate responses were removed, ChatGPT had scored between 52.4 per cent and 75 percent across the three USMLE exams, the study published in the journal PLOS Digital Health said. Unlike most chatbots, ChatGPT cannot search the internet, the study said. Instead, it generates text using word relationships predicted by its internal processes, the study said.

According to the study, ChatGPT also demonstrated 94.6 percent concordance across all its responses and produced at least one significant insight, something that was new, non-obvious, and clinically valid, for 88.9 percent of its responses.