ChatGPT and OphthoQuestions

In the June 2023 edition of JAMA Ophthalmology, Andrew Mihalache and colleagues published a study in which they assessed the success of ChatGPT in answering ophthalmology questions. The authors used text-based questions from the company OphthoQuestions and found that ChatGPT answered 46% of the questions correctly.

The authors concluded that “ChatGPT … did not answer sufficient mulitple-choice questions correctly for it to provide substantial assistance in preparing for board certification at this time.” While this may be true, it is a reasonably impressive success rate in my opinion; human users certainly do not answer at nearly 100%. While the 46% success rate noted definitely seems lower than what one may expect from human users, humans are not that much more successful.