If you are one of the more than a billion users of ChatGTP, you know that the AI is kind of amazing. It’s great for holding a bizarre but interesting conversation. While many people treat it like a search engine, it’s not one. It doesn’t search the internet for information; it was trained on a set body of sources, including scientific journals, books, websites and Wikipedia. Because it has a limited library that has not been updated since 2021, it’s not always right when you ask it a question.
Researchers at the Univ. of Maryland School of Medicine wanted to find out how good ChatGTP is at giving medical advice. The number of people turning to it for answers is growing steadily; it answers in clear, simple language. They asked it 25 questions about getting screened for breast cancer. It got 22 questions right. One question it answered with outdated information. It also cited fake journal articles and organizations to support its claims.
The researchers pointed out that Google would give more information without adding fictitious sources but pointed out the benefits of ChatGTP. “We found ChatGPT answered questions correctly about 88 percent of the time, which is pretty amazing,” said study corresponding author Dr. Paul Yi, Assistant Professor of Diagnostic Radiology and Nuclear Medicine at UMSOM. “It also has the added benefit of summarizing information into an easily digestible form for consumers to easily understand.”
But, Dr. Yi said, “ChatGPT gave inconsistent responses, meaning if you asked it on any given day you’ll hear different answers that often contradicted each other, or the responses were just flat-out wrong.”
The advice always used to be, “Don’t trust Dr. Google.” It seems it should be updated to “Trust Dr. ChatGTP even less than Dr. Google.”
And ChatGTP admits its flaws stating, “ChatGPT can be a helpful tool for gaining a general understanding of medical concepts and terminology. In any case, it’s essential to verify any information provided by ChatGPT with a qualified medical professional to ensure that it’s accurate and applicable to your situation.”
Dr. Yi said, “Consumers should be aware that these are new, unproven technologies, and should still rely on their doctor, rather than ChatGPT, for advice.”
It’s nice to think of a time when we will have all the medical answers just a few clicks away. But, as convincing as ChatGTP may be, it’s not the solution it seems to be. While it was correct 88 percent of the time, the 12 percent when it was wrong matters — especially when it’s making up sources about cancer information. So, instead of turning to the internet, talk to your doctor about all your medical concerns.
