Professor Jennifer Grant, with UW-Stout’s biology department, has been curious about how well everyday chatbots dispense health information, including pharmacology advice and influences on human health.
“You sometimes hear that these tools can give distorted or misleading answers, and that’s what first caught my attention,” she said.
To quell her curiosity, Grant began several lines of AI research and developed projects for her introductory pharmacology and 100-level human biology students. For the results of the first study, Grant published a peer-reviewed manuscript, with reviews that noted that she is developing a unique angle on chatbot use in health care.
“Looking ahead, I plan to develop curricula that teach both students and, eventually, patients how to critically evaluate the quality of chatbot responses. Those skills will be of paramount importance in a world overflowing with digital advice,” Grant said.
When everyday chatbots dispense advice
Grant asked her pharmacology students to put three chatbots to the test: ChatGPT, Copilot and Gemini. Their task was simple: ask the chatbots about the safety profiles of specific medications and look for errors. They looked at correctness and rated how thorough and helpful the generated answers were.
“To my surprise, and theirs, students found that the chatbot responses were highly accurate,” she said. “And yes, the chatbots differed. Each had their own style, and students preferred when chatbots organized information in a helpful way.”
Grant recently published an article on her findings in the open-access journal Discover Artificial Intelligence, titled “Undergraduates perceive differences in helpfulness and thoroughness of responses of ChatGPT 3.0, Gemini 1.5, and Copilot responses about drug interactions.”
“I published this research because this is truly a new academic frontier. This work is also very timely, because we believe that many people already consult everyday chatbots for medical advice,” she said.
Access to better information on human health
The pharmacology project experience led Grant to wonder how this approach could help students in her human biology class. Because the goals in that course are different, Grant redesigned the activity.
Instead of focusing on medications, students used Copilot and ChatGPT to explore how molecules influence human health. Every other week, students asked the chatbots a question tied to the organ system the class was studying. Early results showed that the chatbots provided accurate information, and Grant is excited to work up the rest of the data.
“Chatbots are everywhere now, which means learners and patients have more information at their fingertips than ever before. My hope is that this work will help people get better information, not just more of it,” she said.
A third line of Grant’s research addresses biochemistry, using AI software to look at how proteins fold. These student projects are in mid-stage, with outcomes expected by June.
“I am very excited about these projects and will start looking for high-power collaborators with bespoke AI chatbots this spring,” she said.
