Introduction  

As the demands on the healthcare industry grow more complex and diverse, so too does the need for innovative approaches to training future physicians, nurses, and allied healthcare professionals who are faced with never-ending coursework material and research articles. With the advent of publicly available large language models (LLMs), the integration of artificial intelligence (AI) into medical education has emerged as a powerful tool, with the potential to augment – and ultimately transform – the way students and practitioners learn and hone their skills (Alam et al., 2023; Jeyaraman et al., 2023; Tsang, 2023). 

Among the myriad of AI-driven technologies, ChatGPT – Generative Pre-trained Transformer 3.5, an advanced language model developed by OpenAI (San Francisco, CA) – has garnered significant attention for its remarkable capacity to comprehend and generate human-like text. Released in November 2022, ChatGPT’s engagement skyrocketed, reaching over 100 million global users in just two months (Roose, 2023; Shivaprakash, 2023; Subbaraman, 2023). ChatGPT-4 is touted as having the capacity to understand and generate text with higher relevance and context sensitivity, as well as the ability to accept multiple input types, including video, voice and non-textual data.  

This LLM model has shown promise across numerous applications, including natural language understanding, content generation, and problem-solving (Shorey et al., 2024). In the context of health professions education, ChatGPT represents a groundbreaking advancement, offering educators and learners the methods and tools to revolutionize knowledge acquisition, clinical decision-making, and continuous professional development. Among other applications, the technology has already passed National Board of Medical Examiners (NBME) and United States Medical Licensing Examination (USMLE) Step exams, formulated disease scripts and contemplated differential diagnoses and assisted in scientific manuscript writing (Kung et al., 2023; Mohammad et al., 2023).  

The introduction of ChatGPT has also garnered criticism and stirred controversy in academia and in the broader medical community (Kanjee et al., 2023; Palmer, 2023). The current model was only trained on information until September 2021, leaving significant gaps in current evolutions in scientific discovery and thought, particularly in the post-COVID era. While the model correctly identifies commonly cited medical diagnoses and red flag symptoms, it struggles with more advanced analyses of human disease (Duong & Solomon, 2023; Eriksen et al., 2023; Kanjee et al., 2023). In addition, the diagnostic accuracy of ChatGPT compared to experienced physicians remains an ongoing area of research. Recent studies highlight that while ChatGPT can generate differential diagnoses well, its performance varies based on case complexity and may lack the nuanced clinical reasoning of seasoned physicians (Hirosawa et al., 2023). Mehnen et al. tested ChatGPT’s diagnostic performance across 50 clinical cases, including ten rare presentations (Mehnen et al., 2023). The model accurately identified common conditions within its top two differentials, whereas rare diseases required at least eight suggestions to achieve 90% diagnostic coverage.  Tan et. al. addressed challenges and risks of using ChatGPT in medicine, most prominently AI “hallucinations,” where the system generates seemingly credible but factually incorrect information, including fabricated references (Tan et al., 2024).  

 In clinical settings, these challenges are compounded by privacy concerns, as optimal performance would require access to sensitive patient data, raising questions about information security and confidentiality. The current absence of clear regulatory frameworks further complicates potential clinical implementation (Mehnen et al., 2023; Tan et al., 2024). Addressing these challenges will be essential to responsibly harness AI’s potential while mitigating risks to patient care, research integrity, and medical education.  

 Whether ChatGPT can be harnessed to enhance the learning experience of interdisciplinary health education students in collaboration with a traditional educational environment is unknown. Therefore, this study tested the hypothesis that a prospectively designed and innovative educational session on ChatGPT would bolster the immediate and sustained ability of health professions students to understand and utilize ChatGPT and to determine their perception of ChatGPT’s utility in their future medical careers.  

Methods

Participants 

Students at the Emory School of Medicine, including those from genetic counseling, first and second-year medical students, and first and second-year physician assistant students, provided the cohort for the study. Fifty-nine self-selected students with varying exposure to ChatGPT participated in the workshop, which had the learning objectives listed below.  

 By the end of this workshop, learners will be able to: 

1. Understand the framework of an LLM like ChatGPT and be able to effectively explain it to a peer. 
2. Develop comfort in using ChatGPT within medical contexts, as demonstrated by generating relevant prompts and getting appropriate responses based on simulated scenarios. 
3. Compile a list of at least five potential use cases and associated constraints for implementing ChatGPT in upcoming medical scenarios. 

Workshop Design – The study was designed around a single-session workshop lasting one hour. The session was divided into a 30-minute learning session and a 30-minute active hands-on exercise utilizing ChatGPT (Appendix A). The learning session was designed to provide students with an understanding of ChatGPT, including its functionality, potential applications, and limitations. The learning session began with a discussion of participant familiarly, thoughts, and rumors about ChatGPT. We then transitioned to a 15-minute presentation about how ChatGPT is made and how it works using an analogy of opening a bakery (more details to be found in Appendix A). The second half of the workshop, the active half, concluded with an active demo that allowed students to interact with ChatGPT and discuss its potential uses in a medical setting. In this segment, students worked in pairs to go through a case-based learning prompt from a small group session using ChatGPT. They also prompted ChatGPT to identify relevant medications and clinical guidelines, and to write board-style questions. We concluded with a 5-minute discussion on ChatGPT “hallucinations,” meaning instances where the model generates responses that are factually incorrect, fabricated, or not grounded in the input data or established knowledge, despite appearing coherent and plausible. We further discussed limitations with the tool and that it should be used as a learning aide rather than a thought and problem-solving replacement. 

Surveys – We aimed to evaluate participants in four categories with respect to ChatGPT usage: frequency, use cases, comfort, and perceived future application. We collected attitudinal data at three time points: immediately before the workshop, immediately after the workshop, and then eight weeks post-workshop. Using a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree), we asked the same attitudinal questions in each survey about the participants’ use, and perceived future use, of ChatGPT, self-perceived understanding of how ChatGPT works at a basic level, and comfort using ChatGPT (Appendix B). Additionally, participants were allowed to provide narrative comments about their ChatGPT usage and other feedback and comments regarding the workshop. 

Data Analysis - To provide an overview of the participant characteristics, descriptive statistics were calculated for limited demographic variables such as age and program of study. Paired t-tests were utilized to compare the responses to the same eight-question surveys administered before and immediately after the workshop. This statistical test was chosen due to the dependent nature of the samples (i.e., responses from the same participants at different time points) and the normal distribution of the data. Paired t-tests were also used to compare the mean responses from the pre-workshop survey to the survey administered six weeks post-workshop, assessing the longevity of the workshop’s impact. We considered p values <0.05 to be statistically significant. 

Results and Discussion

The final study cohort consisted of 51 students who completed all three surveys. Eight students were lost to follow-up and not included in the study population. The persons lost to follow up were three PA students and five MD students. The final study cohort’s background and exposure to ChatGPT are provided in Table 1. Study participants had a mean age of 25.4 years (SD = 5.1), with representation from MD students (80%), PA students (18%), and genetic counseling students (2%).