Evaluating advanced artificial intelligence in oncology education and clinical knowledge assessment

Abstract

Background: The rapid advancement of artificial intelligence (AI), particularly multimodal large language models (MLLMs), holds promise for revolutionizing oncology practices. This study evaluates the performance of two MLLMs, GPT-4o and Gemini Advanced, in answering oncology examination questions from the American Society of Clinical Oncology Self-Evaluation Program (ASCO-SEP) question bank.

Methods: A total of 832 multiple-choice questions covering various oncological tasks were extracted from the ASCO-SEP question bank. Both models were independently presented with these questions, and their responses were compared to the official answer key. Statistical analyses were performed to assess accuracy differences between the models.

Results: Gemini advanced outperformed GPT-4o, achieving 74.84% accuracy compared to 60% for GPT-4o (p=0.025). Gemini advanced consistently excelled across all task categories, particularly in making diagnoses, ordering and interpreting test results, and recommending treatment. Both models struggled with questions related to pathophysiology and basic science knowledge.

Conclusions: While both MLLMs demonstrate significant understanding of oncological knowledge, Gemini Advanced shows superior performance, highlighting the influence of model architecture and training data. These findings underscore the potential of AI in augmenting clinical practice and medical education but emphasize the need for further improvements, particularly in handling complex clinical scenarios and integrating foundational science knowledge.