A cross-sectional study to assess and compare artificial intelligence-generated content for medical professionals with UpToDate on management of acute coronary syndrome
DOI:
https://doi.org/10.18203/2320-6012.ijrms20262803Keywords:
Artificial intelligence, DeepSeekAI, UpToDate, Readability, Acute coronary syndrome, Medical education, Flesch reading ease, Flesch-Kincaid grade level, SMOG indexAbstract
Background: The rapid rise of artificial intelligence (AI) in medicine offers new ways to generate educational content, but its readability and accessibility for both clinicians and patients remain underexplored. This study compares AI-generated medical information to a standard clinical reference.
Methods: This cross-sectional study analyzed educational content on acute coronary syndrome (ACS) management generated by DeepSeekAI and retrieved from UpToDate. Readability was assessed using four established metrics: Flesch reading ease (FRE), Flesch-Kincaid grade level (FKGL), SMOG index, and difficult word percentage. Data were collected across six distinct topics related to ACS management, and the scores for each tool were statistically compared.
Results: Readability analysis revealed notable disparities between the two sources. UpToDate consistently produced content with significantly higher FRE scores (e.g., UpToDate FRE up to 40.2 vs. DeepSeek AI FRE down to 2.7 for topic 1) and considerably lower FKGLs, SMOG Indices, and percentages of difficult words across all examined topics. While both sources generated text with relatively high reading levels indicative of medical content, DeepSeekAI consistently demonstrated lower readability, suggesting more complex vocabulary and sentence structures.
Conclusions: UpToDate content on ACS management is more readable and accessible for clinicians than DeepSeekAI-generated content, highlighting the need to improve readability in AI-generated medical materials for effective clinical use.
References
Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on medical challenge problems. arXiv. 2023;arXiv:2303.13375.
Rao A, Pang M, Kim J, Schnickel R, Merrill JA, Ensley CL, et al. Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study. JMIR Med Inform. 2023;11:e48659.
UpToDate. About Us. Available at: https://www.uptodate.com/contents/search. Accessed on 13 April 2026.
DeepSeek AI. DeepSeek homepage. Available at: https://chat.deepseek.com/. Accessed on 13 April 2026.
DeepSeek AI. DeepSeek Medical GitHub repository. Available at: https://github.com/deepseek-ai. Accessed on April 13, 2025.
Salvagno M, Taccone FS, Gerli AG. Can artificial intelligence help for scientific writing? Crit Care. 2023;27(1):75.
WebFX. Readable: Free readability test tool. Available at: https://www.webfx.com/tools/read-able/. Accessed on 13 April 2026.
Fijačko N, Gosak L, Štiglic G, Picard CT, Douma MJ. Can ChatGPT pass the life support exams without entering the American Heart Association course? Resuscitation. 2023;185:109732.
Kung TH, Cheatham M, Medenilla A, Chavez C, Hicks L, Foster N, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198.
Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med. 2023;183(6):589-96.
Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-80.
Hancı V, Ergün B, Gül Ş, Uzun Ö, Erdemir İ, Hancı F. Assessment of readability, reliability, and quality of ChatGPT®, BARD®, Gemini®, Copilot®, and Perplexity® responses on palliative care. Medicine (Baltimore). 2024;103(33):e39305.
Lee M, Cadiente A, Chen J, Zhou Y, Greenwald B. An evaluation of the reliability and readability of large language models in the dissemination of traumatic brain injury information. Digit Health. 2025;11:20552076251350760.
Patil R, Heston TF, Bhuse V. Prompt engineering in healthcare. Electronics. 2024;13(15):2961.
MacDonald S, Steven K, Trzaskowski M. Interpretable AI in healthcare: enhancing fairness, safety, and trust. In: Raz M, Nguyen TC, Loh E, editors. Artificial Intelligence in Medicine. Singapore: Springer. 2022;85-110.
Downloads
Published
How to Cite
Issue
Section
