AMIA 2026 Annual Symposium Tutorial onDevelopment and Evaluation of Agentic, Multimodal Large Language Models in Healthcare Applications |
||
Location: Hilton Anatole, Dallas, TX, USA Time: November 7 - 11, 2026 |
||
Language models are increasingly used in natural language processing (NLP) applications, which require neither the development of a task-specific architecture nor customized training on large datasets. In particular, large language models (LLMs), such as DeepSeek, GPT, PaLM, and Llama, have demonstrated significant advances in NLP tasks. Recent developments in multimodal foundation models, capable of integrating text, images, and structured data, are transforming biomedical research and clinical practice. Simultaneously, agentic AI systems, which enable autonomous decision-making and task execution, are emerging as promising tools in healthcare workflows.
Despite their potential, concerns have also been raised about the impact of these tools in health care, education, research, and beyond. One is the potential for LLMs to reinforce disparities in healthcare, as these models are typically trained on data that is historically biased against certain disadvantaged groups. Another is the potential for LLMs to be applied for malicious purposes. Although it is widely accepted that LLMs should be used with integrity, transparency, and honesty, how to do so appropriately and, if needed, regulate the development and use of this technology warrants further discussion.
This tutorial provides students with an understanding of LLMs, using DeepSeek, GPT, Llama, and other models as examples, and their applications in health. Students will acquire knowledge of natural language processing, large language models, chain-of-thought, Retrieval-Augmented Generation (RAG), and the range of available prompting methods for processing clinical text. Hands-on experience and a toolkit will provide useful skills for managing text data to solve a variety of problems in the health domain.
An introduction to LLMs and their development in the medical domain
Evaluation of LLMs in medical applications: bias, ethical, and implementation issues
Multimodal LLMs and their applications
Agentic AI and Agents in Healthcare
Yifan Peng, Ph.D., FACMI, s an Associate Professor in the Department of Population Health Sciences at Weill Cornell Medicine. His main research interests include BioNLP and medical image analysis. He has published in major AI and healthcare informatics venues, including ACL, NAACL, CVPR, and MICCAI, as well as medical venues, including Nature Medicine, Nucleic Acids Research, npj Digital Medicine, and JAMIA. His research has been funded by federal agencies, including NIH and NSF, and industries such as Amazon and Google. He has given tutorials and keynote speeches at various conferences, including CVPR, Mayo Clinic AI Summit, and Pharmaceutical Data Science.
Hua Xu, Ph.D., FACMI, is the Robert T. McCluskey Professor and Vice Chair for Research and Development at the Department of Biomedical Informatics and Data Science of Yale School of Medicine. He also serves as Assistant Dean for Biomedical Informatics at Yale School of Medicine. He has worked on different clinical NLP topics and has built multiple clinical NLP systems. Dr. Xu served as the Chair of the AMIA NLP working group between 2014 and 2015 and currently leads the OHDSI NLP working group. He taught NLP tutorials at various conferences, including AMIA, Medinfo, and AIME. Recently, Dr. Xu has worked on building foundation medical LLMs, including the recently released Me LLaMA models based on the open LLaMA2 model. He will provide a generation introduction to LLMs and hands-on experience in developing medical LLMs and their applications in clinical NLP tasks such as information extraction.
Yanshan Wang, Ph.D., FAMIA, is vice chair of Research and assistant professor within the Department of Health Information Management at the University of Pittsburgh. His research interests focus on artificial intelligence (AI), natural language processing (NLP), and machine/deep learning methodologies and applications in health care. Dr. Wang has led several NIH-funded projects to develop NLP and AI algorithms that automatically extract information from free-text electronic health records (EHRs). Dr. Wang has been actively serving the informatics and NLP communities. He has served on the Student Paper Competition Committee for the AMIA Annual Symposium and has been an associate editor for the MedInfo conference. He is also a regular reviewer for a dozen prestigious journals, such as Nature Communications, JAMIA, and JBI. Wang also organized several shared tasks, including the first BioCreative/OHNLP challenge in 2018 and the second n2c2/OHNLP challenge in 2019, to encourage the informatics and NLP communities to tackle NLP problems in the clinical domain. He is also a member of the steering committee for the HealthNLP workshop. In 2020, he was inducted as a Fellow of AMIA (FAMIA). Dr. Wang serves as the Chair of the AMIA NLP working group from 2023 to 2024.
Yi Lin, Ph.D., is a Postdoctoral Associate in the Department of Population Health Sciences at Weill Cornell Medicine. His research interests encompass medical image analysis, computer vision, and multi-modal learning, with a focus on developing efficient deep learning algorithms for computer-aided diagnosis and image-guided intervention. He has published in major biomedical AI venues and journals, including Nature Biomedical Engineering, IEEE Transactions on Medical Imaging, and Medical Image Analysis. He has won three Medical AI competitions. He was awarded first place in the 2025 AMIA Doctoral Dissertation Award.
Please contact Yifan Peng if you have question. The webpage template is by the courtesy of awesome Georgia.
