1:45-2:05pm. Hidden flaws behind expert-level accuracy of GPT-4 vision in medicine (Zhiyong Lu)

2:05-2:25pm. Challenges of de-identifying and sharing multimodal data (Kevin B. Johnson)

2:25-2:45pm. Precision health in the age of multimodal generative AI (Hoifung Poon)

2:45-3:05pm. Health disparities in large visual-language models (Imon Banerjee)

3:05-3:15pm. Pannel discussion

Zhiyong Lu, Ph.D., is a tenured Senior Investigator in the NIH Intramural Research Program, leading research in biomedical text and image processing, information retrieval, and AI/machine learning. In his role as Deputy Director for Literature Search at the National Center of Biotechnology Information (NCBI), Dr. Lu oversees the overall R&D efforts to improve literature search and information access in resources like PubMed and LitCovid that are used by millions worldwide on a daily basis. Additionally, Dr. Lu holds an Adjunct Professor position with the Department of Computer Science at the University of Illinois Urbana-Champaign (UIUC). In this panel, Dr. Lu will discuss their latest research related to multi-modal foundation models such as GPT4 Vision and their applications in various medical applications such as automated disease diagnosis and medical report generation.

Kevin B. Johnson, MD, MS, is the David L. Cohen University Professor of Pediatrics, Biomedical Informatics, and Science Communication at the University of Pennsylvania. He is an internationally known developer and evaluator of clinical information technology. His main research is focused on the use of multimodal data and machine learning to summarize and quantify patient signs and symptoms in the EHR, assist with generating medical communications, and create decision support tools using real-time streaming data. In this panel, Dr. Johnson will discuss the challenges of deidentifying multimodal data and developing robust pipelines that promote the sharing of these data.

Hoifung Poon, Ph.D., is General Manager at Health Futures in Microsoft Research and an affiliated faculty at the University of Washington Medical School. He leads biomedical AI research and incubation, with the overarching goal of structuring medical data to optimize delivery and accelerate discovery for precision health. His team and collaborators are among the first to explore large language models (LLMs) in health applications, producing popular open-source foundation models such as PubMedBERT, BioGPT, BiomedCLIP, LLaVA-Med. His latest publication in Nature features GigaPath, the first whole-slide digital pathology foundation model pretrained on over 1 billion pathology image tiles. He has led successful research partnerships with large health providers and life science companies, creating AI systems in daily use for applications such as molecular tumor board and clinical trial matching. In this panel, Dr. Poon will discuss the exciting frontier of multimodal generative AI in Precision Health, where multimodal, longitudinal real-world patient data can be used to pretrain powerful multimodal patient embedding, enable patient-like-me reasoning at scale, and unlock population-level real-world evidence for advancing precision medicine.

Imon Banerjee, Ph.D., pertains to computer science, particularly artificial intelligence (AI) and data mining. Her studies show implicit bias in the AI model toward race across multiple imaging modalities. Dr. Banerjee's goal is to decrease AI-driven healthcare disparities. She reduces this tendency by using model unlearning and adversarial debiasing. These techniques decrease inaccurate, harmful, and outdated information learned by the AI models. She collaborates with institutions and centers that serve minority groups, such as Emory University in Atlanta and the Mountain Park Health Center in Arizona, to train and evaluate AI models with diverse datasets. In this panel, Dr. Banerjee will discuss the challenges in reducing bias and vulnerability in the pre-training of large visual-language models. She will also highlight the effect of pre-training bias in the downstream targeted tasks.

Yifan Peng (Moderator), Ph.D., FACMI, is an Associate Professor in the Department of Population Health Sciences at Weill Cornell Medicine. His main research interests include BioNLP and medical image analysis. He has published in major AI and healthcare informatics venues, including ACL, NAACL, CVPR, and MICCAI, as well as medical venues, including Nature Medicine, Nucleic Acids Research, npj Digital Medicine, and JAMIA. His research has been funded by federal agencies, including NIH and NSF and industries such as Amazon and Google. He received the AMIA New Investigator Award in 2023.