Eyal Klang, MD, Associate Professor of Medicine, and Director of the Generative AI Research Program within the Division of Data-Driven and Digital Medicine (D3M), at the Icahn School of Medicine at Mount Sinai.

Artificial intelligence (AI) can help radiologists analyze images or doctors make diagnoses with a high degree of accuracy even with traditional machine learning techniques, but they tend to require large amounts of training data to accomplish this.

Researchers at the Icahn School of Medicine at Mount Sinai are exploring using the latest technique in generative AI—specifically large language models (LLMs)—to see if it can achieve accurate predictions with less training data. Generative AI is rooted in the concept of generating new content typically by understanding data distribution.

Using a specially prepared, secure version of GPT-4—a product from OpenAI, the company that runs the popular generative AI platform ChatGPT—the team applied the model to predict admissions in the Emergency Department, based on objective data collected from patients and triage notes.

“One of the advantages of LLMs over traditional methods is that you can use just a few examples to train the model for any use case,” says Eyal Klang, MD, Associate Professor of Medicine, and Director of the Generative AI Research Program within the Division of Data-Driven and Digital Medicine (D3M), at Icahn Mount Sinai. “You don’t need to retrain models again and again for each use case, which is very hard when that can take millions of data points.”

“Another advantage of LLMs is its ability to explain to the user how it arrived at its answer,” says Dr. Klang. The model’s ability to explain its reasoning provides confidence for a physician to use it in assisting in making medical decisions.

Here’s an animated explainer on how Dr. Klang and his team tested GPT-4 against traditional machine learning methods for predicting whether patients who go to the ER need to be admitted.

The study used patient visit data from seven hospitals within the Mount Sinai Health System. More than 864,000 emergency room visits were included in the data cohort. The ensemble model comprising traditional machine learning techniques achieved an AUC score of 0.878 in predicting admissions, with an accuracy of 82.9 percent. (An AUC score measures the ability to make correct positive and negative guesses, with an 0.5 score meaning the model performed no better than a random guess.)

The GPT-4 model was given the same task of predicting ER admissions, but under a few different conditions: “off the shelf” (not given any examples of patients, also known as “zero-shot”); given some probabilities of how machine learning models would perform; given 10 examples of patients with triage notes (“few-shot”); given 10 contextually similar cases (retrieval-augmented generation, or RAG); and various combinations of these conditions. In the setting with the most information provided (few-shot with RAG and machine learning probabilities), GPT-4 had an AUC score of 0.874, and an accuracy of 83.1 percent—results statistically similar to the ensemble model.

The findings were published in the Journal of the American Medical Informatics Association on Tuesday, May 21.

In this Q&A, Dr. Klang discusses the team’s research.

What was the motivation for your study?

Our study was motivated by the need to test if generative AI, like the GPT-4 model, can improve prediction of admission—and thus clinical decision-making—in a high-volume setting like the Emergency Department. We compared it against older machine learning methods, as well as evaluated its performance in combination with older machine learning methods.

What are the implications?

It suggests that AI, specifically large language models, could soon be used to support doctors in emergency rooms by making quick, informed predictions about whether a patient should be admitted or not.

What are the limitations of the study?

The study relied on data from a single urban health system, which may not represent conditions in other medical settings. Additionally, our study also didn’t prospectively assess the impact of integrating this AI technology into the daily workflow of emergency departments, which could influence its practical effectiveness.

How might these findings be put to use?

These findings could be used to develop AI tools, such as those that integrate GPT-4, that support making accurate clinical decisions. This could promote a model of AI-assisted care that is data-driven and streamlined, using only very few examples to train the platform. It also sets the stage for further research into the integration of AI in health care, potentially leading to more sophisticated AI applications that are capable of reasoning and learning from limited data in real-time clinical settings.

What is your plan for following up on this study?

Our group is actively working on the practical application of LLMs in real-world settings. We are exploring the most effective ways to combine traditional machine learning with LLMs to address complex problems in these environments.

Learn more about how Mount Sinai researchers and clinicians are leveraging machine learning to improve patient lives

Kuan-lin Huang, PhD, Assistant Professor of Genetics and Genomic Sciences at the Icahn School of Medicine at Mount Sinai

Dementia is a neurodegenerative disorder, commonly known to affect cognitive function—including memory and reasoning. It is also a factor contributing to death. According to the Centers for Disease Control and Prevention, dementia is currently the seventh leading cause of death in the United States. Alzheimer’s disease is the most common form of dementia, accounting for approximately 70 percent of cases.

Researchers have used artificial intelligence and machine learning to help diagnose and classify dementia. But less effort has been put into understanding mortality among patients with dementia.

A group of researchers at the Icahn School of Medicine at Mount Sinai seeks to tackle this problem by developing a machine learning model to predict risks of death for a patient within 1-, 3-, 5-, and 10-year thresholds of a dementia diagnosis.

“We really want to call attention to how Alzheimer’s disease is actually a major cause of death,” says Kuan-lin Huang, PhD, Assistant Professor of Genetics and Genomic Sciences and Principal Investigator of the Precision Omics Lab at Icahn Mount Sinai.

“When people think of dementia, they think of patients losing their memory, as opposed to when people think about cardiovascular disease or cancer, they think about mortality,” says Dr. Huang. “As someone who has a family member who unfortunately passed away from Alzheimer’s disease, I’ve seen how the late stage of the disease—because you lose certain bodily functions—can become quite lethal.” In late-stage dementia, the disease destroys neurons and other brain cells, which could inhibit swallowing, breathing, or heart rate regulation, or cause deadly associated complications such as urinary tract infections or falls.

In the study, the team focused on this question: Given a person’s age, specific type of dementia, and other factors, what will be the risk the person will end up passing within a certain number of years?

For its model, the team used XGBoost, a machine learning algorithm that utilizes “gradient boosting.” This algorithm is based on the use of many decision trees—“if-this, then-that”-type reasoning. It learns from errors made by previous simple trees and collectively can make strong predictions.

Here’s how the study’s lead authors, Jimmy Zhang and Luo Song in Dr. Huang’s research team, leveraged machine learning to shed light on mortality in dementia.

The study used data from more than 40,000 unique patients from the National Alzheimer’s Coordinating Center, a database spanning about 40 Alzheimer’s disease centers across the United States. The model achieved an area under the receiver operating characteristic curve (AUC-ROC) score of more than 0.82 across the 1-, 3-, 5-, and 10-year thresholds. Compared to an AUC-ROC of 0.5, which amounts to a random guess that correctly predicts 50 percent of the time, the model performed reasonably well in predicting a dementia patient’s mortality, but still has room for improvement. By conducting stratified analyses within each dementia type, the researchers also identified distinct predictors of mortality across eight dementia types.

Findings were published in Communications Medicine on February 28.

In this Q&A, Dr. Huang discusses the team’s research.

What was the motivation for your study?

We wanted to address the challenges in dementia care: namely, to identify patients with dementia at high risk of near-term mortality, and to understand the factors contributing to mortality risk across different types of dementia.

What are the implications?

Clinically, it supports the early identification of high-risk patients, enabling targeted care strategies and personalized care. On a research level, it underscores the value of machine learning in understanding complex diseases like dementia and paves the way for future studies to explore predictive modeling in other aspects of dementia care.

What are the limitations of the study?

While our study includes nationwide data, to make the model more generalizable, it still needs to be adapted to different research and clinical settings.

How might these findings be put to use?

These findings could enhance the care of dementia patients by identifying those at high risk of mortality for more personalized management strategies. On a broader scale, the study’s methodologies and insights could influence future research in predictive modeling for dementia, potentially leading to improved patient outcomes and more efficient health care systems.

What is your plan for following up on this study?

We plan to refine our dementia models by including treatment effects and genetic data, and exploring advanced deep learning techniques for more accurate predictions.

Learn more about how Mount Sinai researchers and clinicians are leveraging machine learning to improve patient lives

Avi Ma’ayan, PhD, Director of the Mount Sinai Center for Bioinformatics at the Icahn School of Medicine at Mount Sinai

Birth defects can be linked to many factors—genetic, environmental, even pure chance. Characterizing the links of any factor to congenital abnormalities is a daunting task, given the vastness of the problem.

In the face of this challenge, a team of researchers at the Icahn School of Medicine at Mount Sinai tapped artificial intelligence (AI) methods to shed light on associations between existing medications and their potential to induce specific birth abnormalities.

“We wanted to improve our understanding of reproductive health and fetal development, and importantly, warn about the potential of new drugs to cause birth defects before these drugs are widely marketed and distributed,” says Avi Ma’ayan, PhD, Professor of Pharmacological Sciences and Director of the Mount Sinai Center for Bioinformatics at Icahn Mount Sinai.

The team developed a knowledge graph—a descriptive model that maps out the relationships between entities and concepts—called ReproTox-KG to integrate data about small-molecule drugs, birth defects, and genes. In addition to constructing the knowledge graph, the team also used machine learning, specifically semi-supervised learning, to illuminate unexplored links between some drugs and birth defects.

Here’s how ReproTox-KG works as a knowledge graph to predict birth defects.

The study examined more than 30,000 preclinical small-molecule drugs for their potential to cross the placenta and induce birth defects, and identified more than 500 “cliques”—interlinked clusters between birth defects, genes, and drugs—that can be used to explain molecular mechanisms for drug-induced birth defects. Findings were published in Communications Medicine on July 17, and the platform has been made available on a web-based user interface.

In this Q&A, Dr. Ma’ayan, senior author of the paper, discusses ReproTox-KG and its potential impacts.

What was the motivation for your study?

The motivation for the study was to find a use case that combines several datasets produced by National Institutes of Health (NIH) Common Fund programs to demonstrate how integrating data from these resources can lead to synergistic discoveries, particularly in the context of reproductive health.

The study identifies some relationships between approved drugs and birth defects to identify existing drugs that are currently not classified as harmful but which may pose risks to the development of a fetus. It also provides a new global framework to assess potential toxicity for new drugs and explain the biological mechanisms by which some drugs known to cause birth defects may operate.

What are the implications?

Identifying the causes of birth defects is complicated and difficult. But we hope that through complex data analysis integrating evidence from multiple sources, we can improve our understanding of reproductive health and fetal development, and also warn about the potential of new drugs to cause birth defects before these drugs are widely marketed and distributed.

What are the limitations of the study?

We have not yet experimentally validated any of the predictions. There are currently no considerations of tissue and cell type, and the knowledge graph representation omits some detail from the original datasets for the sake of standardization. The website that supports the study may not be appealing to a large audience.

How might these findings be put to use?

Regulatory agencies such as the U.S. Environmental Protection Agency or the Food and Drug Administration may use the approach to evaluate the risk of new drug or other chemical applications. Manufacturers of drugs, cosmetics, supplements, and foods may consider the approach to evaluate the compounds they include in products.

What is your plan for following up on this study?

We plan to use a similar graph-based approach for other projects focusing on the relationship between genes, drugs, and diseases. We also aim to use the processed dataset as training materials for courses and workshops on bioinformatics analysis. Additionally, we plan to extend the study to consider more complex data, such as gene expression from specific tissues and cell types collected at multiple stages of development.

Learn more about how Mount Sinai researchers and clinicians are leveraging machine learning to improve patient lives