Eyal Klang, MD, Associate Professor of Medicine, and Director of the Generative AI Research Program within the Division of Data-Driven and Digital Medicine (D3M), at the Icahn School of Medicine at Mount Sinai.

Artificial intelligence (AI) can help radiologists analyze images or doctors make diagnoses with a high degree of accuracy even with traditional machine learning techniques, but they tend to require large amounts of training data to accomplish this.

Researchers at the Icahn School of Medicine at Mount Sinai are exploring using the latest technique in generative AI—specifically large language models (LLMs)—to see if it can achieve accurate predictions with less training data. Generative AI is rooted in the concept of generating new content typically by understanding data distribution.

Using a specially prepared, secure version of GPT-4—a product from OpenAI, the company that runs the popular generative AI platform ChatGPT—the team applied the model to predict admissions in the Emergency Department, based on objective data collected from patients and triage notes.

“One of the advantages of LLMs over traditional methods is that you can use just a few examples to train the model for any use case,” says Eyal Klang, MD, Associate Professor of Medicine, and Director of the Generative AI Research Program within the Division of Data-Driven and Digital Medicine (D3M), at Icahn Mount Sinai. “You don’t need to retrain models again and again for each use case, which is very hard when that can take millions of data points.”

“Another advantage of LLMs is its ability to explain to the user how it arrived at its answer,” says Dr. Klang. The model’s ability to explain its reasoning provides confidence for a physician to use it in assisting in making medical decisions.

Here’s an animated explainer on how Dr. Klang and his team tested GPT-4 against traditional machine learning methods for predicting whether patients who go to the ER need to be admitted.

The study used patient visit data from seven hospitals within the Mount Sinai Health System. More than 864,000 emergency room visits were included in the data cohort. The ensemble model comprising traditional machine learning techniques achieved an AUC score of 0.878 in predicting admissions, with an accuracy of 82.9 percent. (An AUC score measures the ability to make correct positive and negative guesses, with an 0.5 score meaning the model performed no better than a random guess.)

The GPT-4 model was given the same task of predicting ER admissions, but under a few different conditions: “off the shelf” (not given any examples of patients, also known as “zero-shot”); given some probabilities of how machine learning models would perform; given 10 examples of patients with triage notes (“few-shot”); given 10 contextually similar cases (retrieval-augmented generation, or RAG); and various combinations of these conditions. In the setting with the most information provided (few-shot with RAG and machine learning probabilities), GPT-4 had an AUC score of 0.874, and an accuracy of 83.1 percent—results statistically similar to the ensemble model.

The findings were published in the Journal of the American Medical Informatics Association on Tuesday, May 21.

In this Q&A, Dr. Klang discusses the team’s research.

What was the motivation for your study?

Our study was motivated by the need to test if generative AI, like the GPT-4 model, can improve prediction of admission—and thus clinical decision-making—in a high-volume setting like the Emergency Department. We compared it against older machine learning methods, as well as evaluated its performance in combination with older machine learning methods.

What are the implications?

It suggests that AI, specifically large language models, could soon be used to support doctors in emergency rooms by making quick, informed predictions about whether a patient should be admitted or not.

What are the limitations of the study?

The study relied on data from a single urban health system, which may not represent conditions in other medical settings. Additionally, our study also didn’t prospectively assess the impact of integrating this AI technology into the daily workflow of emergency departments, which could influence its practical effectiveness.

How might these findings be put to use?

These findings could be used to develop AI tools, such as those that integrate GPT-4, that support making accurate clinical decisions. This could promote a model of AI-assisted care that is data-driven and streamlined, using only very few examples to train the platform. It also sets the stage for further research into the integration of AI in health care, potentially leading to more sophisticated AI applications that are capable of reasoning and learning from limited data in real-time clinical settings.

What is your plan for following up on this study?

Our group is actively working on the practical application of LLMs in real-world settings. We are exploring the most effective ways to combine traditional machine learning with LLMs to address complex problems in these environments.

Learn more about how Mount Sinai researchers and clinicians are leveraging machine learning to improve patient lives

Top, from left: Rheoneil Lascano, MSN, FNP, Vishal Kapur, MD, David Song, MD, Prakash Krishnan, MD, FACC, FSCAI, Raman Sharma, MD, and Moinuddin Syed, MBBS. Bottom, from left: Dr. Song, Samin K. Sharma, MD, Annapoorna S. Kini, MD, Dr. Kapur, Dr. Krishnan, and Raman Sharma, MD.

A team at the Mount Sinai Fuster Heart Hospital has successfully performed the first procedure in the United States using an innovative drug-eluting, below-the-knee resorbable scaffold.

The team was led by Prakash Krishnan, MD, FACC, FSCAI, Assistant Professor, Medicine (Cardiology and Radiology), and Director of Endovascular Interventions at The Mount Sinai Hospital’s Cardiac Catheterization Laboratory. The Food and Drug Administration approved the device in April.

“FDA approval of this long-awaited new technology of a dissolving bioabsorbable scaffold is a remarkable addition in the treatment of lower limb arterial blockages where balloon and regular stents results are suboptimal, and these scaffolds are poised to restore blood flow for a long time,” says Samin K. Sharma, MD, Director of Interventional Cardiology for the Mount Sinai Health System. “We are proud of Dr. Krishnan, who played a major role in getting this innovative device approved.”

More than 20 million people in the United States are living with peripheral artery disease (PAD) yet there have been limited treatment options. The new procedure is the first to treat people with chronic limb-threatening ischemia (CLTI) below-the-knee, a severe stage of PAD. The goal is to prevent amputation, heart attack, and stroke among these patients.

For individuals battling chronic limb-threatening ischemia, blocked vessels restrict blood flow to the lower extremities. While balloon angioplasty serves as the current standard of care, recurrent blockages often necessitate further intervention.

Unlike traditional metal stents, the drug-eluting below-the-knee resorbable scaffold is a temporary implant. Crafted from naturally dissolving material, this scaffold gradually disappears after opening a clogged artery, presenting a significant advancement in sustaining open arteries below the knee.

“A resounding applause to the team at Mount Sinai Fuster Heart Hospital for their relentless pursuit of excellence in advancing patient care,” said Dr. Krishnan. “Many times doctors tell patients amputation is the only option, so this new technology is a game changer.”

The 12th annual symposium of the BioMedical Engineering and Imaging Institute (BMEII) at the Icahn School of Medicine at Mount Sinai focused on precision medicine and the future of digital health.

The event featured renowned academic and industry representatives from around the world who participated in panel discussions about the precision medicine imperative and the quest to extend the “health span”—defined as the number of years people live a heathy life free of disease—in a multidisciplinary manner with a focus on medical imaging and engineering.

The innovation station and poster session provided hands-on demonstrations of current research.

More than 325 people, including researchers, physicians, industry leaders, medical students, and high school students, attended the event at the New York Academy of Medicine. The event was held Wednesday and Thursday, March 20-21.

The symposium began with welcome remarks from BMEII’s Director Zahi A. Fayad, PhD, who emphasized the need to back up recent developments in screening, longevity, and pharmaceuticals with evidence and data. He shared this vision during the event, ensuring attendees walked away with a better and more holistic understanding of the current and future state of precision medicine.

A major highlight of the symposium was the panel discussion about the future of digital medicine. With five industry and academic leaders, this discussion addressed many of the challenges health care is facing as consumer health technologies grow rapidly and move into clinical areas. These challenges center on user compliance and data privacy.

“I think the biggest problem is also trying to get the stakeholders around the same table and developing some consensus as to how they can share, if not the proprietary information, but how can they kind of use all of that [data] towards the good of humankind,” said Jagmeet P. Singh, MD, ScM, DPhil, Professor of Medicine at Harvard Medical School. He works closely with patients within the cardiology department who have received implantable cardiac devices and has first-hand experience with the long-term challenges that come with working with multiple manufacturers.

“So, now you have third-party vendors—in fact, over the course of the last year, there are almost 120 different remote monitoring companies that have come up—that are trying to standardize the data from these implantable devices that can be used uniformly by all health care givers,” he added.

Brendan Carr, MD, MA, MS

The symposium also included sessions for high school, undergraduate, and graduate students to build their interest in scientific research. A professional development panel, hosted by PhD candidates at Icahn Mount Sinai, allowed leaders in the field to share their personal experiences, challenges, and successes with the students. The innovation station and poster session provided hands-on demonstrations of current research in the medical imaging space by BMEII members and scientists from other institutes.

Health care is bound to change in dramatic ways in the future, and the symposium addressed the research and innovation that is shaping these changes.

Brendan Carr, MD, MA, MS, Chief Executive Officer of the Mount Sinai Health System, summarized the importance of the research BMEII is doing: “The clinical delivery system doesn’t exist without the pioneering research portfolio, and the pioneering research portfolio exists because it’s so unbelievably compelling to save lives and improve people’s health outcomes. That synergy is special—it’s a big piece of what this conference is about.”

Frorm left: James Tisch, Co-Chairman of the Boards of Trustees of the Mount Sinai Health System, Dennis Charney, MD, Anne and Joel Ehrenkranz Dean of the Icahn School of Medicine at Mount Sinai, and Ash Tewari, MBBS, MCh.

The Milton and Carroll Petrie Department of Urology at the Icahn School of Medicine at Mount Sinai hosted its annual Prostate Cancer Research Gala and exceeded its goal of raising more than $1 million for the Center of Excellence for Prostate Cancer within the Department of Urology. Its mission is to eliminate prostate cancer through programs that support innovative research, patient care, prevention, and education for students, trainees, professionals, and the public.

“We are fortunate that at Mount Sinai the best minds across specialties like immunology, pathology, medical oncology, radiology, and others collaborate with the common goal of making prostate cancer insignificant while improving the quality of life for our patients,” said Ash Tewari, MBBS, MCh, Chair of the Department of Urology at the Mount Sinai Health System and the Kyung Hyun Kim, MD Professor of Urology at Icahn Mount Sinai, who spearheaded the gala and directs the Center of Excellence.

Goutam Chakraborty, PhD, left, and Dr. Charney

More than 200 people attended the event, which was held Wednesday, April 10, at the Rainbow Room in Rockefeller Center. Actor and singer Peter Gallagher served as the emcee for the evening.

At the event, the second annual Steven Southwick, MD Memorial Award was presented to Goutam Chakraborty, PhD, Assistant Professor, Urology, and Oncological Sciences and a member of The Tisch Cancer Institute at Mount Sinai. Named after renowned researcher and scientist Steven M. Southwick—a leading expert in psychological trauma and human resilience, who passed away in 2022 after a lengthy battle with prostate cancer—this award recognizes outstanding achievements in prostate cancer research.

Dr. Chakraborty was honored for his focus on lethal metastatic prostate cancer, which has no available treatments. He and his team made the groundbreaking discovery that the BRCA2 gene—most commonly known for its link to breast cancer—also has a connection to prostate cancer. Dr. Chakraborty is now looking at how mutations of genes like BRCA2 change cells from benign to metastatic disease, potentially leading to important new therapies.

“Research is like climbing a mountain, like climbing Everest, but we have not reached the top of Everest yet,” said Dr. Chakraborty in a video shown at the event. “Every day is a different challenge.”

Dr. Tewari, left, and John A. Levin

The gala also honored John A. Levin, who was recognized for his extraordinary commitment to the Prostate Cancer Program and longtime leadership on the Mount Sinai Board of Trustees. Dr. Tewari praised Mr. Levin’s unwavering support of the Mount Sinai Health System’s efforts to redefine cancer care, including his involvement with the International Prostate Cancer and Urology Symposium held at Mount Sinai, which draws the world’s top experts together to explore the latest advancements and challenges in urological cancer.

“His philanthropy has made an immense impact, not just on our prostate cancer program, but on a global understanding of prostate cancer, and to Mount Sinai itself, guiding and supporting research in different subspecialties,” said Dr. Tewari in a video.

In accepting the award, Mr. Levin noted that he’s proud to be part of Mount Sinai’s extraordinary service to New York City and beyond.

“We treat an enormous number of people. It is also an education arm and a research institution which creates new drugs, creates new devices,” he said in a video. “It’s really being part of a community and I want to help that community in whatever limited capacity I can.”