However, most AI models for ECG analysis use a particular deep learning method called convolutional neural networks (CNNs). CNNs require large training datasets to make diagnoses, which spell limitations when it comes to rare heart diseases that do not have a wealth of data.

Researchers at the Icahn School of Medicine at Mount Sinai have developed an AI model, called HeartBEiT, for ECG analysis, which works by interpreting ECGs as language.

The model uses a transformer-based neural network, a class of network that is unlike conventional networks but does serve as a basis for popular generative language models, such as ChatGPT.

Here’s how HeartBEiT works as an artificial intelligence deep-learning model, and how it compares to CNNs.

In this Q&A, Dr. Vaid discusses the impact of this new AI model on reading ECGs.

What was the motivation for your study?

Deep learning as applied to ECGs has had much success, but most deep learning studies for ECGs use convolutional neural networks, which have limitations.

Recently, the transformer class of models has assumed a position of importance. These models function by establishing relationships between parts of the data they see. Generative transformer models such as the popular ChatGPT utilize this understanding to generate plain-language text.

By using another generative image model, HeartBEiT creates representations of the ECG that may be considered “words,” and the whole ECG may be considered a single “document.” HeartBEiT understands the relationship between these words within the context of the document, and uses this understanding to perform diagnostic tasks better.

Deployment of this model and its derivatives into clinical practice can greatly enhance the manner in which clinicians interact with ECGs. We are no longer limited to models for commonly seen conditions, since the paradigm can be extended to nearly any pathology.

What is your plan for following up on this study?

We intend to scale up the model so that it can capture even more detail. We also intend to validate this approach externally, in places outside Mount Sinai.

Learn more about how Mount Sinai researchers and clinicians are leveraging machine learning to improve patient lives

Girish Nadkarni, MD, MPH, and Faris Gulamali

Artificial intelligence (AI) and machine learning (ML) have seen increasing use in health care, from guiding clinicians in diagnosis to helping them decide the best course of treatment. However, AI still has much unrealized potential in various health care settings.

Mount Sinai researchers are exploring bringing AI into intensive care, and developed Spatial Resolved Temporal Networks (SpaRTeN), a model to assess high-frequency patient data and generate representations of their state in real time.

The work was presented at the Time Series Representation Learning for Health workshop on Friday, May 5, hosted by the International Conference for Learned Representations, a premier gathering dedicated to machine learning.

Hear from Girish Nadkarni, MD, MPH, Irene and Dr. Arthur Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai and the leader of the SpaRTeN research, and Faris Gulamali, medical student at Icahn Mount Sinai and member of the Augmented Intelligence in Medicine and Science lab, on what lay behind creating the model and what it could achieve for patients.

What was the motivation for your study?

A growing amount of research is indicating the need to redefine critical illness by biological state rather than a non-specific illness syndrome. Advances in genomics, data science, and machine learning have generated evidence of different underlying etiologies for common ICU syndromes. As a result, patients with the exact same diagnosis can have entirely different outcomes.

What are the implications?

In the ICU, representations of a patient can be used to guide personalized treatments based on personalized diagnoses rather than generic treatments with empirical diagnoses.

What are the limitations of the study?

In this study, we only looked at using one type of data at a time in real time. For example, we looked primarily at measures of intracranial pressure. However, the ICU has many types of data being output simultaneously. Future work hopes to integrate all the different types of data such as electrocardiograms, blood pressure, and imaging to improve patient representations.

How might these findings be put to use?

These patient representations are being combined with data on medications and procedures to determine how to optimize patient treatment based on underlying state rather than common illness syndromes.

What is your plan for following up on this study?

In this study, we focused primarily on creating the algorithm and showing that it works for the case of intracranial hypertension. In future studies, we would like to integrate multiple data modalities such as imaging, electrocardiograms, and blood pressure as well as intervention-based data such as medications and procedures to determine precise empirical interventions that lead to improvements in short-term and long-term patient outcomes.

Learn more about how Mount Sinai researchers and clinicians are leveraging machine learning to improve patient lives

When Kanaka Rajan, PhD, an expert in neural networks, joined the Icahn School of Medicine at Mount Sinai in late 2018, it was the school’s way of investing in computational neuroscience. But since establishing her lab, she has achieved new heights not just in her area of study, but in paving roads for future diverse talents to enter what had been a rather homogenous field.

Dr. Rajan, an Assistant Professor of Neuroscience in The Friedman Brain Institute at Icahn Mount Sinai, was recently awarded the McKnight Scholar Award, a three-year honor that provides funding to early-career scientists, from the McKnight Foundation, a Minnesota-based organization that has supported work in arts and culture, neuroscience, and climate change.

“I am honored to be recognized by the McKnight Foundation this year. The announcement was such a pleasant surprise,” said Dr. Rajan. Awardees of such programs are not often pure theorists like herself, she said. But the less restricted nature of the funding will advance a new research direction her lab has taken on and will bring much needed exposure to a key problem in science: how does the brain work?

Growing the team

The Rajan lab builds recurrent neural networks—artificial networks of neural nodes or regions inspired by biological brains—toward two core goals. The first is to discover the pattern of cell activity and connectivity in the brain, especially in psychiatric disease models, using these networks. These include exploring how there might be unexpected similarities or differences across species.

One study in that vein was based on what Dr. Rajan calls “functional motifs”—brainwide neural maps that tracked motor dysfunction as a correlated passive coping mechanism, a trait associated with depression.

An illustrated look at Dr. Rajan’s work

Illustration credit: Jorge Cham

Using this method for recurrent neural networks was born partly out of recognition for how animals and children learn, and in part to address limitations of current training algorithms, Dr. Rajan said. She adds that her lab is among a handful to use curriculum learning in neuroscience, recognizing that understanding how people generalize requires understanding their full learning trajectory.

“It’s an exciting new chapter for this field and I’m hopeful the McKnight Scholar Award will help scale our efforts on this front,” Dr. Rajan said. Her team—comprising four postdoctoral researchers, some of whom are starting independent faculty positions later this year, and three graduate students—looks to add a few more members with the funding.

“This is a competitive field and city to hire scientists in,” she said. “Not only are we competing with other institutions; we’re also competing with industry, so it’s on us to make it an attractive proposition.”

But Dr. Rajan believes Mount Sinai offers something that other institutions or industry players might not: complete intellectual freedom.

“When I first arrived, I was told, ‘Welcome to the department. Let us know if you need anything,’ without any restrictions on my next steps,” Dr. Rajan recalled of her interactions with leadership in the Department of Neuroscience at Icahn Mount Sinai. “This was unlike previous institutions I had been at, where I had been gently nudged where I could or could not direct my research.”

Mapping new paths ahead

Just as Dr. Rajan felt she was given the opportunity to excel as a woman and person of color, she felt compelled to extend those opportunities to those who follow in her footsteps.

Dr. Rajan was allowed to tap her seed funding to start a pilot project in which she turned complex research papers into comic strips to get high school seniors and college students, especially those from disadvantaged communities, interested in joining the neuroscience field.

A peek at how Dr. Rajan makes complex research topics accessible to young students

Illustration credit: Jordan Collver

“There had been artificially high barriers to entry, like girls had been told they’re not good at math, or that AI and/or computational neuroscience are beyond their understanding,” Dr. Rajan said.

By turning complex ideas into jargon-free and engaging formats such as a comic strip, she hopes to help young students realize that they too can enter and flourish in such a technical field. A series of comic strips have been created and steps are underway to distribute them to schools in New York City and other cities.

“When I first started my lab, I had 116 applications to join my team. Guess how many were women?” Dr. Rajan asked. “Two. Computational neuroscience has a representation problem, and I want to fix what I can.” She continued, “I’ve taken small steps, but the ball rolled from Mount Sinai. Here, you see women really get to thrive.”