A cutting-edge new artificial intelligence (AI) tool can reconstruct images from a person’s vision simply by analyzing brain scans.
The technology recreates a person’s visual field with startling accuracy and can also predict brain activity by analyzing what a person is looking at in real time.
Professor Michal Irani of the Weizmann Institute of Science developed the “mind-reading” AI model Brain-IT, which interprets functional magnetic resonance imaging (fMRI) scans of participants’ brain activity as they view a series of images.
Irani and her team—which includes Roman Beliy, Amit Zalcher, Jonathan Kogman, and Navve Wasserman from Weizmann’s Computer Science and Applied Mathematics Department—released their research on Sept. 14.
While the current study used fixed images, Irani’s lab plans to test these so-called mind-reading methods on participants’ hearing and, eventually, on video.
“What remains especially challenging is decoding video—for example, during dreaming,” Irani said in a statement.
“Dozens of images change every second, while an fMRI scan takes about two seconds. If we overcome all these obstacles, it’s possible that in the future we may even be able to read dreams.”
The study reviewed fMRI scans of eight participants who had viewed thousands of images across 30 to 40 scanning sessions. Participants viewed around 40 images over roughly 10-minute sessions as researchers scanned their brains six times. This created roughly 73,000 image-fMRI pairs for the AI model to analyze.
Irani said other models can translate brain activity into images with “impressive reconstructions,” but that they often make basic composition and color errors.
“The new model we developed outperforms them in reconstructing both the content of the image and its details,” she said.
“What’s more, while every other model requires dozens of hours of brain scans to learn to ‘read’ a new person, our model needs only one hour,” she said.
Scientists have been working on similar technology for years. A paper published in the Journal of Law and the Biosciences in 2017 discussed the possibility of reaching “brain-based mind reading” with neuroimaging techniques.
The paper described how these techniques could be used for lie detection to assess defendants, prisoners, and prospective jurors in court cases to improve the legal system. However, it also notes that brain-based mind reading could have potentially unethical coercive uses.
Improvements in the signal data produced by fMRI machines and in how humans analyze that data in recent years have made this technology possible, according to Kenneth Norman, a psychology and neuroscience professor at Princeton University.
In a discussion with the American Psychological Association in 2022, Norman said the technology could help people with depression or anxiety.
Therapists could use neurofeedback to make participants aware that their thinking has drifted toward negative thought loops and then give them direction to try to control those mental patterns, he said.
Some neuroscientists have heralded the capabilities of Brain-IT’s AI model, highlighting how it could therapeutically help people suffering from neurological conditions. But others have raised concerns about how the technology could be abused to coercively extract information from a person’s mind.