AI Can Now Rebuild Images From Brain Scans. What If Your Dreams Don’t Stay Private?

3 min readHere’s what you’ll learn when you read this story:For years, scientists have trained AI to read fMRI scans to accurately reconstruct visual information.A new tool, known as Brain-IT, uses a Brain Interaction Transformer (BIT) to decode and encode fMRI scans, and it can reconstruct images with stunning clarity using a relatively small training set.This tool could help those with severe paralysis communicate, and give scientists new insight into the inner workings of the brain’s visual processing system.While debates rage on various podcasts, in hand-wringing online essays, and even in the halls of Congress as to whether AI has surpassed human intelligence—a threshold of no-return known as the AI Singularity—there’s one skill where AI algorithms have long trounced the human competition: pattern recognition.Because of modern AI engines’ impressive abilities to sift through large datasets and recognize patterns, they can accurately detect developing cancers and even create entirely new viruses not seen in nature. When trained on the brain, this AI pattern recognition can gain almost supernatural powers. This ability is vividly highlighted in a new paper submitted to the International Conference on Learning Representations by scientists from the Weizmann Institute of Science in Israel. The researchers revealed Brain-IT, an AI model capable of reconstructing mental images with a scary level of accuracy—using only brain-scan data.“There exist nowadays models that translate brain activity into images, and they can even produce impressive reconstructions that preserve the semantic meaning of the image reasonably well,” Weizmann Institute of Science’s Michal Irani, leader of the lab that developed the model, said in a press statement. “However, they tend to make mistakes in basic features such as composition and color. The new model we developed outperforms them in reconstructing both the content of the image and its details.”Of course, AI can’t learn the way humans do. It needs vast amounts of data to draw inferences from, and fMRI scans paired with images are in painfully short supply. So the research team relied on the Natural Scenes Dataset, which contains data from only eight individuals who viewed thousands of images while their brains were scanned. While that might sound like a tedious and claustrophobic experience for the individuals involved, even 73,000 total images isn’t enough data for robust AI training. So the researchers developed a decode/encode method where the model could both decode an image from these brain scans and also predict what brain scans might look like based on just an image alone.“We realized that by translating back and forth – from a random image that had never been viewed in an fMRI machine, to a predicted brain scan, and then back to the image we started with – the models would effectively build themselves a massive dataset,” Irani said in a press statement.Decoding fMRI brain scans is a perfect pattern recognition problem for AI to solve, which is why scientists have been toying with the idea for more than a decade.In 2017, researchers at Purdue University developed a model that could accurately predict the object being viewed, though the reconstructed images were little more than blurs. Fast-forward to 2022, and scientists at Osaka University used an AI model called Stable Diffusion, which created more accurate representations of the images but required captions to leverage text-to-image capabilities.Unlike those previous efforts, the new model uses a Brain Interaction Transformer (BIT), which maps the brain as 40,000 voxels—three-dimensional pixels—and logs how each one responds to an image. Those responses encode color, location, and whether a subject is seeing a face or a sandwich. With all of that in place, Brain-IT began to detect patterns across all eight brains in the Natural Scenes Dataset.Developing such a technology could be incredibly useful for people with severe paralysis, but Brain-IT could also help scientists probe aspects of the human mind itself. For example, the model found 128 brain regions associated with image processing, including some not previously identified by neurologists. And by altering the images, it’s possible to compare differences between predicted fMRI scans to uncover the brain’s underlying behavior. Eventually, one day, AI could even record our dreams.“What remains especially challenging is decoding video ... dozens of images change every second, while an fMRI scan takes about two seconds,” Irani said in a press statement. “If we overcome all these obstacles, it’s possible that in the future we may even be able to read dreams.”Darren lives in Portland, has a cat, and writes/edits about sci-fi and how our world works. You can find his previous stuff at Gizmodo and Paste if you look hard enough.