Reading ‘what you see’: Reconstruction of visual images using brain activity patterns | ATR - DNi

Summary

Imagine being able to replay your dreams and fantasies on a TV screen. This technique is called “visual image reconstruction”. Visual image reconstruction allows us to reconstruct images that a person sees, displaying complex perceptual contents as they are represented in the brain.

Our laboratory has developed “neural decoding” techniques that we can use to predict the presented visual images that a person sees from the brain activity patterns. Neural decoding is performed using pattern recognition programs, also known as “decoders,” that learn statistical relationships between several pre-specified visual images and the corresponding brain activity patterns. Although this approach can predict which stimulus is presented among a set of pre-specified categories (e.g., horizontal or vertical grating), the prediction is just categorical and is not an ‘image’ itself.

In this study, we developed a method to reconstruct visual images by combining local image decoders that predict local image contrast at multiple spatial scales, using brain activity patterns measured by fMRI scanners. Using this method, we can reconstruct arbitrary visual images, including geometric figures and alphabet letters, which are not used to train the decoders. Reconstruction can also be used to identify the presented image among hundreds of millions of candidate images. Furthermore, we succeeded in replaying a sequence of presented visual images as a movie by reconstructing each visual image from 2-s individual fMRI scanning data.

Although we currently only reconstruct the visual images that a person actually sees, it is possible that the same principle can be used to visualize subjective perception such as mental images and dreams. Our approach may serve as an innovative tool to reveal neural mechanisms of the mind as well as practical applications in the field of medical engineering.

Get the latest Flash Player to see this player.

[Javascript required to view Flash movie, please turn it on and refresh this page]
A movie was created by concatenating images reconstructed from 2-s single-volume fMRI data of four runs presenting geometric shapes. Since fMRI data reflects changes in blood flow, it slightly delays from neural activity in the brain. The delay of the reconstruction from the image presentation corresponds to this hemodynamic response delay.

Data

The fMRI data used in this research can be downloaded freely. The data download site is here.

Tool

The sparse logistic regression (SLR) developed by O. Yamashita is used as a decoder in this research. SLR toolbox site is here.

Reference

Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C. Tanabe, Norihiro Sadato, Yukiyasu Kamitani (2009). “Visual image reconstruction from human brain activity using a combination of multi-scale local image decoders” Neuron 60, 915-929.

Collaborators

Department of Neuroinformatics CNS ATR
Yoichi Miyawaki, Hajime Uchida, Yukiyasu Kamitani

Department of Computational Brain Imaging CNS ATR
Okito Yamashita, Masa-aki Sato

National Institute of Information and Communications Technology
Yoichi Miyawaki

Nara Institute of Science and Technology
Hajime Uchida, Yukiyasu Kamitani

National Institute for Physiological Sciences
Yusuke Morito, Hiroki Tanabe, Norihiro Sadato

Contact

ATR
2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288
Phone: 0774-95-1172, FAX: 0774-95-1172

http://www.atr.jp/index_j.html