Image Synthesis with Transformer

Projects Title

Image Synthesis with Transformer: Brain MR Image to PET Image


University of Wisconsin-Madison, USA


2022.9 - present


Vikas Singh, PH. D, Professor in the Department of Biostatistics

Yong Jae Lee, PH. D, Associate Professor in the Department of Computer Sciences


To reduce the financial burden on the patient and to speed up the diagnostic process, we propose a novel unsupervised GAN framework to achieve the goal of using magnetic resonance imaging (MRI), an inexpensive and easy-to-use technique, to generate certain types of images of positron emission to mography (PET), which is an expensive and complex time-consuming technique.


Our key idea is to use CycleGAN to map the input image to a specified one in the output domain and use a combination of MobileNetV2 and self-attention in the process to achieve global and local feature capture of the input photo. We present experiments on the Alzheimers Disease Neuroimaging Initiative (ADNI) dataset with 726 training and 80 testing subjects and obtain acceptable per- formance in PET image synthesis. We also use various metrics to evaluate the generated images.

GitHub LINK:

Paper draft(on going):

**Related study:

Swapping Auto-Encoder presentation: Slides