医学图像生成 MRI

Multi-modal magnetic resonance imaging (MRI) plays a critical role in clinical diagnosis and treatment nowadays. Each modality of MRI presents its own specific anatomical features which serve as complementary information to other modalities and can provide rich diagnostic information. However, due to the limitations of time consuming and expensive cost, some image sequences of patients may be lost or corrupted, posing an obstacle for accurate diagnosis. Although current multi-modal image synthesis approaches are able to alleviate the issues to some extent, they are still far short of fusing modalities effectively. In light of this, we propose a multi-scale gate mergence based generative adversarial network model, namely MGM-GAN, to synthesize one modality of MRI from others. Notably, we have multiple down-sampling branches corresponding to input modalities to specifically extract their unique features. In contrast to the generic multi-modal fusion approach of averaging or maximizing operations, we introduce a gate mergence (GM) mechanism to automatically learn the weights of different modalities across locations, enhancing the task-related information while suppressing the irrelative information. As such, the feature maps of all the input modalities at each down-sampling level, i.e., multi-scale levels, are integrated via GM module. In addition, both the adversarial loss and the pixel-wise loss, as well as gradient difference loss (GDL) are applied to train the network to produce the desired modality accurately. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art multi-modal image synthesis methods.

Medical imaging plays a critical role in various clinical applications. However, due to multiple considerations such as cost and radiation dose, the acquisition of certain image modalities may be limited. Thus, medical image synthesis can be of great benefit by estimating a desired imaging modality without incurring an actual scan. In this paper, we propose a generative adversarial approach to address this challenging problem. Specifically, we train a fully convolutional network (FCN) to generate a target image given a source image. To better model a nonlinear mapping from source to target and to produce more realistic target images, we propose to use the adversarial learning strategy to better model the FCN. Moreover, the FCN is designed to incorporate an image-gradient-difference-based loss function to avoid generating blurry target images. Long-term residual unit is also explored to help the training of the network. We further apply Auto-Context Model to implement a context-aware deep convolutional adversarial network. Experimental results show that our method is accurate and robust for synthesizing target images from the corresponding source images. In particular, we evaluate our method on three datasets, to address the tasks of generating CT from MRI and generating 7T MRI from 3T MRI images. Our method outperforms the state-of-the-art methods under comparison in all datasets and tasks.

Computed tomography (CT) is critical for various clinical applications, e.g., radiation treatment planning and also PET attenuation correction in MRI/PET scanner. However, CT exposes radiation during acquisition, which may cause side effects to patients. Compared to CT, magnetic resonance imaging (MRI) is much safer and does not involve radiations. Therefore, recently researchers are greatly motivated to estimate CT image from its corresponding MR image of the same subject for the case of radiation planning. In this paper, we propose a data-driven approach to address this challenging problem. Specifically, we train a fully convolutional network (FCN) to generate CT given the MR image. To better model the nonlinear mapping from MRI to CT and produce more realistic images, we propose to use the adversarial training strategy to train the FCN. Moreover, we propose an image-gradient-difference based loss function to alleviate the blurriness of the generated CT. We further apply Auto-Context Model (ACM) to implement a context-aware generative adversarial network. Experimental results show that our method is accurate and robust for predicting CT images from MR images, and also outperforms three state-of-the-art methods under comparison.

Background Cerebral blood volume (CBV) maps derived from dynamic susceptibility contrast-enhanced (DSC) MRI are useful but not commonly available in clinical scenarios. Purpose To test image-to-image translation techniques for generating CBV maps from standard MRI sequences of brain tumors using the bookend technique DSC MRI as ground-truth references. Materials and Methods A total of 756 MRI examinations, including quantitative CBV maps produced from bookend DSC MRI, were included in this retrospective study. Two algorithms, the feature-consistency generative adversarial network (GAN) and three-dimensional encoder-decoder network with only mean absolute error loss, were trained to synthesize CBV maps. The performance of the two algorithms was evaluated quantitatively using the structural similarity index (SSIM) and qualitatively by two neuroradiologists using a four-point Likert scale. The clinical value of combining synthetic CBV maps and standard MRI scans of brain tumors was assessed in several clinical scenarios (tumor grading, prognosis prediction, differential diagnosis) using multicenter data sets (four external and one internal). Differences in diagnostic and predictive accuracy were tested using the z test. Results The three-dimensional encoder-decoder network with T1-weighted images, contrast-enhanced T1-weighted images, and apparent diffusion coefficient maps as the input achieved the highest synthetic performance (SSIM, 86.29% ± 4.30). The mean qualitative score of the synthesized CBV maps by neuroradiologists was 2.63. Combining synthetic CBV with standard MRI improved the accuracy of grading gliomas (standard MRI scans area under the receiver operating characteristic curve [AUC], 0.707; standard MRI scans with CBV maps AUC, 0.857; z = 15.17; P < .001), prediction of prognosis in gliomas (standard MRI scans AUC, 0.654; standard MRI scans with CBV maps AUC, 0.793; z = 9.62; P < .001), and differential diagnosis between tumor recurrence and treatment response in gliomas (standard MRI scans AUC, 0.778; standard MRI scans with CBV maps AUC, 0.853; z = 4.86; P < .001) and brain metastases (standard MRI scans AUC, 0.749; standard MRI scans with CBV maps AUC, 0.857; z = 6.13; P < .001). Conclusion GAN image-to-image translation techniques produced accurate synthetic CBV maps from standard MRI scans, which could be used for improving the clinical evaluation of brain tumors. Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Branstetter in this issue.

Generative Adversarial Networks (GANs) is one of the vital efficient methods for generating a massive, high-quality artificial picture. For diagnosing particular diseases in a medical image, a general problem is that it is expensive, usage of high radiation dosage, and time-consuming to collect data. Hence GAN is a deep learning method that has been developed for the image to image translation, i.e. from low-resolution to highresolution image, for example generating Magnetic resonance image (MRI) from computed tomography image (CT) and 7T from 3T MRI which can be used to obtain multimodal datasets from single modality. In this review paper, different GAN architectures were discussed for medical image analysis.

Magnetic resonance imaging (MRI) is widely used for analyzing human brain structure and function. MRI is extremely versatile and can produce different tissue contrasts as required by the study design. For reasons such as patient comfort, cost, and improving technology, certain tissue contrasts for a cohort analysis may not have been acquired during the imaging session. This missing pulse sequence hampers consistent neuroanatomy research. One possible solution is to synthesize the missing sequence. This paper proposes a data-driven approach to image synthesis, which provides equal, if not superior synthesis compared to the state-of-the-art, in addition to being an order of magnitude faster. The synthesis transformation is done on image patches by a trained bagged ensemble of regression trees. Validation was done by synthesizing T2-weighted contrasts from T1-weighted scans, for phantoms and real data. We also synthesized 3 Tesla T1-weighted magnetization prepared rapid gradient echo (MPRAGE) images from 1.5 Tesla MPRAGEs to demonstrate the generality of this approach.

… MRI synthesis. Inspired by the previous works of image style transferring, we argue that the MRI images can … method considers both aspects for better synthesis. Specifically, our method …

Multi-contrast MRI images use different echo and repetition times to highlight different tissues. However, not all desired image contrasts may be available due to scan-time limitations, suboptimal signal-to-noise ratio, and/or image artifacts. Deep learning approaches have brought revolutionary advances in medical image synthesis, enabling the generation of unacquired image contrasts (e.g., T1-weighted MRI images) from available image contrasts (e.g., T2-weighted images). Particularly, CycleGAN is an advanced technique for image synthesis using unpaired images. However, it requires two separate image generators, demanding more training resources and computations. Recently, a switchable CycleGAN has been proposed to address this limitation and successfully implemented using CT images. However, it remains unclear if switchable CycleGAN can be applied to cross-contrast MRI synthesis. In addition, whether switchable CycleGAN is able to outperform original CycleGAN on cross-contrast MRI image synthesis is still an open question. In this paper, we developed a switchable CycleGAN model for image synthesis between multi-contrast brain MRI images using a large set of publicly accessible pediatric structural brain MRI images. We conducted extensive experiments to compare switchable CycleGAN with original CycleGAN both quantitatively and qualitatively. Experimental results demonstrate that switchable CycleGAN is able to outperform CycleGAN model on pediatric MRI brain image synthesis.

… ) image. … the image synthesis using the composite MRI image. We determine which of the T1-weighted, T2-weighted, and composite image types can be appropriated as the input image …

In magnetic resonance imaging (MRI), several images can be obtained using different imaging settings (e.g. T1, T2, DWI, and Flair). These images have similar anatomical structures but are with different contrasts, which provide a wealth of information for diagnosis. However, the images under specific imaging settings may not be available due to the limitation of scanning time or corruption caused by noises. It is attractive to derive missing images with some settings from the available MR images. In this paper, we propose a novel end-to-end multisetting MR image synthesis method. The proposed method is based on generative adversarial networks (GANs) - a deep learning model. In the proposed method, different MR images obtained by different settings are used as the inputs of a GANs and each image is encoded by an encoder. Each encoder includes a refinement structure which is used to extract a multiscale feature map from an input image. The multiscale feature maps from different input images are then fused to generate several desired target images under specific settings. Because the resultant images obtained with GANs have blurred edges, we fuse gradient prior information in the model to protect high frequency information such as important tissue textures of medical images. In the proposed model, the multiscale information is also adopted in the adversarial learning (not just in the generator or discriminator) so that we can produce high quality synthesized images. We evaluated the proposed method on two public datasets: BRATS and ISLES. Experimental results demonstrate that the proposed approach is superior to current state-of-the-art methods.

… edge image of target modality. We assume that infusing the auxiliary edge image generation … state-of-the-art image synthesis approaches in both qualitative and quantitative measures. …

Background Cross-modality image estimation can be performed using generative adversarial networks (GANs). To date, SPECT image estimation from another medical imaging modality using this technique has not been considered. We evaluate the estimation of SPECT from MRI and PET, and additionally assess the necessity for cross-modality image registration for GAN training. Methods We estimated interictal SPECT from PET and MRI as a single-channel input, and as a multi-channel input to the GAN. We collected data from 48 individuals with epilepsy and converted them to 3D isotropic images for consistence across the modalities. Training and testing data were prepared in native and template spaces. The Pix2pix framework within the GAN network was adopted. We evaluated the addition of the structural similarity index metric to the loss function in the GAN implementation. Root-mean-square error, structural similarity index, and peak signal-to-noise ratio were used to assess how well SPECT images were able to be synthesised. Results High quality SPECT images could be synthesised in each case. On average, the use of native space images resulted in a 5.4% percentage improvement in SSIM than the use of images registered to template space. The addition of structural similarity index metric to the GAN loss function did not result in improved synthetic SPECT images. Using PET in either the single channel or dual channel implementation led to the best results, however MRI could produce SPECT images close in quality. Conclusion Synthesis of SPECT from MRI or PET can potentially reduce the number of scans needed for epilepsy patient evaluation and reduce patient exposure to radiation.

… target images. To enhance interpretable metadata-driven control over image synthesis across … source and target metadata into the image translation process. Extensive experiments on …

OBJECTIVE The purpose of this work is to develop and validate a learning-based method to derive electron density from routine anatomical MRI for potential MRI-based SBRT treatment planning. METHODS We proposed to integrate dense block into cycle generative adversarial network (GAN) to effectively capture the relationship between the CT and MRI for CT synthesis. A cohort of 21 patients with co-registered CT and MR pairs were used to evaluate our proposed method by the leave-one-out cross-validation. Mean absolute error, peak signal-to-noise ratio and normalized cross-correlation were used to quantify the imaging differences between the synthetic CT (sCT) and CT. The accuracy of Hounsfield unit (HU) values in sCT for dose calculation was evaluated by comparing the dose distribution in sCT-based and CT-based treatment planning. Clinically relevant dose-volume histogram metrics were then extracted from the sCT-based and CT-based plans for quantitative comparison. RESULTS The mean absolute error, peak signal-to-noise ratio and normalized cross-correlation of the sCT were 72.87 ± 18.16 HU, 22.65 ± 3.63 dB and 0.92 ± 0.04, respectively. No significant differences were observed in the majority of the planning target volume and organ at risk dose-volume histogram metrics ( p  >  0.05). The average pass rate of γ analysis was over 99% with 1%/1 mm acceptance criteria on the coronal plane that intersects with isocenter. CONCLUSION The image similarity and dosimetric agreement between sCT and original CT warrant further development of an MRI-only workflow for liver stereotactic body radiation therapy. ADVANCES IN KNOWLEDGE This work is the first deep-learning-based approach to generating abdominal sCT through dense-cycle-GAN. This method can successfully generate the small bony structures such as the rib bones and is able to predict the HU values for dose calculation with comparable accuracy to reference CT images.

Deep learning (DL) has recently emerged as a pivotal technology for enhancing magnetic resonance imaging (MRI), a critical tool in diagnostic radiology. This review paper provides a comprehensive overview of recent advances in DL for MRI reconstruction, and focuses on various DL approaches and architectures designed to improve image quality, accelerate scans, and address data-related challenges. It explores end-to-end neural networks, pre-trained and generative models, and self-supervised methods, and highlights their contributions to overcoming traditional MRI limitations. It also discusses the role of DL in optimizing acquisition protocols, enhancing robustness against distribution shifts, and tackling biases. Drawing on the extensive literature and practical insights, it outlines current successes, limitations, and future directions for leveraging DL in MRI reconstruction, while emphasizing the potential of DL to significantly impact clinical imaging practices.

OBJECTIVES Quantitative synthetic magnetic resonance imaging (MRI) enables synthesis of various contrast-weighted images as well as simultaneous quantification of T1 and T2 relaxation times and proton density. However, to date, it has been challenging to generate magnetic resonance angiography (MRA) images with synthetic MRI. The purpose of this study was to develop a deep learning algorithm to generate MRA images based on 3D synthetic MRI raw data. MATERIALS AND METHODS Eleven healthy volunteers and 4 patients with intracranial aneurysms were included in this study. All participants underwent a time-of-flight (TOF) MRA sequence and a 3D-QALAS synthetic MRI sequence. The 3D-QALAS sequence acquires 5 raw images, which were used as the input for a deep learning network. The input was converted to its corresponding MRA images by a combination of a single-convolution and a U-net model with a 5-fold cross-validation, which were then compared with a simple linear combination model. Image quality was evaluated by calculating the peak signal-to-noise ratio (PSNR), structural similarity index measurements (SSIMs), and high frequency error norm (HFEN). These calculations were performed for deep learning MRA (DL-MRA) and linear combination MRA (linear-MR), relative to TOF-MRA, and compared with each other using a nonparametric Wilcoxon signed-rank test. Overall image quality and branch visualization, each scored on a 5-point Likert scale, were blindly and independently rated by 2 board-certified radiologists. RESULTS Deep learning MRA was successfully obtained in all subjects. The mean PSNR, SSIM, and HFEN of the DL-MRA were significantly higher, higher, and lower, respectively, than those of the linear-MRA (PSNR, 35.3 ± 0.5 vs 34.0 ± 0.5, P < 0.001; SSIM, 0.93 ± 0.02 vs 0.82 ± 0.02, P < 0.001; HFEN, 0.61 ± 0.08 vs 0.86 ± 0.05, P < 0.001). The overall image quality of the DL-MRA was comparable to that of TOF-MRA (4.2 ± 0.7 vs 4.4 ± 0.7, P = 0.99), and both types of images were superior to that of linear-MRA (1.5 ± 0.6, for both P < 0.001). No significant differences were identified between DL-MRA and TOF-MRA in the branch visibility of intracranial arteries, except for ophthalmic artery (1.2 ± 0.5 vs 2.3 ± 1.2, P < 0.001). CONCLUSIONS Magnetic resonance angiography generated by deep learning from 3D synthetic MRI data visualized major intracranial arteries as effectively as TOF-MRA, with inherently aligned quantitative maps and multiple contrast-weighted images. Our proposed algorithm may be useful as a screening tool for intracranial aneurysms without requiring additional scanning time.

… of a deep learning-based method for pelvic synthetic CT (sCT) generation that can be used … cycleGAN) framework to effectively learn the nonlinear mapping between MRI and CT pairs. …

… This work sought to establish a novel method on generating liver sCT from corresponding MRI dataset by applying a dense-block cycle GAN model. To quantitatively evaluate the …

BACKGROUND AND PURPOSE MRI-only planning relies on dosimetrically accurate synthetic-CT (sCT) generation to allow dose calculation. Here we validated the dosimetric accuracy of sCTs generated using a deep learning algorithm for pelvic, brain and head and neck (H&N) cancer sites using variable MRI data from multiple scanners. METHODS sCT generation models were trained using a cycle-GAN algorithm, using paired MRI-CT patient data. Input MRI sequences were: T2 for pelvis, T1 with gadolinium (T1Gd) and T2 FLAIR for brain and T1 for H&N. Patient validation sCTs were generated for each site (49 - pelvis, 25 - brain and 30 - H&N). VMAT plans, following local clinical protocols, were calculated on planning CTs and recalculated on sCTs. HU and dosimetric differences were assessed, including DVH differences and gamma index (2%/2mm). RESULTS Mean absolute error (MAE) HU differences were; 48.8 HU (pelvis), 118 HU (T2 FLAIR brain), 126 HU (T1Gd brain) and 124 HU (H&N). Mean primary PTV D95% dose differences for all sites were <0.2 % (range: -0.9 to 1.0 %). Mean 2%/2mm and 1%/1mm gamma pass rates for all sites were > 99.6 % (min: 95.3 %) and >97.3 % (min: 80.1 %) respectively. For all OARs for all sites, mean dose differences were <0.4 %. CONCLUSION Generated sCTs had excellent dosimetric accuracy for all sites and sequences. The cycle-GAN model, available on the research version of a commercial treatment planning system, is a feasible method for sCT generation with high clinical utility due to its ability to use variable input data from multiple scanners and sequences.

Synthesizing computed tomography (CT) images from magnetic resonance imaging (MRI) data can provide the necessary electron density information for accurate dose calculation in the treatment planning of MRI-guided radiation therapy (MRIgRT). Inputting multimodality MRI data can provide sufficient information for accurate CT synthesis: however, obtaining the necessary number of MRI modalities is clinically expensive and time-consuming. In this study, we propose a multimodality MRI synchronous construction based deep learning framework from a single T1-weight (T1) image for MRIgRT synthetic CT (sCT) image generation. The network is mainly based on a generative adversarial network with sequential subtasks of intermediately generating synthetic MRIs and jointly generating the sCT image from the single T1 MRI. It contains a multitask generator and a multibranch discriminator, where the generator consists of a shared encoder and a splitted multibranch decoder. Specific attention modules are designed within the generator for feasible high-dimensional feature representation and fusion. Fifty patients with nasopharyngeal carcinoma who had undergone radiotherapy and had CT and sufficient MRI modalities scanned (5550 image slices for each modality) were used in the experiment. Results showed that our proposed network outperforms state-of-the-art sCT generation methods well with the least MAE, NRMSE, and comparable PSNR and SSIM index measure. Our proposed network exhibits comparable or even superior performance than the multimodality MRI-based generation method although it only takes a single T1 MRI image as input, thereby providing a more effective and economic solution for the laborious and high-cost generation of sCT images in clinical applications.

… of generating high-quality, three-dimensional contrast-enhanced multiphasic liver MRI … results highlight the potential to transform liver MRI workflows by reducing contrast media costs …

Deep learning has been recognized as a paradigm-shifting tool in radiology. Deep learning reconstruction (DLR) has recently emerged as a technology used in the image reconstruction process of MRI, which is an essential procedure in generating MR images. Denoising, which is the first DLR application to be realized in commercial MRI scanners, improves signal-to-noise ratio. When applied to lower magnetic field-strength scanners, the signal-to-noise ratio can be increased without extending the imaging time, and image quality is comparable to that of higher-field-strength scanners. Shorter imaging times decrease patient discomfort and reduce MRI scanner running costs. The incorporation of DLR into accelerated acquisition imaging techniques, such as parallel imaging or compressed sensing, shortens the reconstruction time. DLR is based on supervised learning using convolutional layers and is divided into the following three categories: image domain, k-space learning, and direct mapping types. Various studies have reported other derivatives of DLR, and several have shown the feasibility of DLR in clinical practice. Although DLR efficiently reduces Gaussian noise from MR images, denoising makes image artifacts more prominent, and a solution to this problem is desired. Depending on the training of the convolutional neural network, DLR may change the imaging features of lesions and obscure small lesions. Therefore, radiologists may need to adopt the habit of questioning whether any information has been lost on images that appear clean. ©RSNA, 2023 Quiz questions for this article are available in the supplemental material.

… This work extends on this idea, evaluating what types of MRI inputs are … MRI as the input was effective at generating synthetic CT. This was a surprising result, as we expected ZTE MRI …

… Considering that the image generated by synthetic MRI … MRI, we propose a multichannel U-Net-based deep learning architecture for quantification maps and weighted image generation…

… In this paper, we have developed a novel deep learning-based synthetic CT generation method … We integrated of cycle-GAN and dense block into deep learning-based framework to …

Deep learning models have been used in several domains, however, adjusting is still required to be applied in sensitive areas such as medical imaging. As the use of technology in the medical domain is needed because of the time limit, the level of accuracy assures trustworthiness. Because of privacy concerns, machine learning applications in the medical field are unable to use medical data. For example, the lack of brain MRI images makes it difficult to classify brain tumors using image-based classification. The solution to this challenge was achieved through the application of Generative Adversarial Network (GAN)-based augmentation techniques. Deep Convolutional GAN (DCGAN) and Vanilla GAN are two examples of GAN architectures used for image generation. In this paper, a framework, denoted as BrainGAN, for generating and classifying brain MRI images using GAN architectures and deep learning models was proposed. Consequently, this study proposed an automatic way to check that generated images are satisfactory. It uses three models: CNN, MobileNetV2, and ResNet152V2. Training the deep transfer models with images made by Vanilla GAN and DCGAN, and then evaluating their performance on a test set composed of real brain MRI images. From the results of the experiment, it was found that the ResNet152V2 model outperformed the other two models. The ResNet152V2 achieved 99.09% accuracy, 99.12% precision, 99.08% recall, 99.51% area under the curve (AUC), and 0.196 loss based on the brain MRI images generated by DCGAN architecture.

… brain MRI to a low-dimensional manifold, but have been limited by assumptions of explicit similarity measures. In this work, we use deep learning … The architecture of the generator and …

Recently, deep learning (DL)-based methods for the generation of synthetic computed tomography (sCT) have received significant research attention as an alternative to classical ones. We present here a systematic review of these methods by grouping them into three categories, according to their clinical applications: I) to replace CT in magnetic resonance (MR)-based treatment planning, II) facilitate cone-beam computed tomography (CBCT)-based image-guided adaptive radiotherapy, and III) derive attenuation maps for the correction of positron emission tomography (PET). Appropriate database searching was performed on journal articles published between January 2014 and December 2020. The DL methods' key characteristics were extracted from each eligible study, and a comprehensive comparison among network architectures and metrics was reported. A detailed review of each category was given, highlighting essential contributions, identifying specific challenges, and summarising the achievements. Lastly, the statistics of all the cited works from various aspects were analysed, revealing the popularity and future trends and the potential of DL-based sCT generation. The current status of DL-based sCT generation was evaluated, assessing the clinical readiness of the presented methods.

Objectives The aim of this study was to compare deep learning reconstructed (DLR) 0.55 T magnetic resonance imaging (MRI) quality, identification, and grading of structural anomalies and reader confidence levels with conventional 3 T knee MRI in patients with knee pain following trauma. Materials and Methods This prospective study of 26 symptomatic patients (5 women) includes 52 paired DLR 0.55 T and conventional 3 T MRI examinations obtained in 1 setting. A novel, commercially available DLR algorithm was employed for 0.55 T image reconstruction. Four board-certified radiologists reviewed all images independently and graded image quality, noted structural anomalies and their respective reporting confidence levels for the presence or absence, as well as grading of bone, cartilage, meniscus, ligament, and tendon lesions. Image quality and reader confidence levels were compared (P < 0.05, significant), and MRI findings were correlated between 0.55 T and 3 T MRI using Cohen kappa (κ). Results In reader's consensus, good image quality was found for DLR 0.55 T MRI and 3 T MRI (3.8 vs 4.1/5 points, P = 0.06). There was near-perfect agreement between 0.55 T DLR and 3 T MRI regarding the identification of structural anomalies for all readers (each κ ≥ 0.80). Substantial to near-perfection agreement between 0.55 T and 3 T MRI was reported for grading of cartilage (κ = 0.65–0.86) and meniscus lesions (κ = 0.71–1.0). High confidence levels were found for all readers for DLR 0.55 T and 3 T MRI, with 3 readers showing higher confidence levels for reporting cartilage lesions on 3 T MRI. Conclusions In conclusion, new-generation 0.55 T DLR MRI provides good image quality, comparable to conventional 3 T MRI, and allows for reliable identification of internal derangement of the knee with high reader confidence.

In recent years, Magnetic Resonance Imaging (MRI) has emerged as a prevalent medical imaging technique, offering comprehensive anatomical and functional information. However, the MRI data acquisition process presents several challenges, including time-consuming procedures, prone motion artifacts, and hardware constraints. To address these limitations, this study proposes a novel method that leverages the power of generative adversarial networks (GANs) to generate multi-domain MRI images from a single input MRI image. Within this framework, two primary generator architectures, namely ResUnet and StarGANs generators, were incorporated. Furthermore, the networks were trained on multiple datasets, thereby augmenting the available data, and enabling the generation of images with diverse contrasts obtained from different datasets, given an input image from another dataset. Experimental evaluations conducted on the IXI and BraTS2020 datasets substantiate the efficacy of the proposed method compared to an existing method, as assessed through metrics such as Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR) and Normalized Mean Absolute Error (NMAE). The synthesized images resulting from this method hold substantial potential as invaluable resources for medical professionals engaged in research, education, and clinical applications. Future research gears towards expanding experiments to larger datasets and encompassing the proposed approach to 3D images, enhancing medical diagnostics within practical applications.

Recently, cross-modal medical image synthesis has been receiving significant attention due to its potential applications such as registration, segmentation, etc. In this chapter, we …

Acquiring complementary information about tissue morphology from multimodal medical images is beneficial to clinical disease diagnosis, but it cannot be widely used due to the cost of scans. In such cases, medical image synthesis has become a popular area. Recently, generative adversarial network (GAN) models are applied to many medical image synthesis tasks and show prior performance, since they enable to capture structural details clearly. However, GAN still builds the main framework based on convolutional neural network (CNN) that exhibits a strong locality bias and spatial invariance through the use of shared weights across all positions. Therefore, the long-range dependencies have been destroyed in this processing. To address this issue, we introduce a double-scale deep learning method for cross-modal medical image synthesis. More specifically, the proposed method captures locality feature via local discriminator based on CNN and utilizes long-range dependencies to learning global feature through global discriminator based on transformer architecture. To evaluate the effectiveness of double-scale GAN, we conduct folds of experiments on the standard benchmark IXI dataset and experimental results demonstrate the effectiveness of our method.

… We innovatively address the challenge of heterogeneous domain matching in cross-modal medical image synthesis by introducing a domain constraint based on MMD. This term …

Cross-modal generation has emerged as a crucial method for addressing the challenge of filling in missing modalities in medical imaging. Existing approaches predominantly utilize convolutional neural networks (CNNs) or vision transformers (ViTs) and their variants as model backbones. Consequently, issues arise concerning limited receptive fields and significant increases in computational costs. This paper proposes a latent feature space-based multi-scale residual ViT generative adversarial model (LMRT-NET), which leverages the global sensitivity of ViTs and the local precision of CNNs while reducing computational costs. The generator of LMRT-NET comprises an encoder-decoder architecture that enhances performance and lowers computational expenses through the use of multi-scale dynamic aggregation residual ViT (DART) blocks in the latent feature space. This module consists of two layers of residual convolutional blocks and transformer blocks of different scales, where the transformer blocks assist the convolutional blocks in capturing contextual features, and lower-level blocks support higher-level blocks in learning high-dimensional global information. Additionally, a multi-level information fusion (MIF) module is integrated into the encoder-decoder and latent feature space, consisting of a dual-scale selective fusion (DSF) module that adaptively aggregates multi-scale information to generate target modality images. Extensive experimental results across three different datasets demonstrate that LMRT-NET outperforms baseline methods in terms of image generation quality and generalization capability. Our code will be released on https://github.com/ffan14/LMRT-NET.

… function for cross-modal medical image registration [19]. Motivated by this, in this paper, we use mutual information as a cost function for cross-modal medical image synthesis. Since we …

Synthesis of unavailable imaging modalities from available ones can generate modality-specific complementary information and enable multi-modality based medical images diagnosis or treatment. Existing generative methods for medical image synthesis are usually based on cross-modal translation between acquired and missing modalities. These methods are usually dedicated to specific missing modality and perform synthesis in one shot, which cannot deal with varying number of missing modalities flexibly and construct the mapping across modalities effectively. To address the above issues, in this paper, we propose a unified Multi-modal Modality-masked Diffusion Network (M2DN), tackling multi-modal synthesis from the perspective of “progressive whole-modality inpainting”, instead of “cross-modal translation”. Specifically, our M2DN considers the missing modalities as random noise and takes all the modalities as a unity in each reverse diffusion step. The proposed joint synthesis scheme performs synthesis for the missing modalities and self-reconstruction for the available ones, which not only enables synthesis for arbitrary missing scenarios, but also facilitates the construction of common latent space and enhances the model representation ability. Besides, we introduce a modality-mask scheme to encode availability status of each incoming modality explicitly in a binary mask, which is adopted as condition for the diffusion model to further enhance the synthesis performance of our M2DN for arbitrary missing scenarios. We carry out experiments on two public brain MRI datasets for synthesis and downstream segmentation tasks. Experimental results demonstrate that our M2DN outperforms the state-of-the-art models significantly and shows great generalizability for arbitrary missing modalities.

Medical generative models, acknowledged for their high-quality sample generation ability, have accelerated the fast growth of medical applications. However, recent works concentrate on separate medical generation models for dis-tinct medical tasks and are restricted to inadequate medi-cal multimodal knowledge, constraining medical compre-hensive diagnosis. In this paper, we propose MedM2G, a Medical Multi-Modal Generative framework, with the key innovation to align, extract, and generate medical multimodal within a unified model. Extending beyond single or two medical modalities, we efficiently align medical multimodal through the central alignment approach in the unified space. Significantly, our framework extracts valuable clini-cal knowledge by preserving the medical visual invariant of each imaging modal, thereby enhancing specific medical information for multimodal generation. By conditioning the adaptive cross-guided parameters into the multi-flow diffusion framework, our model promotes flexible interactions among medical multimodalfor generation. MedM2G is the first medical generative model that unifies medical generation tasks of text-to-image, image-to-text, and unified generation of medical modalities (CT, MRI, X-ray). It performs 5 medical generation tasks across 10 datasets, consistently outperforming various state-of-the-art works.

… The aim of the study is to devise a cross-device and cross-modal medical image synthesis (MIS) method Cross 2 SynNet for synthesizing routine brain MRI sequences of T1WI, T2WI, …

ABSTRACT Deep learning-based computer-aided diagnosis (CAD) of medical images requires large datasets. However, the lack of large publicly available labelled datasets limits the development of deep learning-based CAD systems. Generative Adversarial Networks (GANs), in particular, CycleGAN, can be used to generate new cross-domain images without paired training data. However, most CycleGAN-based synthesis methods lack the potential to overcome alignment and asymmetry between the input and generated data. We propose a two-stage technique for the synthesis of abdominal MRI using cross-modality translation of abdominal CT. We show that the synthetic data can help improve the performance of the liver segmentation network. We increase the number of abdominal MRI images through cross-modality image transformation of unpaired CT images using a CycleGAN inspired deformation invariant network called EssNet. Subsequently, we combine the synthetic MRI images with the original MRI images and use them to improve the accuracy of the U-Net on a liver segmentation task. We train the U-Net on real MRI images and then on real and synthetic MRI images. Consequently, by comparing both scenarios, we achieve an improvement in the performance of U-Net. In summary, the improvement achieved in the Intersection over Union (IoU) is 1.17%. The results show the potential to address the data scarcity challenge in medical imaging.

… Furthermore, super-resolution has been coupled with cross-modal synthesis in a dictionary … Unpaired data has also been used for cross-modal synthesis in an optimisation scheme [22]…

Image reconstruction in magnetic resonance imaging (MRI) and computed tomography (CT) is a mathematical process that generates images at many different angles around the patient. Image reconstruction has a fundamental impact on image quality. In recent years, the literature has focused on deep learning and its applications in medical imaging, particularly image reconstruction. Due to the performance of deep learning models in a wide variety of vision applications, a considerable amount of work has recently been carried out using image reconstruction in medical images. MRI and CT appear as the ultimate scientifically appropriate imaging mode for identifying and diagnosing different diseases in this ascension age of technology. This study demonstrates a number of deep learning image reconstruction approaches and a comprehensive review of the most widely used different databases. We also give the challenges and promising future directions for medical image reconstruction.

Abstract Medical image reconstruction aims to acquire high-quality medical images for clinical usage at minimal cost and risk to the patients. Deep learning and its applications in medical imaging, especially in image reconstruction have received considerable attention in the literature in recent years. This study reviews records obtained electronically through the leading scientific databases (Magnetic Resonance Imaging journal, Google Scholar, Scopus, Science Direct, Elsevier, and from other journal publications) searched using three sets of keywords: (1) Deep learning, image reconstruction, medical imaging; (2) Medical imaging, Deep learning, Image reconstruction; (3) Open science, Open imaging data, Open software. The articles reviewed revealed that deep learning-based reconstruction methods improve the quality of reconstructed images qualitatively and quantitatively. However, deep learning techniques are generally computationally expensive, require large amounts of training datasets, lack decent theory to explain why the algorithms work, and have issues of generalization and robustness. The challenge of lack of enough training datasets is currently being addressed by using transfer learning techniques.

Magnetic resonance imaging (MRI) is a sophisticated and versatile medical imaging modality. The inverse FFT has served the MR community very well as the conventional image reconstruction method for k-space data with full Cartesian sampling. And for well sampled non-Cartesian data, the gridding method with appropriate density compensation factors is fast and effective. But when only under-sampled data is available, or when non-Fourier physical effects like field inhomogeneity are important, then gridding/FFT methods for image reconstruction are suboptimal, and iterative algorithms based on appropriate models can improve image quality, rat the price of increased computation. This article reviews the use of iterative algorithms for model-based MR image reconstruction.

BACKGROUND AND OBJECTIVE With the high-resolution (HR) requirements of medical images in clinical practice, super-resolution (SR) reconstruction algorithms based on low-resolution (LR) medical images have become a research hotspot. This type of method can significantly improve image SR without improving hardware equipment, so it is of great significance to review it. METHODS Aiming at the unique SR reconstruction algorithms in the field of medical images, based on subdivided medical fields such as magnetic resonance (MR) images, computed tomography (CT) images, and ultrasound images. Firstly, we deeply analyzed the research progress of SR reconstruction algorithms, and summarized and compared the different types of algorithms. Secondly, we introduced the evaluation indicators corresponding to the SR reconstruction algorithms. Finally, we prospected the development trend of SR reconstruction technology in the medical field. RESULTS The medical image SR reconstruction technology based on deep learning can provide more abundant lesion information, relieve the expert's diagnosis pressure, and improve the diagnosis efficiency and accuracy. CONCLUSION The medical image SR reconstruction technology based on deep learning helps to improve the quality of medicine, provides help for the diagnosis of experts, and lays a solid foundation for the subsequent analysis and identification tasks of the computer, which is of great significance for improving the diagnosis efficiency of experts and realizing intelligent medical care.

Over past several years, machine learning, or more generally artificial intelligence, has generated overwhelming research interest and attracted unprecedented public attention. As tomographic imaging researchers, we share the excitement from our imaging perspective [item 1) in the Appendix], and organized this special issue dedicated to the theme of “Machine learning for image reconstruction.” This special issue is a sister issue of the special issue published in May 2016 of this journal with the theme “Deep learning in medical imaging” [item 2) in the Appendix]. While the previous special issue targeted medical image processing/analysis, this special issue focuses on data-driven tomographic reconstruction. These two special issues are highly complementary, since image reconstruction and image analysis are two of the main pillars for medical imaging. Together we cover the whole workflow of medical imaging: from tomographic raw data/features to reconstructed images and then extracted diagnostic features/readings.

… The MRI delivers high-resolution images of the human body … medical image reconstruction, with a focus on machine learning and deep learning. Because medical image reconstruction …

Abstract Better algorithms for medical image reconstruction can improve image quality and enable reductions in acquisition time and radiation dose. A prior understanding of the distribution of plausible images is key to realising these benefits. Recently, research into deep-learning image reconstruction has started to look into using unsupervised diffusion models, trained only on high-quality medical images (ie, without needing paired scanner measurement data), for modelling this prior understanding. Image reconstruction algorithms incorporating unsupervised diffusion models have already attained state-of-the-art accuracy for reconstruction tasks ranging from highly accelerated MRI to ultra-sparse-view CT and low-dose PET. Key advantages of diffusion model approach over previous deep learning approaches for reconstruction include state-of-the-art image distribution modelling, improved robustness to domain shift, and principled quantification of reconstruction uncertainty. If hallucination concerns can be alleviated, their key advantages and impressive performance could mean these algorithms are better suited to clinical use than previous deep-learning approaches. In this review, we provide an accessible introduction to image reconstruction and diffusion models, outline guidance for using diffusion-model-based reconstruction methodology, summarise modality-specific challenges, and identify key research themes. We conclude with a discussion of the opportunities and challenges of using diffusion models for medical image reconstruction.

… 2D radial cine MRI Here we applied our method to image reconstruction in undersampled 2D radial cine MRI. Typically, MRI is performed using multiple receiver coils and therefore, the …

… This paper provides a complete overview of image reconstruction process in MRI (Magnetic Resonance Imaging). It reviews the computational aspect of medical image reconstruction. …

… We were also the forerunners in applying 3-D imaging based on MRI to the study of the heart [70] and the skeletal joints [71], [72]. Other projects included the study of left-ventricular …

Magnetic resonance (MR) imaging visualises soft tissue contrast in exquisite detail without harmful ionising radiation. In this work, we provide a state‐of‐the‐art review on the use of deep learning in MR image reconstruction from different image acquisition types involving compressed sensing techniques, parallel image acquisition and multi‐contrast imaging. Publications with deep learning‐based image reconstruction for MR imaging were identified from the literature (PubMed and Google Scholar), and a comprehensive description of each of the works was provided. A detailed comparison that highlights the differences, the data used and the performance of each of these works were also made. A discussion of the potential use cases for each of these methods is provided. The sparse image reconstruction methods were found to be most popular in using deep learning for improved performance, accelerating acquisitions by around 4–8 times. Multi‐contrast image reconstruction methods rely on at least one pre‐acquired image, but can achieve 16‐fold, and even up to 32‐ to 50‐fold acceleration depending on the set‐up. Parallel imaging provides frameworks to be integrated in many of these methods for additional speed‐up potential. The successful use of compressed sensing techniques and multi‐contrast imaging with deep learning and parallel acquisition methods could yield significant MR acquisition speed‐ups within clinical routines in the near future.

Magnetic Resonance Imaging (MRI) plays a vital role in diagnosis, management and monitoring of many diseases. However, it is an inherently slow imaging technique. Over the last 20 years, parallel imaging, temporal encoding and compressed sensing have enabled substantial speed-ups in the acquisition of MRI data, by accurately recovering missing lines of k-space data. However, clinical uptake of vastly accelerated acquisitions has been limited, in particular in compressed sensing, due to the time-consuming nature of the reconstructions and unnatural looking images. Following the success of machine learning in a wide range of imaging tasks, there has been a recent explosion in the use of machine learning in the field of MRI image reconstruction. A wide range of approaches have been proposed, which can be applied in k-space and/or image-space. Promising results have been demonstrated from a range of methods, enabling natural looking images and rapid computation. In this review article we summarize the current machine learning approaches used in MRI reconstruction, discuss their drawbacks, clinical applications, and current trends.

Image reconstruction from undersampled k-space data has been playing an important role in fast magnetic resonance imaging (MRI). Recently, deep learning has demonstrated tremendous success in various fields and also shown potential in significantly accelerating MRI reconstruction with fewer measurements. This article provides an overview of deep-learning-based image reconstruction methods for MRI. Two types of deep-learningbased approaches are reviewed, those that are based on unrolled algorithms and those that are not, and the main structures of both are explained. Several signal processing issues for maximizing the potential of deep reconstruction in fast MRI are discussed, which may facilitate further development of the networks and performance analysis from a theoretical point of view.

In modern medical imaging, although there have been advances in the application of super‐resolution technology to MRI in recent years, current applications still cannot meet practical needs. For example, for MRI images under specific pathological or physiological conditions, the existing super‐resolution technology still lacks effectiveness in processing noise and restoring details. And when processing images with complex organizational structures, such as white matter fiber bundles in the brain, existing super‐resolution techniques often fail to accurately restore image details, resulting in structural distortion. To address these deficiencies, we propose in this study an advanced super‐resolution (SR) reconstruction framework tailored specifically for magnetic resonance imaging (MRI). Our approach makes use of the Denoising Diffusion Probabilistic Model (DDPM) and CrossAttention, an advanced technique known for its ability to maintain data accuracy while making the most of available conditions, leading to high‐quality image restoration. By incorporating sophisticated priors and innovative network architecture, our method significantly outperforms traditional SR techniques, particularly in preserving fine anatomical details and enhancing overall image quality. The proposed framework undergoes rigorous validation through extensive experiments on diverse MRI datasets, demonstrating its robustness and effectiveness in various scenarios. Furthermore, we provide a comprehensive analysis of the performance metrics, including structural similarity index (SSIM), peak signal‐to‐noise ratio (PSNR), Normalized Mean Squared Error (NMSE), and Universal Quality Index (UQI), to underscore the superiority of our DDPM‐based approach. This work not only contributes to advancing the state‐of‐the‐art in MRI SR but also paves the way for broader applications in medical imaging and related fields.

BACKGROUND AND PURPOSE: Magnetic resonance imaging (MRI)-based synthetic computed tomography (sCT) simplifies radiation therapy treatment planning by eliminating the need for CT simulation and error-prone image registration, ultimately reducing patient radiation dose and setup uncertainty. In this work, we propose a MRI-to-CT transformer-based improved denoising diffusion probabilistic model (MC-IDDPM) to translate MRI into high-quality sCT to facilitate radiation treatment planning. METHODS: MC-IDDPM implements diffusion processes with a shifted-window transformer network to generate sCT from MRI. The proposed model consists of two processes: a forward process, which involves adding Gaussian noise to real CT scans to create noisy images, and a reverse process, in which a shifted-window transformer V-net (Swin-Vnet) denoises the noisy CT scans conditioned on the MRI from the same patient to produce noise-free CT scans. With an optimally trained Swin-Vnet, the reverse diffusion process was used to generate noise-free sCT scans matching MRI anatomy. We evaluated the proposed method by generating sCT from MRI on an institutional brain dataset and an institutional prostate dataset. Quantitative evaluations were conducted using several metrics, including Mean Absolute Error (MAE), Peak Signal-to-Noise Ratio (PSNR), Multi-scale Structure Similarity Index (SSIM), and Normalized Cross Correlation (NCC). Dosimetry analyses were also performed, including comparisons of mean dose and target dose coverages for 95% and 99%. RESULTS: MC-IDDPM generated brain sCTs with state-of-the-art quantitative results with MAE 48.825 ± 21.491 HU, PSNR 26.491 ± 2.814 dB, SSIM 0.947 ± 0.032, and NCC 0.976 ± 0.019. For the prostate dataset: MAE 55.124 ± 9.414 HU, PSNR 28.708 ± 2.112 dB, SSIM 0.878 ± 0.040, and NCC 0.940 ± 0.039. MC-IDDPM demonstrates a statistically significant improvement (with p < 0.05) in most metrics when compared to competing networks, for both brain and prostate synthetic CT. Dosimetry analyses indicated that the target dose coverage differences by using CT and sCT were within ± 0.34%. CONCLUSIONS: We have developed and validated a novel approach for generating CT images from routine MRIs using a transformer-based improved DDPM. This model effectively captures the complex relationship between CT and MRI images, allowing for robust and high-quality synthetic CT images to be generated in a matter of minutes. This approach has the potential to greatly simplify the treatment planning process for radiation therapy by eliminating the need for additional CT scans, reducing the amount of time patients spend in treatment planning, and enhancing the accuracy of treatment delivery.

Magnetic Resonance Imaging (MRI) is crucial for clinical diagnostics, offering high-resolution anatomical and functional imaging without ionizing radiation. However, prolonged acquisition times in conventional MRI lead to motion artifacts, limiting efficiency and reliability. While deep learning models such as GANs and DDPMs show promise in MRI synthesis, DDPMs suffer from stochastic variability that affects image consistency. This study proposes Synthetic Modality Diffusion (SynthModDiff), a novel multi-domain image-to-image translation framework featuring a two-stage diffusion process with a noise-aware Forward Process and Reverse Process to enhance fidelity and reduce residual noise. Experiments across multiple datasets demonstrate state-of-the-art performance in NMAE, SSIM, and PSNR metrics, while preserving fine anatomical details, making SynthModDiff highly suitable for clinical applications like radiotherapy planning.

… image synthesis. In this work, we propose to integrate the conditional variable discrete diffusion process into the HR prostate MRI synthesis. Rather than directly synthesizing the HR MRI…

… for high-resolution brain MRI synthesis using the brain MRI BraTS2020 dataset. Multiple … To assess the performance of the proposed diffusion model, we have implemented two …

In the field of healthcare, the acquisition of sample is usually restricted by multiple considerations, including cost, labor- intensive annotation, privacy concerns, and radiation hazards, therefore, synthesizing images-of-interest is an important tool to data augmentation. Diffusion models have recently attained state-of-the-art results in various synthesis tasks, and embedding energy functions has been proved that can effectively guide the pre-trained model to synthesize target samples. However, we notice that current method development and validation are still limited to improving indicators, such as Fréchet Inception Distance score (FID) and Inception Score (IS), and have not provided deeper investigations on downstream tasks, like disease grading and diagnosis. Moreover, existing classifier guidance which can be regarded as a special case of energy function can only has a singular effect on altering the distribution of the synthetic dataset. This may contribute to in-distribution synthetic sample that has limited help to downstream model optimization. All these limitations remind that we still have a long way to go to achieve controllable generation. In this work, we first conducted an analysis on previous guidance as well as its contributions on further applications from the perspective of data distribution. To synthesize samples which can help downstream applications, we then introduce uncertainty guidance in each sampling step and design an uncertainty-guided diffusion models. Extensive experiments on four medical datasets, with ten classic networks trained on the augmented sample sets provided a comprehensive evaluation on the practical contributions of our methodology. Furthermore, we provide a theoretical guarantee for general gradient guidance in diffusion models, which would benefit future research on investigating other forms of measurement guidance for specific generative tasks.

Incomplete or faulty MRI sequences are common in clinical practice and can impair AI-based analyses that rely on complete multi-contrast data. The relative effectiveness of classical generative adversarial networks (GANs) versus modern diffusion and transformer-based models for clinically usable MRI synthesis remains unclear. This study evaluates cross-modality MRI synthesis using the BraTS 2019 brain tumour dataset, focusing on T1-to-T2 translation. We assess paired and unpaired CycleGAN models and compare them with two stronger but computationally intensive baselines, a conditional denoising diffusion probabilistic model (DDPM) and a transformer-enhanced GAN, using identical data splits and preprocessing pipelines. Inter-modality correlation was evaluated to estimate the achievable similarity between modalities. Conceptually, modality synthesis may be viewed as a representation-learning approach that compensates for missing imaging information by reconstructing clinically relevant features from available contrasts. Paired CycleGAN achieved correlations of r≈0.92–0.93 and SSIM ≈0.90–0.92, approaching natural T1–T2 correlation (r≈0.95) while maintaining very fast inference (<50 ms/slice). Unpaired CycleGAN achieved r≈0.74–0.78 and SSIM ≈0.82–0.85, producing clinically interpretable reconstructions without voxel-level supervision. DDPM achieved the highest fidelity (SSIM ≈0.93–0.95, r≈0.94) but required substantially greater computational resources, while transformer-enhanced GAN performance was intermediate. Qualitative analysis showed that CycleGAN and DDPM best preserved tumour and tissue boundaries, whereas unpaired CycleGAN occasionally over-smoothed subtle lesions. These findings highlight the trade-off between fidelity and efficiency in cross-modality MRI synthesis, suggesting paired CycleGAN for time-sensitive clinical workflows and diffusion models as a computationally expensive accuracy upper bound.

Introduction Acquisition and pre-processing pipelines for diffusion-weighted imaging (DWI) volumes are resource- and time-consuming. Generating synthetic DWI scalar maps from commonly acquired brain MRI sequences such as fluid-attenuated inversion recovery (FLAIR) could be useful for supplementing datasets. In this work we design and compare GAN-based image translation models for generating DWI scalar maps from FLAIR MRI for the first time. Methods We evaluate a pix2pix model, two modified CycleGANs using paired and unpaired data, and a convolutional autoencoder in synthesizing DWI fractional anisotropy (FA) and mean diffusivity (MD) from whole FLAIR volumes. In total, 420 FLAIR and DWI volumes (11,957 images) from multi-center dementia and vascular disease cohorts were used for training/testing. Generated images were evaluated using two groups of metrics: (1) human perception metrics including peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), (2) structural metrics including a newly proposed histogram similarity (Hist-KL) metric and mean squared error (MSE). Results Pix2pix demonstrated the best performance both quantitatively and qualitatively with mean PSNR, SSIM, and MSE metrics of 23.41 dB, 0.8, 0.004, respectively for MD generation, and 24.05 dB, 0.78, 0.004, respectively for FA generation. The new histogram similarity metric demonstrated sensitivity to differences in fine details between generated and real images with mean pix2pix MD and FA Hist-KL metrics of 11.73 and 3.74, respectively. Detailed analysis of clinically relevant regions of white matter (WM) and gray matter (GM) in the pix2pix images also showed strong significant (p < 0.001) correlations between real and synthetic FA values in both tissue types (R = 0.714 for GM, R = 0.877 for WM). Discussion/conclusion Our results show that pix2pix’s FA and MD models had significantly better structural similarity of tissue structures and fine details than other models, including WM tracts and CSF spaces, between real and generated images. Regional analysis of synthetic volumes showed that synthetic DWI images can not only be used to supplement clinical datasets, but demonstrates potential utility in bypassing or correcting registration in data pre-processing.

Hepatobiliary phase (HBP) imaging is of great value in the diagnosis of liver cancer. However, the imaging process of HBP is very time-consuming, and it needs to wait 20 minutes after the injection of contrast agent. In order to obtain HBP images more conveniently and efficiently in clinic, we propose a novel method based on multimodal image synthesis with deep learning, which generates HBP images through contrast-enhanced images and diffusion models. We first independently train the late diffusion models for generating HBP images for each modality, and pre-align all encoders through contrastive learning. Then, these diffusion models effectively learn to focus on cross-modal joint multimodal generation through a transformer mechanism called "latent spatiotemporal alignment" (LSAT). In order to model spatiotemporal characteristics of contrast-enhanced MR while maintaining visual generation quality, we propose a spatiotemporal module to construct diffusers and achieve cross-attention in the diffusion stream during the joint generation process. The results of clinical experiments demonstrate the effectiveness of the proposed method, indicating that the proposed method is superior to the typical image synthesis methods. In addition, the ablation study also demonstrates the effectiveness of the proposed modules. Finally, we quantitatively reveal the contribution of different modalities of contrast-enhanced MR to the generation of HBP images, making it easier for clinical interpretability.

… Cola-diff: conditional latent diffusion model for multi-modal mri synthesis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 398–408. …

There is a growing demand for high-resolution (HR) medical images for both clinical and research applications. Image quality is inevitably traded off with acquisition time, which in turn impacts patient comfort, examination costs, dose, and motion-induced artifacts. For many image-based tasks, increasing the apparent spatial resolution in the perpendicular plane to produce multi-planar reformats or 3D images is commonly used. Single-image super-resolution (SR) is a promising technique to provide HR images based on deep learning to increase the resolution of a 2D image, but there are few reports on 3D SR. Further, perceptual loss is proposed in the literature to better capture the textural details and edges versus pixel-wise loss functions, by comparing the semantic distances in the high-dimensional feature space of a pre-trained 2D network (e.g., VGG). However, it is not clear how one should generalize it to 3D medical images, and the attendant implications are unclear. In this paper, we propose a framework called SOUP-GAN: Super-resolution Optimized Using Perceptual-tuned Generative Adversarial Network (GAN), in order to produce thinner slices (e.g., higher resolution in the ‘Z’ plane) with anti-aliasing and deblurring. The proposed method outperforms other conventional resolution-enhancement methods and previous SR work on medical images based on both qualitative and quantitative comparisons. Moreover, we examine the model in terms of its generalization for arbitrarily user-selected SR ratios and imaging modalities. Our model shows promise as a novel 3D SR interpolation technique, providing potential applications for both clinical and research applications.

… In this paper, based on the neural network model termed as GAN-CIRCLE (… super-resolution for both MRI and CT. In this study, we demonstrate two-fold resolution enhancement for MRI …

MRI super-resolution (SR) and denoising tasks are fundamental challenges in the field of deep learning, which have traditionally been treated as distinct tasks with separate paired training data. In this paper, we propose an innovative method that addresses both tasks simultaneously using a single deep learning model, eliminating the need for explicitly paired noisy and clean images during training. Our proposed model is primarily trained for SR, but also exhibits remarkable noise-cleaning capabilities in the super-resolved images. Instead of conventional approaches that introduce frequency-related operations into the generative process, our novel approach involves the use of a GAN model guided by a frequency-informed discriminator. To achieve this, we harness the power of the 3D Discrete Wavelet Transform (DWT) operation as a frequency constraint within the GAN framework for the SR task on magnetic resonance imaging (MRI) data. Specifically, our contributions include: 1) a 3D generator based on residual-in-residual connected blocks; 2) the integration of the 3D DWT with 1 × 1 convolution into a DWT+conv unit within a 3D Unet for the discriminator; 3) the use of the trained model for high-quality image SR, accompanied by an intrinsic denoising process. We dub the model "Denoising Induced Super-resolution GAN (DISGAN)" due to its dual effects of SR image generation and simultaneous denoising. Departing from the traditional approach of training SR and denoising tasks as separate models, our proposed DISGAN is trained only on the SR task, but also achieves exceptional performance in denoising. The model is trained on 3D MRI data from dozens of subjects from the Human Connectome Project (HCP) and further evaluated on previously unseen MRI data from subjects with brain tumours and epilepsy to assess its denoising and SR performance. Our code is available at https://github.com/wqlevi/DISGAN.

Abstract Generative Adversarial Models (GANs) have been quite popular and are currently and active area of research. They can be used for generative new data and study adversarial samples and attacks. We have used the similar approach to apply super-resolution to medical images. In Radiology MRI is a commonly used method to produce medical imaging but the limitations of lab equipment and health hazard of being in an MRI radiation environment to obtain good quality scans lead to lower quality scans and also it takes a lot of time to get a high-resolution data. This problem can be solved by using super-resolution using deep learning as a post-processing step to improve the resolution of the scans. Super-resolution is a process of generating higher resolution images from lower resolution data. For this, we are proposing a generative adversarial network architecture which is a dual neural network designed to generate lifelike images. In this deep learning algorithm, two neural networks compete with each other to improve alternatively. Given a training set, this technique learns to generate new data with the same statistics as the training set. To apply this technique to our problem statement we are using generator as the network to improve the resolution and discriminator as a network to train generator better. We used transfer learning in our generative neural network and training our discriminator from scratch and using the perceptual loss [1] to train our network. This will help in improving the performance of the network. We are using Lung MRI scans of tuberculosis with a set of 216 MRI samples containing around 60-130 channels each and each channel having 512x512 dimensions.

The high-resolution magnetic resonance image (MRI) provides detailed anatomical information critical for clinical application diagnosis. However, high-resolution MRI typically comes at the cost of long scan time, small spatial coverage, and low signal-to-noise ratio. The benefits of the convolutional neural network (CNN) can be applied to solve the super-resolution task to recover high-resolution generic images from low-resolution inputs. Additionally, recent studies have shown the potential to use the generative advertising network (GAN) to generate high-quality super-resolution MRIs using learned image priors. Moreover, existing approaches require paired MRI images as training data, which is difficult to obtain with existing datasets when the alignment between high and low-resolution images has to be implemented manually.This paper implements two different GAN-based models to handle the super-resolution: Enhanced super-resolution GAN (ESRGAN) and CycleGAN. Different from the generic model, the architecture of CycleGAN is modified to solve the super-resolution on unpaired MRI data, and the ESRGAN is implemented as a reference to compare GAN-based methods performance. The results of GAN-based models provide generated high-resolution images with rich textures compared to the ground-truth. Moreover, results from experiments are performed on both 3T and 7T MRI images in recovering different scales of resolution.

The diagnosis of brain pathologies usually involves imaging to analyze the condition of the brain. Magnetic resonance imaging (MRI) technology is widely used in brain disorder diagnosis. The image quality of MRI depends on the magnetostatic field strength and scanning time. Scanners with lower field strengths have the disadvantages of a low resolution and high imaging cost, and scanning takes a long time. The traditional super-resolution reconstruction method based on MRI generally states an optimization problem in terms of prior information. It solves the problem using an iterative approach with a large time cost. Many methods based on deep learning have emerged to replace traditional methods. MRI super-resolution technology based on deep learning can effectively improve MRI resolution through a three-dimensional convolutional neural network; however, the training costs are relatively high. In this paper, we propose the use of two-dimensional super-resolution technology for the super-resolution reconstruction of MRI images. In the first reconstruction, we choose a scale factor of 2 and simulate half the volume of MRI slices as input. We utilize a receiving field block enhanced super-resolution generative adversarial network (RFB-ESRGAN), which is superior to other super-resolution technologies in terms of texture and frequency information. We then rebuild the super-resolution reconstructed slices in the MRI. In the second reconstruction, the image after the first reconstruction is composed of only half of the slices, and there are still missing values. In our previous work, we adopted the traditional interpolation method, and there was still a gap in the visual effect of the reconstructed images. Therefore, we propose a noise-based super-resolution network (nESRGAN). The noise addition to the network can provide additional texture restoration possibilities. We use nESRGAN to further restore MRI resolution and high-frequency information. Finally, we achieve the 3D reconstruction of brain MRI images through two super-resolution reconstructions. Our proposed method is superior to 3D super-resolution technology based on deep learning in terms of perception range and image quality evaluation standards.

In magnetic resonance imaging (MRI), reducing long scan times is an urgent issue that could be addressed with super-resolution (SR) techniques. Most of the SR networks using deep …

… an innovative GCR-GAN model for super-resolution in stroke disease classification. Here, the GCR-GAN is developed by modifying the Generative Adversarial Network (GAN) by …

AIMS To assess the feasibility of using a generative adversarial network (GAN) to improve diffusion-weighted imaging (DWI) resolution in rectal MR scans for rectal carcinoma (RC), and to evaluate both the image quality and the diagnostic utility of super-resolution DWI (SR-DWI) in T stage assessment. MATERIALS AND METHODS In this retrospective investigation, a total of 291 patients diagnosed with RC during the period spanning May 2018 to December 2021 were included. The generated SR-DWI was evaluated against the original DWI using multi-scale structural similarity and peak signal-to-noise ratio. Two radiologists scored the SR-DWI and original DWI using a 4-point Likert scale in image quality. Moreover, both radiologists independently evaluated the T category staging based on T2WI and SR-DWI. Interobserver agreement was assessed using Cohen's kappa. RESULTS The PSRN and MS-SSIM values of SR-DWI (4 ×) were significantly higher compared to those of SR-DWI (16 ×). Regarding the details of anatomic structures and overall image quality parameters, both radiologists exhibited a preference for SR DWI with 16 × enlargement over SR DWI with 4 × enlargement, yielding significantly superior ratings (both p < 0.001). The T-staging accuracy rates of SR-DWI (16 ×) performed by radiologist 1 and radiologist 2 were significantly superior to those achieved with T2WI (0.621 vs. 0.768, p = 0.027; 0.653 vs 0.810, p = 0.014). CONCLUSIONS Our study demonstrates that the adapted super-resolution approach can significantly improve the overall image quality and details of anatomic structure of DWI in rectal MR. And SR-DWI offer better diagnostic accuracy in RC T staging when compared with T2WI.

Magnetic Resonance Imaging (MRI) is a critical non-invasive imaging technique widely used in clinical diagnosis. However, the inherent limitations of acquisition time and hardware often restrict the spatial resolution of MRI, potentially impacting diagnostic accuracy. Super-resolution (SR) techniques have emerged as a promising solution to enhance image quality by reconstructing high-resolution (HR) images from low-resolution (LR) inputs. Among various deep learning approaches, Generative Adversarial Networks (GANs) have demonstrated exceptional capabilities in learning complex data distributions and generating high-quality images. This paper presents a comprehensive review of GAN-based super-resolution methods applied to MRI. It systematically analyzes recent architectures, training strategies and evaluation metrics, while discussing critical challenges such as limited datasets, training instability and the need for clinical validation. Furthermore, it highlights the current research gaps and proposes future directions to enhance the integration of GAN-based SR techniques into routine medical imaging workflows.

… magnetic resonance images (MRI) and thereby accelerate … the forward model of accelerated MRI reconstruction in the … We aim to embed data consistency (DC) in deep networks …

Brain magnetic resonance imaging (MRI) offers intricate soft tissue contrasts that are essential for diagnosing diseases and conducting neuroscience research. At 7 Tesla (7T) magnetic field intensity, MRI enables increased resolution, enhanced tissue contrast, and improved SNR, compared to MRI collected from the commonly employed 3 Tesla (3T) MRI scanners. However, the exorbitant expenses associated with 7T MRI scanners hinder their broad use in research and clinical facilities. Efforts are underway to develop algorithms that can generate 7T MRI from 3T MRI to achieve better image quality without the need for 7T MRI machines. In this study, we have adopted a cycle consistent generative adversarial network (CycleGAN)-based approach for 3T MRI to 7T MRI translation, and vice versa, using a recently published dataset of paired T1-weighted MR images collected at 3T and 7T from a total of ten subjects. Various CycleGAN architectures were experimented with and compared on this dataset. The best performing CycleGAN architecture successfully produced the reconstructed images with a high level of accuracy based on different quantitative and qualitative evaluation criteria. Utilizing a post-processing technique, the best performing model generated 7T MRI from 3T MRI with a structural similarity index measure (SSIM) of 83.80%, peak SNR (PSNR) of 26.25, normalized mean squared error (NMSE) of 0.0088 and normalized mean absolute error (NMAE) of 0.0630. Utilizing CycleGAN to convert images from 3T to 7T MRI has shown a substantial improvement in MRI resolution, setting the stage for advancements in more informative and precise diagnostic imaging.

… is further improved by gradually improving the translated MRI images combined with the MRN generator architecture. The generator learn more detailed regional structure and details of …

… MRI, this paper introduces the modal label into pix2pix, so that a single modal T2 can synthesize the other three target modalities through a single-generation … reconstruction consistency …

This study compares volumetric measurements of various brain regions using different magnetic resonance imaging (MRI) modalities and deep learning models, specifically 3T MRI, ultra-low field (ULF) MRI at 64mT, and AI-enhanced ULF MRI using SynthSR and HiLoResGAN. The aim is to evaluate the alignment and agreement among field strengths and ULF MRI with and without AI. Descriptive statistics, paired t-tests, effect size analyses, and regression analyses are employed to assess the relationships and differences between modalities. The results indicate that volumetric measurements derived from 64mT MRI deviate significantly from those obtained using 3T MRI. By leveraging SynthSR and LoHiResGAN models, these deviations are reduced, bringing the volumetric estimates closer to those obtained from 3T MRI, which serves as the reference standard for brain volume quantification. These findings highlight that deep learning models can reduce systematic differences in brain volume measurements across field strengths, providing potential solutions to minimize bias in imaging studies.

… MRI physics to incorporate data consistency during the training of a conditional diffusion probabilistic model, which we refer to as the data consistency-… enhanced T1W MRI from partially …

Various harmonization methods have been employed for obtaining MRI from different scanners. However, no study has yet focused on the clinical utility of the CycleGAN technique in reducing MRI interscanner variability for patients with brain metastasis across longitudinal visits. We developed a head-to-head and longitudinal CycleGAN-based deep learning (DL) algorithm for MRI harmonization and validated its utility for follow-up (FU) MRI evaluation in patients with unchanged brain metastasis, who had FU MRI taken using a different MRI scanner. We trained the head-to-head and longitudinal CycleGAN to generate harmonized second postcontrast 3D T1W MR images with similar image impressions as the initial postcontrast 3D T1W MR images. The image similarity scores between the baseline (BL) and harmonized FU images were higher than those between the baseline and original FU images. As compared with baseline, differences in the CNRs of brain subregions were lower for the harmonized FU images than for the original FU images. More cases were read to be unchanged on the harmonized FU images than on the original FU images in terms of border, size, and contrast enhancement at a higher level of diagnostic confidence. The proposed CycleGAN algorithm may potentially decrease false positivity for the diagnosis of progression in FU MRI evaluation of brain metastasis. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-026-43755-7.

Purpose: The common practice in acquiring the magnetic resonance (MR) images is to obtain two-dimensional (2D) slices at coarse locations while keeping the high in-plane resolution in order to ensure enough body coverage while shortening the MR scan time.The aim of this study is to propose a novel method to generate HR MR images from low-resolution MR images along the longitudinal direction. In order to address the difficulty of collecting paired low- and high-resolution MR images in clinical settings and to gain the advantage of parallel cycle consistent generative adversarial networks (CycleGANs) in synthesizing realistic medical images, we developed a parallel CycleGANs based method using a self-supervised strategy. Methods and materials: The proposed workflow consists of two parallely trained CycleGANs to independently predict the HR MR images in the two planes along the directions that are orthogonal to the longitudinal MR scan direction. Then, the final synthetic HR MR images are generated by fusing the two predicted images. MR images, including T1-weighted (T1), contrast enhanced T1-weighted (T1CE), T2-weighted (T2), and T2 Fluid Attenuated Inversion Recovery (FLAIR), of the multimodal brain tumor segmentation challenge 2020 (BraTS2020) dataset were processed to evaluate the proposed workflow along the cranial–caudal (CC), lateral, and anterior–posterior directions. Institutional collected MR images were also processed for evaluation of the proposed method. The performance of the proposed method was investigated via both qualitative and quantitative evaluations. Metrics of normalized mean absolute error (NMAE), peak signal-to-noise ratio (PSNR), edge keeping index (EKI), structural similarity index measurement (SSIM), information fidelity criterion (IFC), and visual information fidelity in pixel domain (VIFP) were calculated. Results: It is shown that the proposed method can generate HR MR images visually indistinguishable from the ground truth in the investigations on the BraTS2020 dataset. In addition, the intensity profiles, difference images and SSIM maps can also confirm the feasibility of the proposed method for synthesizing HR MR images. Quantitative evaluations on the BraTS2020 dataset shows that the calculated metrics of synthetic HR MR images can all be enhanced for the T1, T1CE, T2, and FLAIR images. The enhancements in the numerical metrics over the low-resolution and bi-cubic interpolated MR images, as well as those genearted with a comparative deep learning method, are statistically significant. Qualitative evaluation of the synthetic HR MR images of the clinical collected dataset could also confirm the feasibility of the proposed method. Conclusions: The proposed method is feasible to synthesize HR MR images using self-supervised parallel CycleGANs, which can be expected to shorten MR acquisition time in clinical practices.

This research presents the concept of using diffusion probabilistic model and transformer-based architecture as the framework of synthetic MRI generation to provide assistance for the neuro-imaging workflows acceleration. We train Stable Diffusion XL with a controlnet (ControlNet). We are able to synthesize high quality brain MRI sequence from k-space undersampled data. The proposed framework combines Score-based Generative Models (SGM) with physical techniques using the proposed neural networks with 8x acceleration without any loss in diagnostic-quality. Finally, we utilize BraTS2024 dataset for training and perceptual loss functions are used along with StyleGAN3 discriminators with adversarial training. Quantitative evaluation with SSIM (0.94), PSNR (38.2dB) and FID score (12.3), which was better than the results obtained in the traditional compressed sensing methods. The framework offers a gain ranging from 45 to 6 min scan time and enables a fast clinical implementation without anatomical loss and pathological features, which are crucial for correct diagnosis.

Multi-modal neuroimages (e.g., MRI and PET) have been widely used for diagnosis of brain diseases such as Alzheimer’s disease (AD) by providing complementary information. However, in practice, it is unavoidable to have missing data, i.e., missing PET data for many subjects in the ADNI dataset. A straightforward strategy to tackle this challenge is to simply discard subjects with missing PET, but this will significantly reduce the number of training subjects for learning reliable diagnostic models. On the other hand, since different modalities (i.e., MRI and PET) were acquired from the same subject, there often exist underlying relevance between different modalities. Accordingly, we propose a two-stage deep learning framework for AD diagnosis using both MRI and PET data. Specifically, in the first stage, we impute missing PET data based on their corresponding MRI data by using 3D Cycle-consistent Generative Adversarial Networks (3D-cGAN) to capture their underlying relationship. In the second stage, with the complete MRI and PET (i.e., after imputation for the case of missing PET), we develop a deep multi-instance neural network for AD diagnosis and also mild cognitive impairment (MCI) conversion prediction. Experimental results on subjects from ADNI demonstrate that our synthesized PET images with 3D-cGAN are reasonable, and also our two-stage deep learning method outperforms the state-of-the-art methods in AD diagnosis.

… generation … MRI dataset demonstrate that, compared with a baseline diffusion model without structural priors, the proposed method achieves consistent improvements in generation …

Brain tumor diagnostics rely heavily on Magnetic Resonance Imaging (MRI) for accurate diagnosis and treatment planning due to its non-invasive nature and detailed soft tissue visualization. Integrating multiple MRI modalities enhances diagnostic precision by providing complementary perspectives on tumor characteristics and spatial relationships. However, acquiring specific modalities like T1 Contrast Enhanced (T1CE) can be challenging, as they require contrast agents and longer scan times, which can cause discomfort, particularly in vulnerable patient groups such as the elderly, pregnant women, and infants. In the medical imaging domain, researchers face significant challenges in developing robust models due to data scarcity and data sparsity. Data scarcity, arising from limited access to diverse datasets, complex annotation processes, privacy concerns, and the difficulty of acquiring certain modalities in some patient groups, impedes the development of comprehensive brain tumor segmentation models. Data sparsity, driven by the highly imbalanced distribution between tumor subregions and background levels in annotated labels, complicates accurate segmentation. The study addresses these challenges by generating synthetic T1CE scans from T1 using an image-to-image translation framework, thereby reducing the reliance on hard-to-acquire modalities. A novel patch-based data sampling approach, Adaptive Random Patch Selection (ARPS), is introduced to combat data sparsity, ensuring detailed segmentation of intricate tumor structures while maintaining context through overlapping patches and context-aware sampling strategies. The impact of these synthetic images on segmentation performance is also assessed, emphasizing their role in addressing situations where certain modalities cannot be acquired. When integrated into the nnUNet model, this approach achieves a dice similarity coefficient (DSC) of 86.47, demonstrating its efficacy in handling complex MRI scans of brain tumors. An ablation study is also conducted to assess the individual contributions of the translated images and the proposed data sampling approach. This comprehensive evaluation allows us to understand the effectiveness of ARPS and the potential synergy between multi-modal translation and brain tumor segmentation.

Multi-modal images play a crucial role in comprehensive evaluations in medical image analysis providing complementary information for identifying clinically important biomarkers. However, in clinical practice, acquiring multiple modalities can be challenging due to reasons such as scan cost, limited scan time, and safety considerations. In this paper, we propose a model based on the latent diffusion model (LDM) that leverages switchable blocks for image-to-image translation in 3D medical images without patch cropping. The 3D LDM combined with conditioning using the target modality allows generating high-quality target modality in 3D overcoming the shortcoming of the missing out-of-slice information in 2D generation methods. The switchable block, noted as multiple switchable spatially adaptive normalization (MS-SPADE), dynamically transforms source latents to the desired style of the target latents to help with the diffusion process. The MS-SPADE block allows us to have one single model to tackle many translation tasks of one source modality to various targets removing the need for many translation models for different scenarios. Our model exhibited successful image synthesis across different source-target modality scenarios and surpassed other models in quantitative evaluations tested on multi-modal brain magnetic resonance imaging datasets of four different modalities and an independent IXI dataset. Our model demonstrated successful image synthesis across various modalities even allowing for one-to-many modality translations. Furthermore, it outperformed other one-to-one translation models in quantitative evaluations. Our code is available at https://github.com/jongdory/ALDM/

… the lack of labelled Magnetic Resonance Imaging (MRI) data … (GANs) to generate synthetic MRI images. We present a dual-… a high degree of similarity to real MRI scans. These findings …

Generative modeling has seen significant advancements in recent years, especially in the realm of text-to-image synthesis. Despite this progress, the medical field has yet to fully leverage the capabilities of large-scale foundational models for synthetic data generation. This paper introduces a framework for text-conditional magnetic resonance (MR) imaging generation, addressing the complexities associated with multi-modality considerations. The framework comprises a pre-trained large language model, a diffusion-based prompt-conditional image generation architecture, and an additional denoising network for input structural binary masks. Experimental results demonstrate that the proposed framework is capable of generating realistic, high-resolution, and high-fidelity multi-modal MR images that align with medical language text prompts. Further, the study interprets the cross-attention maps of the generated results based on text-conditional statements. The contributions of this research lay a robust foundation for future studies in text-conditional medical image generation and hold significant promise for accelerating advancements in medical imaging research.

BACKGROUND AND OBJECTIVE Multi-modal medical images, such as magnetic resonance imaging (MRI) and positron emission tomography (PET), have been widely used for the diagnosis of brain disorder diseases like Alzheimer's disease (AD) since they can provide various information. PET scans can detect cellular changes in organs and tissues earlier than MRI. Unlike MRI, PET data is difficult to acquire due to cost, radiation, or other limitations. Moreover, PET data is missing for many subjects in the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. To solve this problem, a 3D end-to-end generative adversarial network (named BPGAN) is proposed to synthesize brain PET from MRI scans, which can be used as a potential data completion scheme for multi-modal medical image research. METHODS We propose BPGAN, which learns an end-to-end mapping function to transform the input MRI scans to their underlying PET scans. First, we design a 3D multiple convolution U-Net (MCU) generator architecture to improve the visual quality of synthetic results while preserving the diverse brain structures of different subjects. By further employing a 3D gradient profile (GP) loss and structural similarity index measure (SSIM) loss, the synthetic PET scans have higher-similarity to the ground truth. In this study, we explore alternative data partitioning ways to study their impact on the performance of the proposed method in different medical scenarios. RESULTS We conduct experiments on a publicly available ADNI database. The proposed BPGAN is evaluated by mean absolute error (MAE), peak-signal-to-noise-ratio (PSNR) and SSIM, superior to other compared models in these quantitative evaluation metrics. Qualitative evaluations also validate the effectiveness of our approach. Additionally, combined with MRI and our synthetic PET scans, the accuracies of multi-class AD diagnosis on dataset-A and dataset-B are 85.00% and 56.47%, which have been improved by about 1% and 1%, respectively, compared to the stand-alone MRI. CONCLUSIONS The experimental results of quantitative measures, qualitative displays, and classification evaluation demonstrate that the synthetic PET images by BPGAN are reasonable and high-quality, which provide complementary information to improve the performance of AD diagnosis. This work provides a valuable reference for multi-modal medical image analysis.

… Automatic generation of synthetic images has a wide range … Here we focus on Magnetic Resonance Imaging (MRI), a non-… Another key problem in MRI synthesis is that many different …

Magnetic Resonance Imaging (MRI) is a widely used, non-invasive imaging technology that plays a critical role in clinical diagnostics. Multi-modal MRI, which combines images from different modalities, enhances diagnostic accuracy by offering comprehensive tissue characterization. Meanwhile, multi-modal MRI enhances downstream tasks, like brain tumor segmentation and image reconstruction, by providing richer features. While recent advances in diffusion models (DMs) show potential for high-quality image translation, existing methods still struggle to preserve fine structural details and ensure accurate image synthesis in medical imaging. To address these challenges, we propose a Frequency-Aware Diffusion Model (FADM) for generating high-quality target modality MRI images from source modality images. The FADM incorporates a discrete wavelet transform within the diffusion model framework to extract both low- and high-frequency information from MRI images, enhancing the capture of tissue structural and textural features. Additionally, a wavelet downsampling layer and supervision module are incorporated to improve frequency awareness and optimize high-frequency detail extraction. Experimental results on the BraTS 2021 dataset and a 1.5T–3T MRI dataset demonstrate that the FADM outperforms existing generative models, particularly in preserving intricate brain structures and tumor regions while generating high-quality MRI images.

Multimodality is often necessary for improving object segmentation tasks, especially in the case of multilabel tasks, such as tumor segmentation, which is crucial for clinical diagnosis and treatment planning. However, a major challenge in utilizing multimodality with deep learning remains: the limited availability of annotated training data, primarily due to the time-consuming acquisition process and the necessity for expert annotations. Although deep learning has significantly advanced many tasks in medical imaging, conventional augmentation techniques are often insufficient due to the inherent complexity of volumetric medical data. To address this problem, we propose an innovative slice-based latent diffusion architecture for the generation of 3D multi-modal images and their corresponding multi-label masks. Our approach enables the simultaneous generation of the image and mask in a slice-by-slice fashion, leveraging a positional encoding and a Latent Aggregation module to maintain spatial coherence and capture slice sequentiality. This method effectively reduces the computational complexity and memory demands typically associated with diffusion models. Additionally, we condition our architecture on tumor characteristics to generate a diverse array of tumor variations and enhance texture using a refining module that acts like a super-resolution mechanism, mitigating the inherent blurriness caused by data scarcity in the autoencoder. We evaluate the effectiveness of our synthesized volumes using the BRATS2021 dataset to segment the tumor with three tissue labels and compare them with other state-of-the-art diffusion models through a downstream segmentation task, demonstrating the superior performance and efficiency of our method. While our primary application is tumor segmentation, this method can be readily adapted to other modalities. Code is available here : https://github.com/Arksyd96/multi-modal-mri-and-mask-synthesis-with-conditional-slice-based-ldm.