Super-Resolution of PET Images using A Convolutional Network with Skip Connections

Super-Resolution of PET Images using A Convolutional Network with Skip Connections

Jose Mejia, Leticia Ortega

Dept. of Electrical and Computation Engineering.

University autonomous of Ciudad Juarez Ciudad Juárez, Mexico

AbstractPositron emission tomography is a medical imaging technique used mainly to help diagnose and monitor cancer treatment. However, the resolution of these images is very low compared to other modalities such as computed tomography and magnetic resonance imaging. The objective of this research is to build a system based on Deep Learning to increase the resolution in positron emission tomography images. The algorithm is based on convolutional networks for super-resolution, and it is proposed to use a loss function with quadratic error. To demonstrate its performance, the proposed algorithm is compared with spline interpolation using different images. The results obtained show better performance of the proposed algorithm based on the metrics of the mean square error and the signal-to-noise ratio.

KeywordsPET; super-resolution; deep learning; (key words)

1. INTRODUCTION

   Positron emission tomography, or PET, is a medical imaging technique that involves using small quantities of radioactive material to create high-resolution images of the internal structures and functions of the body [1]. In PET studies, a patient is injected with a small amount of a radioactive tracer and subsequently the tracer undergoes transportation through the circulatory system to targeted regions of the body, generally neoplastic lesions or regions of inflammation [2].

   Spatial resolution in positron emission tomography (PET) is limited by the hardware used by the scanner, for example the physical size of the sensors is a dominant factor in the resolution that can be achieved, in addition, other influencing factors are the non-collinearity in the trajectories of the particles, the range of the distance traveled by the positrons, the penetration in the detector ring and the detection errors [3] [4]. The poor resolution of PET cameras has been observed as a limitation in the clinical setting. The need for an improvement in PET resolution. Improved spatial resolution and lesion detectability could be beneficial for guiding the biopsy and reducing staging and treatment [5]. The increment in spatial resolution also could be greatly beneficial for early diagnosis and could provide strong support for surgery.

   Image super-resolution is a technique used to increase the resolution of an image. It involves using algorithms to add more pixels to an image, thereby increasing its overall resolution and making it appear sharper and more detailed. This can be used to improve the quality of low-resolution images, or to create high-resolution versions of images that were originally captured at a lower resolution. In medical

   imaging, image super-resolution can be used to enhance the quality of low-resolution medical images, such as ultrasound or MRI images. This can improve the diagnostic capabilities of the images and make it easier for doctors to detect and diagnose certain conditions. For example, in ultrasound imaging, super-resolution can be used to improve the resolution of images of fetuses, making it easier to detect and diagnose abnormalities. In PET, super-resolution can be used to improve the resolution of images of internal organs, making it easier to detect and diagnose tumors or other abnormal growths. Additionally, super-resolution can be used to obtain high-resolution images from low-resolution images which can save cost and time in medical imaging. Image super-resolution can be used to enhance the resolution of PET images, which can improve the diagnostic capabilities of the images. This can make it easier for doctors to detect and diagnose certain conditions. In particular, super-resolution can be used to improve the resolution of PET images of small structures, such as tumors, which can be difficult to detect with traditional imaging methods. Super-resolution can also be used to improve the resolution of images of internal organs, making it easier to detect and diagnose abnormalities.

   The specific method of applying super-resolution technique on PET images depends on the type of super-resolution algorithm used. Some methods involve using a training dataset to learn the relationship between low-resolution and high-resolution images and then applying that relationship to the low- resolution PET images to generate high-resolution images. Other methods involve using mathematical algorithms to add additional pixels to the low-resolution images.

   1. Previous work

      Super-resolution in PET images has been studied since its introduction, some of the most recent works are reviewed below. In [6] it is presented an image deblurring and super- resolution methodology for PET using anatomical guidance provided by high-resolution Magnetic Resonance images. The methodology relies on image-domain post-processing of already-reconstructed PET images by means of spatially variant deconvolution stabilized by a joint entropy penalty function.

      In [7] it is explored Gated Cardiac-PET super-resolution based on the Total Variation model. They used the frames available as priors for super-resolution algorithms, because of the high noise content and low resolution of those frames. A second imaging modality, such as Ultrasound is also used. They chose

      ultrasound images because they are a reference technique for cardiac imaging in humans. More recently deep learning algorithms have been used in the area of medical imaging, in particular for super-resolution most works have used several type of neural networks such as the work of [8] where a Single Image Super Resolution is proposed by using a convolutional neural network trained on PET images, they estimate a high resolution PET image from a unique low resolution image. The work of [9] presents an approach for super-resolution for PET images based on a deep learning network. Their proposed network is trained on images from the high-resolution research tomograph. In [10] A generative adversarial network which consists of two separate neural networks, a generator that creates synthetic data and a discriminator that evaluates them, is used and the two networks are individually pretrained and then jointly trained and validated. Their network uses anatomical information in the form of high-resolution magnetic resonance images and additional spatial information in the form of radial and axial location patches. They use simulated data for training and validation. In [11] it is also used a generative Adversarial Networks combined with Wasserstein distance and gradient penalty technology used to achieve PET image super-resolution.

      In this research, it is proposed to employ convolutional networks for single-image super-resolution. A streamlined architecture with minimized size is obtained, thereby enhancing overall performance in terms of computation time and the number of floating-point operations.

   2. Literature analysis

   In this section, an analysis of the state of the art is provided using the Lens platform (www.lens.org) and Google Scholar (GS). A search was conducted on GS with the keywords PET, super-resolution, using the AND operator as follows:

   PET AND super-resolution

   The search yields a total of 9,230 results, encompassing works from the year 2019 onwards. Fig. 1, displays the countries in darker tones that are more active, contrasted with those in lighter tones representing countries with fewer publictions.

   In Fig. 2, a word cloud image is presented, depicting the topics covered in the analyzed articles. "Computer science" emerges as the most commonly identified topic in the reviewed publications, followed by "artificial intelligence." It is noteworthy that these topics were identified using a topic recognition software program. Therefore, the super-resolution theme does not emerge prominently, possibly because it is not as prevalent, and the program might not have been extensively trained on data related to this specific subject.

   Fig. 1. Colored surface for PET image super-resolution. Darker tones represent countries with higher activity, while lighter tones represent countries with fewer publications. (www.lens.org).

   Fig. 2. Word cloud representing the topics covered in the analyzed articles.

   Fig. 3. Proportion of articles published in conferences or conferences and those published in scientific journals.

   Fig. 3 illustrates the proportion of articles published in conferences or congresses compared to those published in scientific journals. As observed, the vast majority are published in journals, with less than a quarter of the total articles appearing in conference proceedings.

2. METHODOLOGY

   In this section, the design of the Deep Learning-based architecture for obtaining high-resolution PET images is outlined. The aim is to create a deep learning network for the enhancement of low-resolution images, meaning that the network is intended to generate or reconstruct a high-resolution image from a low-resolution input, as depicted in Fig. 4.

   Fig. 4. Obtaining a high resolution image from a low resolution image.

   Fig. 5. Proposed architecture.

   For this purpose, the designed model comprises three 2D convolutional layers with 32, 30, and 52 filters, respectively, each with a 3×3 kernel size. Subsequently, there is a layer of transposed convolutional upscaling with 50 filters of size 20×20. Following this, there is a 2D convolutional layer with 30 filters of size 3×3, another transposed convolutional layer with 50 filters of size 21×21, a 2D convolutional layer with 75 filters of size 3×3, a transpose layer with 50 filters of size 10×10, two 2D convolutional layer with 70 and 5 filters of size 2×2, the output is added to a skip connection composed of a transpose layer of 5 filters of size 39×39. Finally, a layer of type convolutional with a single filter of size 2×2. ReLU activation functions were applied in all layers. The finalized design is depicted in Fig. 5.

   The model training utilized the Adam optimization algorithm. The neural network training process involved pairs of high and low-resolution image sets. Throughout this training procedure, a total of 200 epochs were conducted, with multiple iterations fine-tuning the network's hyperparameters to achieve an optimal configuration that minimizes mean squared error. Continuous adjustments were made to optimize the network's performance. The results obtained were evaluated, and the configuration demonstrating the best outcomes during the training process was identified.

3. RESULTS AND DISCUSSION

   To assess the proposed network, mean square error (MSE) and peak signal-to-noise ratio (PSNR) metrics were employed. Additionally, spline interpolation of order one was used as a comparative method. Fig. 6 displays the results of the designed network. The first column shows the original high-resolution images (the goal), the second column reveals the outcome of reconstructing a low-resolution image using the constructed

   network, and the third column displays the interpolation using splines for the low-resolution images.

   Table 1 presents quantitative performance metrics for evaluating the network, with metrics assessed using some images from the test set. The metrics utilized include mean squared error (MSE) and peak signal-to-noise ratio (PSNR). The proposed network exhibits lower errors, i.e., lower MSE and higher PSNR, as evident from the results in the table.

   A statistical analysis was conducted using box plots. In Fig. 7, box plots of various metrics are depicted. It is observed that images obtained by the designed network have a mean MSE of

   5.24 and a median of 5.07, in contrast to the interpolation method with an average MSE of 12.93, a median of 11.46, and a significantly higher deviation with the presence of outliers in regions with more significant errors. The behavior is similar in terms of PSNR, with the signal-to-noise ratio significantly higher in the designed network, averaging 41.03, compared to the interpolation method, which only reaches 37.47.

   Image	Proposed Network		Spline Interpolation
   	MSE	PSNR	MSE	PSNR
   Image 1	6.79	39.81	10.08	38.10
   Image 2	3.76	42.38	8.59	38.79
   Image 3	5.85	40.46	20.35	35.04
   Image 4	5.01	41.13	11.46	37.54
   Image 5	5.67	40.60	17.84	35.62
   Image 6	5.05	41.10	8.61	38.78
   Image 7	5.87	40.45	9.93	38.16
   Image 8	4.57	41.54	13.74	36.75
   Image 9	4.91	41.22	13.92	36.70
   Image 10	3.61	42.55	11.35	37.58
   Image 11	5.07	41.08	7.41	39.43
   Image 12	4.95	41.19	8.85	38.66
   Image 13	6.47	40.02	9.67	38.28
   Image 14	5.22	40.95	18.69	35.41
   Image 15	5.72	40.56	11.78	37.42
   Image 16	7.56	39.34	12.60	37.13
   Image 17	4.79	41.33	15.33	36.27
   Image 18	5.59	40.66	19.05	35.33
   Image 19	3.89	42.23	11.20	37.64
   Image 20	7.22	39.55	35.54	32.62
   Image 21	3.13	43.17	4.43	41.66
   Image 22	4.42	41.67	4.39	41.70
   Image 23	5.39	40.82	12.51	37.16

   TABLE I. METRIC RESULTS

   Fig. 6. First column: original high resolution image. Second column:

   Proposed network. Third column interpolation using splines.

   As part of our future work, we aim to extend the assessment by comparing our network against contemporary, state-of-the-art methodologies. Furthermore, we plan to enhance the network's generalization capabilities by augmenting the training dataset with a more extensive collection of images. This proactive approach is expected to contribute to the continued advancement of resolution enhancement techniques in the realm of positron emission tomography.

   (a) (b)

   (c) (d)

   Fig. 7. Evaluation of the metrics, (a) MSE of proposed architecture, (b) MSE interpolation, (c) PSNR of proposed architecture, and (d) PSNR interpolation.

4. CONCLUSIONS

In this study, we introduce a convolutional neural network based in deep learning principles to enhance the resolution of positron emission tomography images. The efficacy of the proposed network was assessed using low-resolution images, employing performance metrics, namely Mean Squared Error and Peak Signal-to-Noise Ratio. The results indicate superior performance compared to conventional techniques like spline interpolation.

REFERENCES

1. J. S. K. Gerd Muehllehner, «Positron emission tomography.,» Physics in Medicine & Biology, vol. 51, nº 13, 10.1088/0031-9155/51/13/R08, 2006.

2. I. e. a. PEÑUELAS, Positron emission tomography imaging of adenoviral-mediated transgene expression in liver cancer patients, Gastroenterology, vol. 128, nº 7, 2005.

3. W. W. MOSES, Fundamental limits of spatial resolution in PET, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, , vol. 648, nº https://doi.org/10.1016/j.nima.2010.11.092, pp. 236-240, 2011.

4. K. GONG y S. R. CHERRY, «On the assessment of spatial resolution of PET systems with iterative image reconstruction.,» Physics in Medicine & Biology, vol. 61, nº 10.1088/0031-9155/61/5/N193, 2016.

5. BAL, H., et al. Improving PET spatial resolution and detectability for prostate cancer imaging, Physics in Medicine & Biology, vol. 59, nº no 15,, p. 4411, 2014.

6. T.-A. e. a. SONG, PET image deblurring and super-resolution with an MR-based joint entropy prior, IEEE transactions on computational imaging, vol. 5, nº 4, 2019.

7. M. e. a. PEREZ-LIVA, Ultrafast ultrasound imaging for super- resolution preclinical cardiac PET, . Molecular imaging and biology, vol. 22, nº 1342-1352, 2020.

8. Garehdaghi, F., Meshgini, S., Afrouzian, R., & Farzamnia, A, PET image super resolution using convolutional neural networks, In 2019 5th Iranian Conference on Signal Processing and Intelligent Systems, pp. 1-5, 2019.

9. Ren, Sijin, et al., Super-resolution PET Brain Imaging using Deep Learning, IEEE Nuclear Science Symposium and Medical Imaging Conference, pp. 1-6, 2021.

10. Song, T. A., Chowdhury, S. R., El Fakhri, G., Li, Q., & Dutta, J., Super-resolution PET imaging using a generative adversarial network, 2019.

11. Liu, M., Feng, Y., Liu, C., Gao, M., & Yan, C, Super resolution reconstruction of PET images based on deep learning, Academic Journal of Computing & Information Science, vol. 4, nº 4, pp. 48-52,

2021.