A Novel Method of Multimodal Medical Image Fusion Based on Hybrid Approach of NSCT and DTCWT

nawar.alseelawi@uomisan.edu.iq Abstract— The approach of multimodal medical image fusion, which extracts complementary information from several multimodality medical pictures, is one of the most significant and beneficial illness study tools. This work proposed an effective strategy for multimodal medical picture fusion based on a hybrid approach of NSCT and DTCWT. The experimental study’s input multimodality medical images included computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET). A suggested approach employs a convolutional network to generate a weight map that incorporates pixel movement information from dual or more multimodality medical pictures. To provide greater visual comprehension by humans, the medical picture fusion method is performed on a multiscale basis using medical image pyramids. Addi-tionally, a local comparison-based method is employed to adaptively alter the fusion mode for the decomposed coefficients. The proposed fusion methodolo gies result in the highest-quality fused multimodal medical pictures, the lowest processing period, and the finest visualization in terms of visual quality and objective assessment


Introduction
Doctors (radiologists) in the medical field require a high level of spatial and spectral information in a particular image for a variety of reasons, including research, monitoring, correct sickness diagnosis, and therapy. This type of data can be gotten by means of a single modality image because computed tomography (CT) images are frequently used to depict bone structures but deficiency of data about magnetic resonance imaging (MRI) images and soft tissues offer soft tissue information [1,2]. The deficient boundary information, in addition to positron emission tomography (PET) images offer soft tissue information with deficient boundary information. Complementary evidence from other modalities is required to address this issue. In this case, fusion stands for a technique for merging multimodality medical imaging, such as CT, MRI, and PET. Image fusion stands for a technique for merging applicable data from two or more photographs to create a single fused image that contains more novel information than any of the input photographs [3,4]. Satellite and Aerial imaging, robot vision, medical imaging and multi-focus image fusion are just a few of the applications that make use of this fused image data. Image fusion is the process of combining two or more separate photos using specialized techniques to create a new image [5][6][7]. It is the process of extracting data from images obtained from a variety of sources. A spatial resolution of creative multi-spectral image is increased while the spectral information is retained. The three layers of picture fusion are pixel level fusion, feature level fusion, and decision level fusion. Due to pixel-level fusion, a significant percentage of the incredible data is kept in the composite image. The sources of distinct features, such as edges and textures, are taken into account during feature-level fusion. The process of integrating a final choice is referred to as decision-level fusion. Image fusion can take one of two kinds. Fusion of single sensor and multisensory images combines the images from a few sensors to generate a composite image, and their individual images are joined to create an interwoven image. For instance, multi-focus and multi-exposure fusion. In multi sensor image fusion, the images from many sensors are combined to create a composite image, and their individual images are blended to create a fused image. For instance, medical imaging or the military [8]. Image fusion is used in a variety of applications, including biometrics, medical imaging, machine vision, automatic change detection, military applications, navigation aids, digital imaging, aerial, remote sensing and satellite imaging. Multimodal medical imaging is employed in a variety of clinical settings, including diagnosis and treatment. In medical image fusion, multiple images are combined to create a single unified modality. Medical image fusion techniques make use of computer vision, image processing, machine learning, pattern recognition and artificial intelligence [9,10]. CT, Magnetic Resonance Imaging (MRI), Magnetic Resonance Angiography (MRA), Ultrasonography (USG), Positron Emission Tomography (PET), Single Photon Emission Tomography Computed Tomography, Nuclear Magnetic Resonance (NMR) spectroscopy and magnetic resonance imaging are examples of multimodality medical imaging (SPECT) Structure-based therapeutic imaging modalities such as MRI, CT, USG, and MRA generate high-resolution images. Functional therapeutic imaging modalities such as PET, SPECT, and functional magnetic resonance imaging (fMRI) produce low-resolution images with functional information. To gain a more constructive understanding of a particular object, anatomical and functional therapeutic images may be integrated. By storing a single fused image rather than a collection of input images, medical image fusion reduces storage expenses [11]. Different imaging techniques offer restricted information. A computed tomography (CT) scan image can reveal detailed bone structures. A soft tissue MRI scan may detect both normal and sick soft tissues. By merging complementary information from CT and MRI scans, it is possible to eliminate duplication and improve diagnosis accuracy [12][13][14]. PET/MRI imaging paired with structural information can be used for clinical diagnosis and therapy [15,16]. Fusion at the pixel level is employed in multimodal medical image fusion [17].
Some common considerations should be considered while utilizing the vision merging approach.
• The fusion method should not throw away any data from the source images.
• Any artefacts or inconsistencies introduced by the fusion method should not divert or mislead a human viewer or any later vision processing stages. • A fusion method must be consistent, robust, and capable of tolerating defects such as noise or missed registrations to the greatest extent feasible. The various sorts of medical images are depicted in Figure 1. Numerous medical scans are available in the medical sector to accurately diagnose a patient's ailment. Medical scans include CT scans, MRI scans, PET scans, and SPECT scans.

Literature review
The needed data from the provided supply pictures is fused together in the Image Fusion technique to create a composite image with a higher quality than the offered feedback images. The methods for combining images may be divided into dual categories, i.e.

Spatial Domain Fusion 2. Transform Domain Fusion
The pixel price of an image is directly dealt with in spatial domain activities. To get the desired effect, the pixel values are changed. Every region where pictures must be evaluated necessitates the use of fusion. Medical image analysis, microscopic imaging, satellite image analysis, remote sensing requests, computer vision, and combat monitoring are just a few examples. The customizable quality analysis tool is being used to do an examination of remote sensing photos. The distinct wavelet convert is one of the most common fusion methods. When compared to other spatial types, strategies similar to these demonstrate enhanced benefits in spatial and spectral quality of a combined image.

Principal component analyses (PCA)
PCA is only an arithmetical tool for converting correlated variables to uncorrelated ones. PCA is used extensively for image categorization and picture pressure. There is a mathematical formula for changing elements, which may be referred to as essential components. It generates a concise and optimal explanation for the data set. Every succeeding aspect based on the remainder of the variance, with the first key aspect corresponding to the greatest amount of difference possible in the data. The first important consideration is made in a direction of maximum variance. The second essential feature is restricted to laying in the subspace at a 90-degree angle to the first. This feature specifies the direction of optimal variance inside this Subspace. The next important factor is considered in the subspace's best difference direction, which is at a 90-degree angle to the previous two [18]. Following Figure 2 shows us the image fusion using PCA.

Discrete cosine transform (DCT)
All of the image fusion algorithms are highly complicated and time-consuming, making them difficult to utilize in real-time applications. As the feedback pictures are numbered and the combined photographs are recovered in JPEG standard, the fusion techniques employed in the DCT domain are highly successful. To use the JPEG development, first split a picture (in color or grey scales) into 8×8 pixel blocks. After some time, the Discrete Cosine Transform (DCT) is applied to each block, resulting in a set of 64 coefficients that are quantized to reduce their magnitude. Prior to entropy encoding, the coefficients are converted into a 1D variety in a crisscross pattern. The compression is accomplished in two steps: the first is quantization, and the second is the entropy generation process. The DCT fusion technique is used to reduce the issues that occur during the fusion of real-time programs and to improve the quality of the combined image. For the experience evaluation, a comparison qualification of 88 blocks calculated using DCT coefficients is employed [19][20][21].

Discrete wavelet transform (DWT)
The filters are built specifically for discrete wavelet transformation (DWT) decomposition to ensure that succeeding layers of the chart include only features that are not currently available at the preceding levels. The decomposition of the DWT signal is based on a series of low-pass and high-pass filters, as well as a subsampling mechanism. The 2D-DWT components are comprised of four photographs with dimensions equal to half the original image's size [22,23]. According to T. Zaveri et al. [24], image fusion stands for the method of merging manifold input images of the identical scene into a particular fused image that has critical information and extracts critical features from each of the original images, thereby improving its suitability for human and machine perception. This article describes a new approach to image fusion that outperforms pixel-based image fusion. At the pixel level, image fusion techniques are influenced through the blurring influence, which has a direct effect on the image's contrast. As a result, the study presents a region-based technique that is more resistant to noise, has a higher contrast, and is less affected by misregistration. The proposed technique is applied to a great amount of registered images, and the results are compared against a reference image. The proposed methodology outperforms previous methods due to its lower susceptibility to noise. Yijian Pei et al. described an enhanced discrete wavelet framework-based image fusion algorithm after exploring the ideas and features of the discrete wavelet framework [25]. The augmentation is based on the high frequency subband characteristic of the image region. When compared to the weighted average technique, the process based on wavelet transforms creates less noise. If the algorithm is correctly synthesized, the multi sensor will yield useful information about each source image. Multi-focus image fusion produces more precise and reliable results. As a result, this technique can significantly reduce data size while simultaneously increasing observers' target detection and situation estimation. The projected technique successfully integrates the properties and information included in each image, demonstrating the wavelet's potential for image fusion. The proposed algorithm's efficacy in image fusion may be demonstrated by multi-focus image fusion studies and medical image fusion experiments. This paper explained how to evaluate the quality of picture fusion and conducted an objective analysis of algorithm performance. The proposed strategy corrects the quality synthetically by merging subjective and objective assessment methodologies. The assessment results suggest that the projected algorithm is capable of better merging the information contained in the images. Patil et al. concentrated on a method for hierarchical PCA image fusion. According to this study, image fusion is the procedure of mixing two or more photographs of the same scene in order to create a more informative image [26]. The author presented a method for image fusion through merging pyramid and PCA procedures and conducted a quality study of the projected fusion algorithm devoid of a reference image. The proposed fusion algorithm is employed for dimension reduction, feature extraction and image fusion in this work. S. Daneshvar et al. described a technique for increasing the functional and spatial information richness of images by combining the benefits of IHS and RIM fusion techniques [27]. When compared to IHS, Brovey, discrete wavelet transform (DWT), and RIM, the proposed technique significantly improves fusion quality in terms of entropy, mutual information, discrepancy, and average gradient. Image fusion has gained popularity as a means of enhancing the quality of image interpretation in medical applications. The data collected may have a high spatial resolution or a useful functional feature (for example, PET) (like MRI). The MRI image reveals an architecture of the brain tissue but provides little information about its function. The PET image has a low spatial resolution but demonstrates brain function. As a result, image fusion is used to enhance the spatial resolution of functional images by integrating them with a high-resolution anatomic image. The original functional features of the image are kept, and utilizing a flawless fusion process, spatial attributes are added to the image without spatial distortion. Phen-Lan Lin et al. [13] described a unique technique for combining PET and MR brain images using wavelet transforms for low and high activity brain imaging areas [28]. The suggested approach may provide extraordinarily noble fusion outcomes by changing the anatomical structure information in the grey matter (GM) area and then patching the spectral information in the white matter (WM) area following wavelet decomposition and graylevel fusion. The graylevel fused image will contain additional anatomical structural information due to a unique alteration of the pixel intensity in the non-white matter section of the high-activity zone. In the high-activity region's white matter area, spectral information patching will preserve more color information from the PET image. A spectral discrepancy (SD) and average gradient performance measurements are used to compare the fusion outcomes (AG). Desale et al. detailed a theory, process flowcharts, and procedures for image fusion techniques based on PCA (Principal Component Analysis), DCT (Discrete Cosine Transform), and DWT [29]. PCA and DCT are established fusion algorithms with a number of drawbacks, however DWT-based strategies are superior because they generate superior image fusion results. The authors present and compare two DWT-based approaches in this work: pixel averaging and maximal pixel replacement strategy.

Proposed methodology
The Hybrid image fusion is proposed to get better results. Conventionally DTCWT and NSCT are used for medical image fusion. Now new method is proposed with hybrid approach of DTCWT and NSCT.

Dual tree complex wavelet transform (DTCWT)
For the execution of the DTCWT parallel, two actual DWT are used. The real component of DTCWT is obtained using one DWT, while the imaginary part is obtained using the other. Shift Invariance, High Directionality, Computational Efficiency and Perfect Reconstruction are vital features of dual tree complex wavelet transformations. Figure 1 depicts the DTCWT analysis and synthesis filter banks. Decompose the medical pictures using DTCWT into the CCoff1 and CCoff2 complex coefficient sets.

Image1 cof 1
Image2 cof 2 2. Using equation, thresholds are computed for every decomposition level for both coefficient sets. 3. As shown below, the absolute difference between all wavelet coefficients and their associated threshold is computed.
To create the coefficient set for a fused multimodal medical image, the absolute differences between the coefficients in the two source medical pictures have compared, and a coefficient with the biggest absolute difference from the threshold has designated.

5.
To obtain the final picture, IDTCWT is applied to the merged coefficient set.
cof Fused Image

Non-subsampled contour let transform (NSCT)
NSCT is a well-structured strategy that is adaptable and rapid to implement. Its enhanced directionality while multi-scaling sensitivity make it simple for capturing the geometrical configurations for source images, while its improved frequency selection feature enables more accurate decomposition of medical images. The NSCT is divided into dual sections: the NSP and the NSDFB. Figure 6 illustrates the fusion of images using NSCT. 1. Choose the photos that will be used as sources. 2. Reduce the picture size to 256 by 256 pixels. 3. Using NSCT, obtain highpass directional subband coefficients and lowpass subband coefficients for each scale and direction of input pictures. 4. The decomposition was carried out using the NSPFB and NSDFB, which were according to full multiscale and multidirection, respectively. 5. To rebuild the fused picture, utilise inverse NSCT.

Proposed fusion algorithm
Traditionally used fusion algorithms necessitate the collection of higher-quality pictures. To increase the superiority and performance of the proposed technique, the two algorithms were combined (DTCWT-NSCT). Prior to the fusion surgery, make the two level modifications. These enhancements improve the quality of fused details, the way curved curves are handled, and the way input images are characterized. The hybrid algorithm's process flow is depicted in Figure 7. 3. Using NSCT, obtain highpass directional subband coefficients and lowpass subband coefficients for each scale and direction of input pictures. 4. The decomposition was carried out using NSPFB and NSDFB, which were according to full multiscale and multidirection, respectively. 5. Decompose multimodal medical pictures into complicated coefficient sets using DTCWT. For each decomposition level, thresholds are computed for both coefficient sets. 6. All wavelet coefficients' absolute deviations from their associated threshold are computed. 7. To construct the coefficient set of the fused picture, absolute differences of matching factors of both source modalities have compared, and the factor with the largest absolute difference from the threshold has chosen. 8. Finally, the merged coefficient set is subjected to IDTCWT and INSCT to produce the final output picture.

Result analysis and discussion
The results are obtained for various sets of medical images. For testing of proposed algorithm different quality parameters are used. The used image quality parameters are listed below.

5.1
Image quality parameters Peak signal to Noise Ratio (PSNR). The PSNR is frequently employed for evaluating the reconstruction quality of lossy compression codecs (like image compression). A signal represents the original data in this case, whereas the noise reflects the compression error. It is used as a rough indication of human perception of reconstruction quality when evaluating compression codecs. As a result, even if one reconstruction has a lower PSNR than another, it may appear to be more accurate than another (a higher PSNR would normally indicate that the reconstruction is of higher quality).
Fusion Symmetry (FS). The MI of the input and output pictures is used to assess this measure. If the value of get fusion symmetry is lower, the output is of higher quality.
Fusion Factor (FF). The variance between the two pictures, i.e. the output and input images, is evaluated by determining how much information is included in them, according to the fusion factor. It is based on the exchange of information between genuine medical pictures.
The three sets of Different multimodal images are tested for proposed algorithm and quality parameters are recorded for the same. The proposed hybrid method is compared with plane DTCWT and NSCT.
Visual Information Fidelity (VIF). VIF is a critical index for evaluating image quality because it is a perceptual distortion metric. VIF evaluates performance in the context of image fusion by computing common data between the source image and its fused counterpart. Because VIF enables reliable distortion identification, this study evaluates the performance of the offered collection of methods using the average VIF value.
The three sets of dissimilar multimodal images have been tested for proposed algorithm and quality parameters are recorded for the same. The projected technique is compared with plane DWT, DTCWT and NSCT.   If we see this above bar chart then we can realize that the PSNR of Proposed method is higher than the DWT, DTCWT and NSCT methods. If we see this above bar chart then we can realize that the Fusion factor of Proposed method is higher than the DWT, DTCWT and NSCT methods. If we see this above bar chart then we can realize that the Fusion symmetry of Proposed method is lower than the DWT, DTCWT and NSCT methods. If we see this above bar chart then we can realize that the Fusion factor of Proposed method is higher than the DWT, DTCWT and NSCT methods. show that the proposed method is better than conventional DWT, DTCWT and NSCT

Conclusion
Multimodal medical image fusion is critical for identifying illnesses, analyzing clinical treatment choices, and enhancing a performance and precision of computerassisted systems. A hybrid fusion approach based on NSCT and DTCWT was presented for multimodal medical images. To validate the proposed hybrid fusion technique, a simulation experiment was conducted utilizing a mixture of multimodality medical images. The method integrates all critical properties from several source images into a single composite image for more precise diagnosis and treatment. The properties of our method are evaluated using a range of distinguished equivalent algorithms.