Image Deblurring using Wiener Filtering and Siamese Neural Network

—Blurred images are difficult to avoid in situations when minor Atmospheric turbulence or camera movement results in low-quality images. We propose a system that takes a blurred image as input and produces a deblurred image by utilizing various filtering techniques. Additionally, we utilize the Siamese Network to match local image segments. A Siamese Neural Network model is used that is trained to account for image matching in the spatial domain. The best-matched image returned by the model is then further used for Signal-to-Noise ratio and Point Spread Function estimation. The Wiener filter is then used to deblur the image. Finally, the deblurring techniques with existing algorithms are compared and shown that the error in deblurring an image using the techniques presented in this paper is considerably lesser than other techniques.


Introduction
Image processing is a method in which an image is processed to get some useful information from it or to enhance it. Many fields like Computer Science, Mathematics, Medical Science, Military, etc. depend on the development and generation of image processing. Image processing uses many techniques that involve the transformation of brightness, contrast, noise level, and resolution of an image to get an enhanced image. There is an intensive amount of ongoing research in blur removal from an image.
Image blur is a common problem that happens due to the movement of objects or camera shakes. Several factors affect the blurring distortion in images in the physical world, such as wind speed, temperature, humidity, and elevation. In most of these cases, atmospheric conditions are not known prior [1,2]. Therefore, in such cases, it is required that a system can deblur an image efficiently as soon as a blur is detected. The system would be especially useful when a blurred image captured from CCTV footage is required to be deblurred. This paper uses Image Processing techniques and a Convolutional Neural Network model to deblur an image that is captured CCTV footage by taking into account different image frames of the footage. Both grayscale and RGB (Red, Green, Blue) blurred images can be deblurred by the system, however, for simplification, an RGB image can be converted to a grayscale image using OpenCV so that the image can be analyzed in 1-channel. In this paper, we focus on deblurring grayscale images.
Image convolution is done by multiplying the pixels of an image with a kernel. This results in a filter being applied to the image depending on the values in the matrix. Images are deblurred using Image deconvolution. First, the blurred image is enhanced and then Image Deconvolution is applied to get the deblurred image. This process requires another image that is similar to the blurred image, which will be available in the training image dataset. Applying a low-pass filter on the images of the training dataset reduces the high-frequency noise from the image. A blurred image can be restored in the frequency domain using Wiener Filter. The Fourier Transform is used in a wide range of applications, such as image analysis, image filtering, image reconstruction, and image compression. In this paper, the blurred images are restored to the best of possibility by making use of the Siamese Neural Network to find the missing data through Image Matching and use it further to produce a deblurred image.

Background
Image deblurring is the recovery of the original image that was convoluted with a Point Spread Function (PSF) [3] and the addition of noise. Blind deconvolution is used to deblur an image without knowing the PSF used for convoluting the original image, whereas non-blind deconvolution methods use a known PSF for performing deconvolution.
Finding matching images across datasets plays a cardinal role in computer vision applications such as image retrieval [4]. A traditional CNN model for image matching would not suffice in cases where large datasets are not available. It is possible to find matching pairs of images by representing them with neural network based feature vectors [5]. These feature vectors are estimated with the help of Convolutional Siamese Neural Networks that are learned from labeled examples of matching pairs by using a contrastive loss function in the Siamese network architecture. [6] shows that matching performance is improved compared to baseline features obtained with networks which are trained for image classification tasks. This happens even though the labeling of matching and non-matching pairs is not perfect in the training image dataset. The system is promising considering the Image Matching process that our system requires. The development of computer vision and the requirement for auto-matching images have brought into existence many feature detection methods [7] to extract unique features from images. Graph matching [7] offers an alternative way to establish correspondences between two similar images. In [6,8], a few Image Segmentation Techniques based on Edge, Region, Clusters, and CNN are compared. CNN-based techniques provide accurate results even though it takes a comparatively longer time to train the model. Image blur may be due to the sensor's Point Spread Function, sensor motion, or Atmospheric Turbulence. From the refraction index structure functions, Hufnagel and Stanley [1] derived a long-exposure optical transfer function, (1) iJES -Vol. 9, No. 3,2021 was used to model the effect of turbulence in optical imaging. This transfer function is based on a similar assumption on which experiments that are related to gaussian blurring were done. In equation (1), 'k' represents the strength of the blur. The higher the value 'k', the higher the severity of the blur. Images with uniform blurring in patches or a large part of the image can be restored using Wiener filter filtering and Siamese Neural Networks which will be discussed further in the paper. The other existing methods used for deblurring images require prerequisite information or some approximation to know the PSF or noise constraints of the image. The proposed system aims to solve this problem by applying Neural networks' techniques discussed in this paper.

System design
A blurred image that needs to be restored is enhanced by adding suitable brightness, and contrast [9]. In addition to the blurred image, 200 grayscale training images are also processed before image matching takes place. We utilize the Siamese Networks for image matching. The Siamese network is a class of Neural Networks that learns in one shot. An image that is similar to the blurred image-in terms of objects, environment, and various other minor features present in the images-is retrieved in the image matching step. All the training images are fed to the model which returns the best-matched image. The best-matched image and blurred image are then converted into a Frequency domain from a spatial domain using Fourier Transformation. From both the images, the Signal-to-Noise Ratio (SNR) is approximated, and Point Spread Function is estimated using techniques that will be discussed in the further sections of the paper. Wiener filtering is done to obtain the deblurred image. This step requires the Signal-to-Noise Ratio and Point Spread Function of the images. Finally, some existing deblurring techniques are applied and the results are compared. The overall working of the system is shown in Figure 1.

Image preprocessing
The blurred image and images present in the training dataset that is given as input are enhanced in the spatial domain. The input blurred image is resized and some border values are added to images followed by computation of Optimal Discrete Fourier Transform (DFT). Performance of DFT is fastest when the array size is the power of two [10]. Arrays with product sizes of 2's, 3's, and 5's also process efficiently. Hence, the array size of the input image is evaluated and some border cells with 0 value are padded to the image to modify it to an appropriate size. The image is preprocessed so that the result is more suitable than the original image in terms of better quality. The blurred image and images present in the training dataset that is given as input are enhanced in the spatial domain. To get better results in image processing, image enhancement must be done before Image segmentation and Image Matching [9]. The image is preprocessed so that the result is more suitable than the original image for improving the quality of the image. Image Enhancement can generally be done in either the spatial domain or the frequency domain. In this case, the image is processed in the spatial domain so that direct manipulation of the pixels of the image can be done on the image plane itself.

Image matching
A Siamese Neural Network model [4,11] is utilized for Image Matching in this system. Even though this Neural Network takes more time than other image matching models to build, this step is simplified because only a 1-channel method is required for processing grayscale images. Siamese networks are implemented using Keras in Python which provides desirable results and works efficiently as compared to any other 2-channel method.
According to [12] an image is split into 9 regions. Textural features are extracted from each region and are assigned a weight. This results in 9 vectors with different weights for each of the 9 regions. Therefore, a similar procedure is carried out with two images to compare and match these images. All 9 regions of the blurred input image are fed separately to the Siamese networks. With each region of the blurred input image, the corresponding region of the training image is also fed into the particular Siamese network. The Siamese network returns the feature vector of each image of each tile. If the difference of feature vectors of the input image and the output image is insignificant, then the images can be considered identical, and hence the images can be matched. The weighted mean [13] of all the 9 tiles vector differences is taken, and the best matching images are returned. Figure 2 illustrates how the system works for 1 region from the blurred image and a corresponding region with the same vector number. OpenCV in Python is used to classify an image into 9 sets and Keras library for building the Siamese networks. A Sigmoid function in the Siamese Neural Network is implemented to compare the image similarity between the matched image and the blurred image.

Image deconvolution
The resulting image from the previous step is processed in the frequency domain modifying the Fourier transform of an image. The Fourier Transform is an important image processing tool that is used to decompose an image into its sine and cosine components which are its real and imaginary components. In the Fourier domain image, each point is a particular frequency contained in the spatial domain image.
The observation equation of a blurred image can also be expressed as, where F(x, y) is the image before degradation, G(x, y) is the image after degradation, H(x, y) is the degradation filter, and N(x, y) is the additive noise. In other words, F(x, y) represents the true image, and G(x, y) represents the observed image. The aim is to find G(u, v) that provides a good estimate of the original image. One of the direct ways to deblur an image is by using Wiener Filter, The blurred images are known to contain noise [6,14]. The estimated noise is calculated for the given blurry image with the help of the training image dataset. The image from the training dataset obtained in the Image Matching step is used to calculate the Signal-to-Noise ratio. After deduction of the noise, the PSF [14] is calculated by comparing the DFT of the blurred image and the DFT of the matched image from the dataset, this gives the DFT of the PSF method, which we can directly substitute in the Wiener filter formula. After substituting the values in the Wiener filter, the original image is obtained in the frequency domain, then the inverse discrete Fourier transform is applied to convert it back to the spatial domain.

Results and discussion
Wiener filtering technique of deconvolution was applied and compared with different approximations and processes which will be discussed further. Apart from the Image Matching with Convolutional Neural Network that was followed in this paper, other common deblurring techniques were directly used so that there is a means for comparing our deblurred image to deblurred images of other techniques. The other deblurring techniques include the K-type approximation of Wiener filtering and CLS filtering [15]. To compare the results achieved by deblurring the image from other common deblurring methods and algorithms, there are 2 ways [16] to compare the deblurred image with the original image: The lesser the value of MSE, the lesser the error in images. As for the SSIM value, if the value is approaching 1, the images tend to be similar. While MSE focuses more on finding out the error difference, SSIM perceives the structural changes.   Table 1 shows the MSE and SSIM values of the original image and deblurred image after applying the deblurring techniques. It is observed that the Image Matching technique that we used provided better results than the others with MSE equal to 559.38 and SSIM equal to 0.7773 as shown in Figure 3(c). Figure 3(b) shows the deblurred image after applying the K-approximation technique where the value of K is 0.1. The error induced in the final deblurred image, after using the techniques presented in this paper, is comparatively lesser than the pre-existing techniques. The image that was retrieved from the Image Matching step is shown in Figure 4. The image shown in Figure 4 has been determined by the Siamese Neural Networks as the closest image that matches the features of the input image. This output image can be further used to calculate the parameters such as PSF and SNR that are needed for the Wiener Filtering algorithm for deblurring the image. Results could be more efficient if larger datasets with similar images are available, especially in cases like CCTV footage where many good-quality image frames can be captured and saved. This would provide more data to the system compared so that the blurred image can be compared with a very similar image from the dataset. It could also work in cases when it is required to detect the car plate number of a car but the image captured by the camera or CCTV is blurred. When it is required by the Siamese Neural Network architecture that an image is to be split into different regions, the blur may be present in some patches (or regions) non-uniformly due to the atmosphere. In this paper, we make use of the 1-channel Siamese network, however, 2-channel methods [12] provide better results because it follows the brute force concept. Due to this, test time tends to be very expensive.

Conclusion
In this paper, the impact of various deblurring techniques on blurred images is observed, and it is found that the best results were provided using Siamese Networks that provided efficient results. Since each image is split into 9 tiles and compared with another, it results in fewer errors while matching. The deblurred image obtained after following the steps mentioned in the paper gave better results than deblurred images obtained from various other deblurring techniques, which was proved by the SSIM value. Apart from the high SSIM value, it can be seen visually that Figure 3(c) provided a better-deblurred image than the other deblurred images. Hence, this system works best when similar types of images exist in a dataset, especially in cases where images could be captured from CCTV footage. This could help provide more data for better Image Matching results.