DR-LL Gan: Diabetic Retinopathy Lesions Synthesis using Generative Adversarial Network

— Diabetic Retinopathy (DR) is a serious consequence of diabetes that seriously impact on the eyes and is a leading cause of blindness. If the lesions in DR arise in the central portion of the fundus, they may result in significant vision loss, which we refer to as Diabetic Macular Edema (DME). Deep learning (DL) techniques are commonly used utilized in ophthalmology for discriminative tasks such as diabetic retinopathy or age-related macular degeneration (AMD) diagnosis. Deep learning techniques typically need huge picture data sets for deep convolutional neural networks (DCNNs) training, it should be graded by human specialists. According to international protocol, it is classified into five sever - ity categories. However, improving a grading model for high generality needs a significant quantity of balanced training data, which is challenging to obtain, especially at high levels of severity. Typical techniques for data augmentation, in many applications of deep learning in the retinal image processing domain, the difficulty of access to huge annotated datasets and legal concerns about patient privacy are limiting issues. As a result, the concept of creating synthetic retinal pictures that are indistinguishable from actual data has garnered more attention. GANs have been certain to be an effective framework for creating synthetic databases of anatomically accurate retinal fundus pictures. GANs, in particular, have garnered increasing attention in ophthalmology. in this article, we present a loss-less generative adversarial network (DR-LL GAN) to generate good resolution fundus pictures that May be adjusted to include random grading and information about the lesion. As a result, large-scale generated data may be used to train a DR grading and lesion segmentation model with more appropriate augmentation. Our model experiments evaluated on IDRID and MESSIDOR datasets, it’s obtained a discrimination loss of 0.69374 and a generation loss of 1.10438, as well as a segmentation accuracy of 0.9840 in our tests. This might support in the optimization techniques of the neural network design and in computer-aided screening of medical picture, thus increasing diagnostic reliability for clinical assessment in the future of sophisticated technological healthcare.


Introduction
Diabetic retinopathy (DR) is a frequent a diabetes-related complication that may result in vision loss or even blindness.Typically, human ophthalmologists diagnose and evaluate the severity of DR according to the kind and a certain number of associated lesions.based on the international grading protocol [1], the technology assisted medical diagnostics is of considerable interest to medical professionals [2] as a means of assisting them in the exploration of biological pictures [3].Utilizing the capabilities of artificial intelligence has garnered considerable interest in recent decades.Especially deep learning (DL) techniques have already been shown to execute exceptionally well when used to analyze medical pictures [4].Clinically effective DL approaches have been built for a number of applications in ophthalmology, including the detection of several eye disorders such as diabetic retinopathy (DR) [1,[5][6][7][8], elated macular degeneration (AMD) [9,10] glaucoma [11].These findings indicate that there is substantial scope for healthcare and ophthalmic treatments, as well as for adoption in screening programs.Nonetheless (DR-LL GAN) Typically, it takes a significant quantity of varied and balanced data to train an effective deep CNN model.However, the distribution of DR data across grades is highly unbalanced, since aberrant fundus pictures account for a tiny proportion.For example, in the biggest publicly available dataset on DR, Eye-PACS [12], pictures with DR stage 3 and 4 account for just 2.35 percent with 2.16 percent in percentage to the overall, respectively, whereas the images without DR (normal level 0) are 73.67 percent.Employing of this kind unbalanced data reduces the model's sensitivity to specimens with greater levels of DR severity and increases the likelihood of overfitting.
Despite popular data augmentation techniques like as rotation, cropping, and flipping may help alleviate the issue, the lack of variety in samples at those levels continues to restrict model performance.Alternatively, these requirements may be fulfilled by using generative deep learning techniques, like GANs, to generate fresh pictures of retina from a set of actual images which dedicates for training.DCGAN [13] expands GAN by adding a convolution layers that serves as an upscaling procedure for low quality pictures.CGAN [14] attempts to include conditions into the generator by concatenating a vector of one-hot with a vector of random noise.Additionally, CycleGAN [15] supports a translation of unpaired images-to-image from one field to another.BigGAN [16] is a technique that combines some of the most advanced current techniques for conditional training pictures and scaling up the parameters of model and batch size.Consequently, the high-fidelity with high-resolution pictures may be generated routinely.We were motivated by this well-crafted conditioned GAN architecture to develop a retinal images generator capable of generating high-resolution photographs that are realistic.The goal of this study was to figure out the capacity of high-resolution GANs to generate realistic fundus pictures that may be used as proxy data sets for teaching and training purposes by retinal experts and deep learning computers.Generative adversarial networks (GAN) are extensively employed to generate images [17].The GAN architecture typically composed of a generator model and a discriminator model G & D respectively, performing a game of min-max, which has resulted in significant advancements in photorealistic picture synthesis.Particularly, one neural network attempts to create actual data, while the others attempt to distinguish between genuine data and synthetic.the proposed design is composed of a generator for retinal pictures constrained by optical disk OD and the masks of lesions.attempts to address the aforementioned problem by presenting a (GAN) for retina image processing.Thus, the analysis's performance will be enhanced, and overcome the training's reliance on a large quantity of retinal pictures.Additionally, it will include a succinct explanation of the solution idea and technology choices utilized to create the smallest practicable prototype to optimally collect retinal pictures and analyze them using machine learning model.

Related work
Since before DL era, synthesizing accurate pictures of the ocular fundus has been a difficult job.Initially, it was addressed by developing sophisticated mathematical models of ocular anatomy [18,19].Nowadays, technological advancements have resulted in significant computing capacity, allowing ML to be applied to neural networks with deep architectures.GAN is a useful framework in context of modern advances in deep learning algorithms.GANs are rapidly improving [20].aided the creation of realistic-looking pictures, resulting in somewhat anatomically consistent colored retinal fundus images with acceptable visual quality [21][22][23][24][25][26].Goodfellow introduced GAN, an unsupervised deep learning system.et al. [27], comprised of two models: a discriminator and a generator Using random noise samples, the generative system learns to obtain the data distribution and produces convincing pictures from it.The discriminative model is charged with distinguishing between genuine and fraudulent pictures by estimating the likelihood that a sample originates from the distribution of data rather than the generator distribution (Figure 1).(DRGAN) GANs in Generation Medical Images: Medical image synthesis with GANs [28] may help to solve the problem of a scarcity of big, varied annotated datasets.Methods which are suggested for a number of imaging domains in medical field, e.g, magnetic resonance imaging (MRI) [29][30][31], chest X-rays [32] and computed tomography (CT), [33][34][35].Radiation exposure through CT imaging, for example, increases the risk of cancer.A series of fully convolutional 3D networks were shown in a cascade [33] CT images from MR acquisitions are synthesized.In addition to adversarial learning, For the creation, pixel-wise reconstructing loss and then an image gradient loss was utilized.To generate segmented masks based images for the chest X-ray, Mahapatra et al. [32] used probabilistic GAN with Bayesian networks for active learn.Furthermore, positron emission tomography (PET) pictures are usually used in cancer diagnostics and grading, although they are costly and a high radiation.
To generate the PET pictures from different volumes of MR, GANs was suggested in [36], in the generators they used 3D U-Nets but they used 3D CNNs in discriminators build.Retinal Image Synthesis using GANs: GANs have recently been used to generate retinal fundus pictures by certain researchers.Costa et al. [23] used a vanilla GAN architecture and a U-Net design to convert segmentation masks of the vessels to fundus pictures.The produced samples, on the other hand, contain block flaws and no grading information that can be controlled Figure 2. Tub-sGAN [37] was suggested to enhance the variety of generated samples by extending style transfer to the generator.Despite some progress, it is unable to synthesize lesions associated with DR and physiological retinal features accurately, the author attempted to create fundus pictures using disease characteristics and vessel segmentation masks more recently.The location and number of lesions may be altered.Guibas [24] presented a two-stage pipeline comprised of a DCGAN [13] structure trained to synthesis vessels of retina used a noise and an another cGAN (Pix2Pix [38]) trained using Messidor fundus pictures to produce the matching colour fundus image.They showed that the network training with just synthetic images produces just marginally worse results by calculating F1 scores.They also used the KL divergence score to quantify the disparity between the synthetic and actual datasets, demonstrating that the generated data are distinct from the genuine data and do not replicate the original pictures.Figure 3 shows some synthetic findings.Zhao et al. [26] created Tub-GAN, a structure that can generating various outputs by segmentation of vessel by changing a code z, Limited training datasets of 10-20 pictures are sufficient for the model to learn.To create retinal fundus pictures using GAN, the authors used the technique of Isola et al. [38].They designed the generator using an encoder-decoder approach that allows for the natural insertion of latent coding without the requirement for dropout.They also included a Tub-sGAN in the structure, which incorporates style transfer.This is feasible thanks to the inclusion of a style picture as a training input.As a result, the method adds two more components of perceptual loss [39]: loss of content and loss of style, and also a loss of variation, into the optimization issue.The authors used 20 resized pictures from DRIVE and 10 pictures from STARE the size changed to 512 × 512, and 22 HRF pictures to train the models.They reduced the data to 2048 × 2048 instead of 3304 × 2336 since HRF raw pictures are extremely big (3304 × 2336).The authors tested the quality of their synthetic pictures extensively and found that the Structural similarity image quality measure (SSIM [40]) was used to give a quality evaluation, with SSIM scores being greater than Costa et al. [22] findings.Figure 4 shows some synthetic findings.Zhao et al. [41] recently published a paper aiming at creating a fake fundus pictures collection for vessel segmenting.The authors looked at a recurrent neural network called a gater recurrent unit [42].Taking use of repetition, many types of pictures may be recreated.This approach, on the other hand, simply introduced lesion modification without taking into account a global description for grading.the generated pictures still require to ophthalmologists decision if they are quick and easy to set as a grades to assist the models of grading.In this paper, we show how our generation approach can be tweaked to create high-resolution pictures with customizable grading and lesion information.As a result, the produced samples may be immediately used to aid in the training and improvement of DR grading models.

Proposed methods
The proposed generator network is structured using a U-Net network topology.The network produces a synthetic diabetic retinopathy fundus RGB picture  > @ 0 1 .Assuming X is the case of original picture ˆ, x p must be trend toward 1, whereas X is the combined picture x p , ust attempt toward zero.We solve the optimization issue that encapsulates the interaction of G with D: During the training process, both networks' performance increases at the same time: the generator attempts to deceive the discriminator by producing pictures that are as genuine as possible, while the discriminator strives to avoid the generator deceiving by increasing its discriminative capacity.One of the most often utilized GAN general loss functions is the Equation 1.
When the real data distribution expressed as Pdata(x), while P(z) is the expression of distribution of fake images that generated, as well as G(x) and D(z) are the Generator and discriminator Neural network, respectively.
The discriminator network D attempts to discriminate between synthetic and is the Discriminator Neural network, and genuine pictures.Furthermore, the discriminator may be defined as a discriminant formula p D X y p > @ J , , , 0 1 .When the X is the original picture x p , hould trend toward one, whereas X is the combined picture ˆ, x p hould tend toward zero.
Both the generator and discriminator networks are neural networks which strive and both designs benefit from many repeated steps throughout the training phase to achieve better training accuracy.The GAN network's operation is as follows: the generator design receives a distribution of images and trains in the same way that the possibility of the discriminator making a mistake is maximized, and the discriminator estimates the possibility of the data distributing from the retina fundus images not from the synthetic that been generated.Throughout the training phase's many stages, the discriminator attempts to maximize its loss until there acquires more confidence that it misclassified the produced picture with the actual photos.
The initial stage is to develop the ability to synthesize a realistic fundus picture, we proposed an adversarial learning model, which is commonly utilized image synthesis based on provided inputs, since this is an image creation job.A fundus generator (G) and a discriminator (D) are the two models we construct.The model is designed to calculate the best solution of lose for generator and discriminator by utilized Adam optimizer for this task, then we add a noise for the images before start training the model to prove that the model can generate a high-resolution samples from noisy images.As a result, the generator may be trained using pairs of actual fundus pictures and their associated structural and lesion masks.Even though we used an adversarial learning mechanism that normally only uses the adversarial loss function for training, G must be trained to accurately synthesize the identity of the lesion and vessel in the same position, despite the fact that the lesion mask that used in this paper is a one-channel image representing the four representative lesions, The vascular, optic disk, and other lesion segmentation models are trained using a U-Net architecture [43].

Experimental results
Dataset: two datasets used for training and evaluation purpose Eyepacs and IDRID which are both public datasets often using for eye disease diagnosis especially DR grading.The datasets are supplied with a retina of both eyes, left and right of the patient and the pictures are labeled with both sides for the sake of clarity with a candidate ID (for example.10 l.jpeg/right eye/ID 10), actually the dataset contains four distinct picture categories: (Normal 0), (Mild 1), (Moderate 2), (Severe 3), as well as (Proliferative DR 4).
Implementation: the experiments was implemented using colabs from google, we trained G and P with reconstruction loss and regularization loss to improve the respective generation and prediction performance by employing the U-NET based conditional.In the first step, we used the IDRID disease grading data and MESSIDOR data for training the G and the D1.Once the fundus generator is trained, we can synthesize not only the realistic fundus images, which can be coarsely predicted from the corresponding grading images, but also control the pathological information by switching the lesion mask.We randomly selected the images from each different DR grading for training and validation purposes by using three datasets we observe that the fundus generator can synthesize fundus images that reflect the lesion information structure, The quantitative analysis, particularly, entails determining the utility of the produced pictures for the training of the semantic segmentation model.This method is similar to [44] and is predicated on the premise that the performance of a DL approach U-Net is directly proportional to the quality and diversity of GAN produced pictures.

Results
We utilized a Generative Adversarial Network with the parameters of 128 image size, 10 noise, the batch size is 64, with 500 epochs, as well as the following learning rates: LR D was 0.00004, whereas LR G was 0.0004, BETA1 was 0.5. the optimizer that used is Adam optimizer, which results in the creation of plausible retinal images.When produced pictures are trained alongside existing actual photographs, the effectiveness of the segmentation model's feature learning improves even with a limited dataset.On lesion segmentation, our approach has a low percentage of false positives.
To allow for long-term training of both the generator and discriminator, we included a backpropagation loss function.
Because the discriminator will eventually become more powerful than generator, it is crucial to use Gaussian noise to weaken it applied to the input photos, as well as one-sided labeling flattening of the final photo.The ultimate loss totals 0.69374 for D and 1.10438 for G.The training diagram is shown in Figure 5 below.The synthetic data is used to train the proposed U-Net approach for segmentation of DR lesions, it's achieved a higher result to that obtained by real data training.This shows that the produced pictures accurately represent the distribution of training images and can therefore be utilized to properly train a deep neural network.

Method Accuracy
K Xu [45] 94.5% Cao [46] 91.6% Harangi [47] 90.7% Ghan [48] 93% Iyyanar [49] 92% Our model 94.2% 99.5% with synthetic data As shown in Table 1 the segmentation accuracy is improved form basic approach which is trained with real noisy and imbalanced data to the high accuracy segmentation using same approach with synthetic high resolution balanced data, which is prove the effectiveness of proposed DR-LL GAN and the high influence of using balance datasets for training task.

Discussion
This research shows that a deep learning (GAN) could be trained to map biological features from several picture modalities, such as fundus images, onto a common feature manifold in order to generate one picture modality from another.After training on a dataset of matched fundus pictures, the DL network is ready to use.Even with considerable noise, blurriness, and color and brightness alterations, our deep learning algorithm was able to generate accurate and trustworthy samples from fundus images.
The proposed DL architecture's most notable feature is that it is the first use of DL in ophthalmology designed to exchange between two distinct data modalities.We also used a thorough data augmentation technique to increase the results of our DL model without requiring a huge training dataset.
Using the findings of our research, detailed retinal structures may be generated without the requirement for data collection.From multiple cases which is very difficult to collect and need more efforts for the privacy issues.Moreover, using DL generative networks to generate fresh pictures from fundus photography allows for a non-invasive, cost-effective, simple-to-use, and low-cost option as an alternative for current imbalance data and collecting new balance and high-resolution datasets.In this research, the suggested DL generative network generates new data from fundus images.This discovery has clinical implications since fundus imaging is a simple, low-cost, non-invasive technique that is one of the most frequently done eye operations, resulting in a huge number of fundus imaging databases.Our research shows that fundus pictures obtained from all of these compact fundus imaging devices may be used to create dependable and anatomically precise retinal lesions.The present study's limitations include using a particular dataset of matched fundus pictures, the dataset's size, and our inability to conduct longitudinal tests of the suggested method's advantages in assessing disease progression owing to data constraints.Whereas this dataset was adequate for demonstrating the efficiency of the suggested DL model, further research on paired fundus pictures is required to confirm the findings.To address the dataset's size constraints, we suggested using conditional GANs with data segmentation model in this research.We believe that utilizing bigger training samples derived from huge datasets will enhance our method's already excellent performance.We want to include these details into our analysis in future research.

Conclusion
GANs have made tremendous progress in recent years, as well as the creation of retinal pictures using GANs has gained increased interest.Experiments indicate that clinicians can generate create high quality synthetic fundus pictures of retinal disorders using GAN and use them instead of the real images (e.g., for train the model and data exchanging); These pictures have the potential to be useful in a deep learning model.Although unethical applications of the technology are possible, we present an efficient high quality and resolution DR image generating model that is conditional on lesion grading.The created data may be utilized to supplement existing data, especially for pictures with high DR grades, to enhance the effectiveness of segmenting and classifying algorithms.For the future planes, we will solve the constraints of our present model in order to improve DR-LL GAN training.We synthesized the fundus picture, which is reflected in the lesion information.We could train the fundus generator more quickly since we could utilize many small DR fundus datasets rather than longitudinal data.nevertheless, the first apparent extension of the current models will include handwritten annotation for retina lesions on training datasets of fewer than 50 pictures.The goal would be to generate synthetic images that appear realistic, and the freshly annotated data would be used to build innovative retina image processing methods or to supplement a database that already exists with relevant data.Additionally, the versatility of GAN implies that they may be utilized to apply the methods employed for retinal synthesis to a variety of medical pictures.