Smartphone Based Image Color Correction for Color Blindness

— Color blind is a type of Color Vision Deficiency, which is the inability that a person could not realize the differences between some colors. There are three types of color blindness: Monochromacy, Dichromacy, and Anomalous Trichromacy. Color blind cannot be cured. Today, technology gets up with solutions to help people with color blindness to see the image and distinguish between the different colors using some algorithms. This paper presents a smartphone based experimental comparison of color correction algorithms for all Dichromacy color-blind viewers: Protanopia, Duteranopia, and Tritanopia. This comparison includes LMS Daltonization algorithm, Color-blind Filter Service (CBFS) algorithm, LAB color corrector algorithm, and the shifting color algorithm. The LMS algorithm is implemented for all the three types of Dichromacy. While CBFS, LAB adjustment, and Shifting color algorithms are applied to correct colors for Protanopia, Duteranopia, and Tritanopia respectively. The results show that the processing time for LMS algorithm is slow compared to other algorithms. For Protanopia people, the LMS algorithm is better than CBFS algorithm as the LMS algorithm only changes color of confused areas with no change in the brightness. For Duteranopia people, the LAB color correction is better than the LMS algorithm. For Tritanopia people, both the shifting color algorithm and the LMS algorithm may produce a new confusion in the proceed images. An application interface is implemented to enable the user to choose the algorithm that gives the most appropriate results.


Introduction
Color blindness affects approximately 1 in 12 men and 1 in 200 women in the world. Most people with color blindness have an ability to see things as clearly as other people but they unable to see red, green or blue light clearly. There are extremely rare cases where people are unable to see any color at all [1], [2].
There are different color blindness causes. For the most people with color vision deficiency, the condition is genetic and has been inherited from their mother. Also, there are some diseases causing color blindness such as diabetes and multiple sclerosis or acquiring the condition over time due to the aging process, medication and so on [1], [2].
For human vision, there are two types of photoreceptors: rods and cones. Rods are sensitive to light while cones are sensitive to colors. Cones have three types; L-cones which are sensitive to long wavelength (red), M-cones which are sensitive to middle wavelength (green), and S-cones which are sensitive to short wavelength (blue). According to these cones, there are three types of color blindness [3], [4]; 1) Monochromacy, in which no cones or only one cone type exist, 2) Dichromacy, in which one cone type is missing, which can be of three types: Protanopia, in which L-cones are missing, Duteranopia, in which M-cones are missing, and Tritanopia, in which Scones are missing, and 3) Anomalous Trichromacy, in which there is a reduction in the sensitivity to a particular color, which can be of three types: Protanomaly which corresponds to a reduced sensitivity to red light, Dueteranomaly which corresponds to a reduced sensitivity to green light, and Tritanomaly which corresponds to a reduced sensitivity to blue light. Protanopia and Duteranopia are the two types of red-green color blindness. Tritanopia is known to be blue-yellow color blindness. Table 1 summarizes the different color-blind types, their causes and effects. Color blind cannot be cured. Therefore, the only method to help color blind people to recognize or differentiate color is with a vision aid kit. Today, technology gets up with solutions to help people with color blindness to see the image and distinguish between the different colors using some algorithms.
This paper focuses on Dichromacy color blind type. Figure 1 shows how each Dichromacy color blind person see an image against the normal person. This paper pre-sents a smartphone based experimental comparison of color correction algorithms for Dichromacy viewers. This comparison includes LMS Daltonization algorithm [5], [6], [7], Color-blind Filter Service (CBFS) algorithm [8], LAB color corrector algorithm [7], and the shifting color algorithm [9]. The LMS algorithm is implemented for all the three types of Dichromacy: Protanopia, Duteranopia, and Tritanopia. The CBFS algorithm is implemented only for Protanopia viewers. The LAB color corrector algorithm is implemented only for Duteranopia viewers. The shifting color algorithm is implemented only for Tritanopia viewers.
The rest of this paper is organized as follows: section 2 introduces the literature review. The tools and methodologies are introduced in section 3. In section 4, the details of the color correction algorithms implementations and experiments are presented. Section 5 shows the results. Finally, Section 6 concludes the paper.

Literature Review
In internet world and websites, there are many computerized aids developed for color blindness: • Color-blind Testers • Color-blind Simulators • Color correction/recognition.

Color-blind Testers
Color-blind testers are used for testing if the person has a color blind, and if yes, determine the type of color blind.
There are many online web-based and smartphone-based applications tests that are available to test the color vision. Examples of these online web-based tests are 1

Color-blind Simulators
Using color blindness simulators reveals how images may appear to users with a variety of color blindness conditions. People use color-blind simulators to check their art work, pictures, documents and web pages for color blind visibility [10] [11]. Some simulators are offered as an offline software program, an online web page, or a mobile application. Table 2 shows some examples of these simulators and their supported color blindness types and platforms. all common colorblindness types are supported.
• an Android application that simulates and corrects color blindness in real-time using the built-in camera of the mobile phone.

Color Correction for Dichromacy
There are many researchers proposed color correction techniques for Dichromacy. These techniques vary in terms of the supported Dicromacy types and the algorithms used. Techniques for modifying images for Deuteranopia viewers are presented in [6] and [7]. They used three algorithms: LMS Daltonization, color contrast enhancement, and LAB color adjustment. The LMS Daltonization algorithm, also used in [5], [12].
K. Erdo!an and N. Yılmaz [9] suggested a method that shifting colors based on HSV color space. The main concept is shifting the Hue channel for all spectrum by a specific value in the range [0.1-0.9].
The Color-blind Filter Service (CBFS) algorithm, suggested by [7], is a userassisted recoloring technique. It modifies the original image by using six parameters provided from the user. In [13], a color compensation vision system is proposed for red and green color blindness people. This system is depending on converting RGB image to HSL.
C. L. Lai et al. [14] developed a portal vision auxiliary agency system for people who have color blindness or serious low vision. Their system passes through three steps, the vision model measurement, perceptual compensation and pattern recognition.
The mentioned color correction algorithms can be used for each type of Dichromacy color blindness. Hence, algorithms applied in the papers are suggested for a particular type of color blindness, but this is not precluded to apply them and work properly for another type.

Tools and Methodology
In this paper, a smartphone based experimental comparison of color correction algorithms for Dichromacy viewers is presented. Android platform is chosen because it is free and open source [15]. Android Studio, which is one of the most used Integrated Development Environment (IDE), is used to develop the application. Android Software Development Kit (SDK) is used to help developing mobile applications for Android. The Android SDK tools include development tools, emulator, and required libraries to build Android applications 7 . OpenCV is an open source computer vision software library 8 , it is used for image processing in the developed application.
In this paper, four algorithms are implemented to correct colors for Dichromacy. As shown in Table 3, the LMS Daltonization algorithm is applied to correct colors for all Dichromacy types. While CBFS, LAB adjustment, and Shifting color algorithms are applied to correct colors only for Protanopia, Duteranopia, and Tritanopia respectively. In Section 4, the smartphone-based implementations of these four algorithms are explained. Application interfaces, shown in Figure 2, are created and ported on LG g2 smartphone that run on Android jelly bean OS v.4.2.2. Figure 2 (a) shows the application icon. The main interface that appears when the user opens the application is shown in Figure 2 (b). In the main interface the user can choose his/her colorblindness type. Then one of the interfaces, shown in Figures 2 (c)-(e), enables the user to choose one of the two solutions created for each color-blindness type. The user can try both solutions, then choose the most suitable and comfortable one to his/her daily use. After choosing one solution, the mobile camera will open; user can capture any view then press OK. The application will process this view image and display the processed image on the mobile screen, as shown in Figure 2 (f).
For comparison purposes, laptops are used to run Android Studio and to run the simulation software. An offline methodology is followed as illustrated in Figure 3. The RGB original image is first entered to a color-blind simulator to see how the original image is seen by Dichromate. Then the RGB original image is processed by the implemented color correction algorithms to get the color corrected image, which is fed to the color-blind simulator to see how the corrected image is seen by Dichromate. Vischeck and Coblis simulators are used to see the difference between the original and the processed images for Dichromate people.

LMS Daltonization Algorithm
The LMS algorithm in the developed application is implemented for Protanopia, Duteranopia, and Tritanopia. It is the most famous algorithm used for color-blindness correction. Its idea is to use the information lost in the simulation of color blindness and use LMS color space to compensate colors missing in each group/type of cones, long (L), medium (M), and short (S) in order to be predictable to the viewer [5], [6], [7]. Algorithm 1 lists the steps of LMS Daltonization algorithm. In step 5 of algorithm 1, there are error modification matrices for Protanopia and Deuteranopia, but there is no special matrix found for Tritanopia. Although the work in [5] mentioned the Tritanopia matrix that convert from 3D LMS space to 2D space, it is not mentioned the Tritanopia matrix of error modification. The same error modification matrix of Protanopia is used also for Tritanopia. In our implementation, a new proposed error modification matrix for tritanopia is used, as in equation (11). Figure 4 shows the differences for Tritanopia on some sample images processed using the Protanopia and the proposed error modification matrices. As shown in Figure 4, both error modification matrices distinguish between blue and green and be-tween yellow and purple. But the Protanopia error modification matrix shifts all spectrum colors to blue and green. As the Tritanopia people have a problem with blue color, shifting colors to blue is not a good solution. The proposed error modification matrix shifts all spectrum colors to green and red.

Color-Blind Filter Service (CBFS) Algorithm
The CBFS algorithm in the developed application is implemented for Protanopia. It uses HSL (Hue, Saturation, Lightness) color space to represent colors [8]. Hue represents the color itself, Saturation is the amount of color, while Lightness is how much light or dark the color is. Algorithm 2 specifies the steps of this algorithm. In step 2 of algorithm 2, a closeness parameter is defined in order to decide how much the HSL pixel color is close to the dominant color of the RGB image pixels and it is equal to the absolute difference between them. For Protanopia, many values for this parameter from 10 to 90 are tested. Figure 5 shows the testing results on some images for closeness parameter equal to 30, 70, and 90. Based on the results shown in Figure  5, the closeness parameter value is set to 70 in the proposed application.

LAB Color Corrector
The LAB Color Corrector algorithm in the developed application is implemented for Duteranopia. It corrects colors for color blinder by modifying reds and greens of an image to increase color contrast for a color blinder [7]. The LAB Color Corrector implementation steps are listed in algorithm 3. The first operation is to adjust each pixel of A component, where a positive A means it is closer to red while negative A means it is closer to green as the LAB color range [-100, 100]. However, in Android platform, LAB ranges from 0 to 255. Hence, we considered the half of 255 (127) as 0 iJIM -Vol. 12, No. 3, 2018 value in the range [-100, 100], values greater than 127 is the positive part, and values less than or equal 127 is the negative part. This algorithm lacks clear theoretical basis. It is based upon experimental procedures relying mostly on trial and error in the presence of a color-blind viewer. Hence, the best adjustments between L, B, and A components for Duteranopia is determined based on experimenting different trials. According to algorithm 3, these trials were chosen to increase the color contrast between red-green to be noticeable regards to Duteranopia case.

Shifting Color Algorithm
The Shifting color algorithm in the developed application is implemented for Tritanopia. In the Shifting color algorithm RGB image is read and colors are shifted by different ratios in HSV color space (Hue, Saturation, and Value) [9]. In algorithm 4, shifting color steps are explained. The ! value in the algorithm defines the shifting ratio and it would be changed according to the color vision deficiency type [9]. To specify the ! value for Tritanopia algorithm, ! values are changed from 0.1 to 0.9 and tested for each one on some images as shown in Figure 6. The Hue channel in OpenCV has a range value from 0 to 179, so, the ! value is multiplied by 180. Hue values of the whole image are shifted with a fixed rate, the colors will be changed but the information will be preserved. As shown in Figure 6, the green color is shifted to the inverse range of the blue color. We can see clearly the difference in the range of blue and green when ! between 0. 3  .
4: Convert HSV image back to RGB.

Results
The implemented algorithms explained in section 4 are ported on LG G2 smartphone with 2 GB Ram, 13 MP camera, quad-core 2.26 GHz CPU, and run on Android jelly bean OS v.4.2.2.
A set of test images are used to compare the results from the implemented algorithms for each Dicromate type, as listed in Table 2. Figure 7 shows the resulted processed images from the implemented algorithms as seen by Protanopia: LMS Deltonization, Figure 7 (c), and CBFS, Figure 7 (d). People with Protanopia have a difficulty to distinguish between black and many shades of red, dark brown and dark green, dark orange and dark red, blue and red, purple and dark pink, and mid green with some oranges [4]. The tested images processing average time is 5.43 seconds for LMS and 2.65 seconds for CBFS. The LMS algorithm only changes color of confused areas with no change in the brightness, while almost all colors of the image are changed by the CBFS algorithm. Unlike the CBFS proceeded images, in the LMS processed images, different colors are converted to different colors after processing.  Figure 8 shows the resulted processed images from the implemented algorithms as seen by Duteranopia: LMS, Figure 8 (c), and LAB, Figure 8 (d). People who suffer from Duteranopia confuse mid-reds with mid-greens, bright greens with yellows, pale pinks with light grey, mid-reds with mid-brown, and light blues with lilac [4]. The tested images processing average time is 9.1 seconds for LMS and 1.3 seconds for LAB. Both algorithms achieved their purpose. In LAB color correction, there is a noticeable difference in the color range that is not visible for color-blind. In the LMS Daltonization, the Colors after processing are not quite noticeable for color-blind. Figure 9 shows the resulted processed images from the implemented algorithms as seen by Tritanopia: LMS algorithm, using the proposed error modification matrix in equation (11), Figure 9 (c), and the shifting color algorithm, Figure 9 (d). Tritanopia is a very rare form of color blindness, occurring in less than 0.003% of males and females. The most colors that Tritanopia people confuse are light blues with greys, dark purples with black, mid-greens with blues and oranges with reds [4]. The tested images processing average time is 12.2 seconds for LMS and 3.4 seconds for the Shifting algorithm. In LMS, some converted colors become similar and cause a new confusion. The shifting color algorithm makes a difference between the colors that must be different for Tritanopia people, but some other colors may be affected.

Conclusions
The goal of this paper is to present a smartphone based experimental comparison of color correction algorithms for Dichromacy viewers. This comparison included the LMS Daltonization algorithm, Color-blind Filter Service (CBFS) algorithm, LAB color corrector algorithm, and the shifting color algorithm. The LMS algorithm is implemented for all the three types of Dichromacy: Protanopia, Duteranopia, and Tritanopia. In our implementation of the LMS algorithm, a new proposed error modification matrix for Tritanopia is used, as in equation (11). The CBFS algorithm is implemented only for Protanopia viewers. The LAB color corrector algorithm is implemented only for Duteranopia viewers. The shifting color algorithm is implemented only for Tritanopia viewers. These algorithms convert colors to other colors that color-blind persons can distinguish.
The results show that the processing time for LMS algorithm is slow compared to other algorithms. For Protanopia people, the LMS algorithm is better than CBFS algorithm as the LMS algorithm only changes color of confused areas with no change in the brightness. For Duteranopia people, the LAB color correction is better than the LMS algorithm. For Tritanopia people, both the shifting color algorithm and the LMS algorithm may produce a new confusion in the proceed images.
Although these algorithms have some weaknesses, they can solve their specific color-blindness type problem properly. In the developed application, the user can try solutions from the different implemented algorithms, then choose the most suitable and comfortable one to his/her daily use.