OmniColor – A Smart Glasses App to Support Colorblind People

—Colorblind people or people with a color vision deficiency have to face many challenges in their daily activities. Their disadvantage to perceive colors incorrectly leads to frustration when determining the freshness of fruits and the rawness of meat as well as the problem to distinguish clothes with confusing colors. With the rise of the smartphone, numerous mobile applications are developed to overcome those problems, improving the quality of live. However, smartphones also have some limitations in certain use cases. Especially activities where both hands are needed do not suit well for smartphone applications. Furthermore, there exist tasks in which a continuous use of a smartphone is not possible or even not legally allowed such as driving a car. In recent years, fairly new devices called smart glasses become increasingly popular, which offer great potential for several use cases. One of the most famous representatives of smart glasses is Google Glass, a head-mounted display that is worn like normal eyeglasses produced by Google. This paper introduces an experimental prototype of a Google Glass application for colorblind people or people with a color vision deficiency, called OmniColor and meets the challenge if Google Glass is able to improve the color perception of those people. To show the benefits of OmniColor, an Ishihara color plate test is performed by a group of 14 participants either with, or without the use of OmniColor.


Introduction
Colorblind people or people with a color vision deficiency (CVD) have to face many challenges in their everyday life. This involves tasks in which color is used as an information medium such as recognizing traffic lights while driving or determining the rawness of meat while cooking. Career choices are also limited, since a working color perception is prerequisite for some professions (e.g. train director) [1]. The usage of mobile devices such as smartphones together with applications that provide color correcting or color highlighting methods can support affected persons. However, those devices also come up with some limitations. Many tasks require a hands-free usage -it's not possible to drive a car with one hand while holding the smartphone permanently in the other hand. Hence, other mobile devices which are already worn by the user and allow hands-free navigation suit better for such purposes. Smart glasses are mobile devices that are worn as normal glasses and provide a new platform to assist people with a limited color perception.

Color Vision
Joblove and Greenberg [2] describe color as a function of the spectral distribution of amplitude and wavelength. The retina of the human eye has two light sensitive bodies, which are called rods and cones. Under normal light conditions, visual information is provided by cones. Rods on the other hand are used to perceive visual information in dim light where cones are insensitive. According to Huang et al. [3], normal color vision is based on the absorption of photons, which are processed by the three different types of fundamental photoreceptor cells, which are called cone cells. Furthermore, these cone cells or cones can be classified in three different types depending on their different spectral sensitivities: The long-(L), middle-(M) and short-(S) cones, which form the LMS color space. Although the particular receptors of the LMS triple overlap, each of those cones has a peak response in different wavelength regions of the spectrum: red for the long-, green for the middle and blue for the short cones [2]. Olivera et al. [4] stated, that the human eye consists of plenty more rods than cones. With the discovery of visual pigments in the nineteenth century, it was founded out that rods and cones contain a light-sensitive protein called rhodopsin, which allows color vision. Wakita and Shimamura [5] stated that humans with normal color vision can be referred to as trichromats. Color stimulus for light can be given by a numerical integration: Whereas ! describes the wavelengths, " is the spectral power distribution of light and l L , l M and l S are the spectral sensitivities for the long-, middle and shortwavelength cones.
Color Vision Deficiency (CVD) can be caused by internal and external factors [6]. Internal factors are defined to be intrinsic to the user (e.g. genetic causes or acquired CVD), external factors are caused by environmental or situational issues outside the user (e.g. lighting levels). If the retina of a human eye does not have specific cone types, one can speak of a genetic CVD. The genetic information for the presence and sensitivity of those cones are determined by the human X chromosome, which can have different variations. Men only have one X chromosome, whereas women have two. Therefore, men are more likely to have a CVD than women. A distinction is made between the lack of one of the cone types and variant forms of dysfunctions of cone types that result in a shifted peak of wavelength sensitivity. The lack of one cone type is called dichromacy whereas the dysfunction of one or more cone types is called anomalous trichromacy. The lack of two cone types (cone monochromacy) or three cone types (rod monochromacy) is possible, but very rare. Jefferson and Harvey [7] define the three main types of abnormal color vision as anomalous trichromatism, dichromatism and monochromatism. Furthermore, anomalous trichromatism can be separated in protanomaly (abnormal L cone) and deuteranomaly (abnormal M cone). The color perception for people which are affected by protanomaly or deuteranomaly can range from slightly normal vision to nearly dichromatic. Dichromats, which have a lack of a specific cone type, are classified as protanopes (missing L cone), deuteranopes (missing M cone) and tritanopes (missing S cone). Beside genetic CVD, acquired CVD is caused by a damage of the vision system [6]. This can occur by events an accident or disease. Since there are plenty less short-than mid-and longwavelength cones, the chance of getting a color perception similar to tritanomaly or tritanope is much higher by an acquired CVD.
Color-blindness Tests help to diagnose color-blindness or a CVD. Different tests came up in the past century. The most famous is the one described by Dr. Shinobu Ishihara [8]. According to [4], the Ishihara color test was developed as a color perception test for military recruits. Nigam and Bhattacharya [9] stated that the tests consist of a number of plates which contain multiple colored dots. The dots show hidden objects, where an object may be a number, alphabets or any other shape. These plates are often called Ishihara plates and are well known to diagnose color-blindness for dichromats that have red-green and blue-yellow CVD. Figure 1 shows an example of an Ishihara color plate. While humans with normal color vision should be able to recognize the number six on the plate, whereas protanopes and deuteranopes are normally not able recognize the number (depending on how strong their CVD is). Another test for color-blindness was presented by Farnsworth [11] and is called the Farnsworth-Munsell 100-hue test. Meyer and Greenberg [12] describe the test as a widely-used color vision test administered using physical color samples. The goal for the test subject is to rearrange a set of colors into a continuous color sequence. This set of colors form a continuous hue circuit when ordered correctly. However, after the rearrangement, an error score is computed for each sample. Protanopes, deuteranopes and tritanopes have different lines of confusion, which where plotted together with the chromaticity of the color samples. Figure 2 shows an online implementation of the Farnsworth-Munsell 100-Hue Test [13]. Color-correction Methods came up in the past century. Numerous algorithms and methods to correct images and/or video streams where discovered. This work only treat two of the most common used methods, although there are plenty more methods that can be used to adapt images according to the viewer's needs.
Daltonization. Anagnostopoulos et al. [14] describe daltonization as a process to modify an image or a sequence of images in a way that the color perception of colordeficient viewers is improved, which is first introduced by John Dalton [15]. The key idea behind this approach is to use the lost information by simulating a CVD which can then be used to improve the original image [16]. Hence, the lost information is used to map colors of the original image to wavelengths that are perceptible by people color-deficient viewers, resulting in shifted colors the viewer can see. The daltonization algorithm [17] basically consists of four steps: • First, the RGB coordinates are converted into the LMS color space, which is a color space suitable for calculating color blindness represented by the three different types of cones of the human eye. • Next, a simulation of color blindness is achieved by reducing the colors along a dichromatic confusion line, the line parallel to the axis of the missing photoreceptor, to a single color. • Then a compensation for color blindness is accomplished by shifting wavelengths away from the portion of the spectrum invisible to the dichromat, towards the visible portion. • At last, the LMS coordinates are converted back to RGB color space by using the inverse matrix of step one.
Color Contrast Enhancement. Another method is presented by Khurge and Peshwani [18], which is called color contrast enhancement. The RGB values of an image are adjusted in a way that the contrast is enhanced between confusing colors.
Thus, deuteranopes have a lack of green cone cells and therefore green pixels are adapted to appear bluer. The algorithm can be summarized by performing three steps: • First, the red component of the RGB values is increased relative to pure red. Reds that are further away from pure red are increased significantly, while reds which are close to pure red are only increased marginally. • Second, the green component of the RGB values is increased relative to pure green. Therefore, the same procedure is applied with green as with red in step one. • Finally, more contrast is produced by reducing the blue component for pixels that are mostly red and increasing the blue component for pixels that are mostly green.

Smart glasses
According to Due [19], smart glasses are worn by the user on the head like normal glasses. Beside their main purpose to display information, they can also be used to take pictures or to record videos. Their technology in terms of optics is usually based on a Heads-Up Display (HUD), Head-Mounted Display (HMD) or an Optical Head Mounted Display (OHMD). Independent from the underlying technology, the user of smart glasses is able to see a digital world (online) and a physical world (offline). Pentland [20] stated that the biggest advantage of smart glasses is the fact, that unlike smartphones and other wearables including body integrated technology, which come with need to look away from the actual point of view (normally on a display) in the physical world, these devices are already worn and allow the user to look at the physical world parallel to immersed, digital world. However, in order to recognize a sharp projected image, the area of interest has to be focused. This means that it is not possible to apply the same focus on the physical world as on the digital world at the same time. Also, the context switch between the two worlds need some form of trainingpeople often feel unfamiliar with smart glass when they wear the device for the first time. Another big advantage when compared to smartphones is that smart glasses can be used hands-free.
Google Glass as an example of smart glasses is used within this study. Brusie et al. stated that the big American company Google presented its first head mounted wearable device in April 2013. [21] In the beginning, the device was not available for the public. With the Explorer Project in 2014, Glass could be bought for 1500$. Nguyen [22] argues that typically for mobile device, it is rich on sensors and became popular cause of its growing API support and the huge community. Glass comes with a screen resolution of 640x360 pixels, which is equivalent to a 25-inch display from eight feet away. For user input, there is a touchpad located on the right side of the device which supports multi touch gestures. There is also the possibility to control the device with head gestures or voice input. The latter two input methods allow hands-free interaction. Additionally, Google Glass has an ambient light sensor, an internal compass sensor and a proximity sensor. For connection purposes, the device supports Bluetooth 4.0 and has an 802.11 b/g wireless module. It also features a camera that is capable to take photos with 5 Megapixel resolution and record videos up to 1280x720 pixels (720p).

Research Design
Out of the numerous possibilities to create the Google Glass application for colorblind people or people with a CVD, it was decided to create this work according to the method given by experimental prototyping as classified by Floyd [23]. Mayhew and Dearnley [24] describe experimental prototyping as the process of building a prototype or a proposed solution to a particular problem. Therefore, a prototype is created and then evaluated by experimental use. The functionality of the prototype could contain either the full-or a smaller, interesting subset functionality of the target system. At first, core functionalities where elaborated and described. For this purpose, related work where searched and evaluated to get an idea what possibilities exists to create a color correcting application with Google Glass. After the prototype had been developed, it was evaluated in form of a study by testing the functionality with a group of participants. The following research questions should be answered: RQ1: Are smart glasses already suitable for image correction or image enhancing and where are the limits of those devices? RQ2: How can Google Glass improve the color perception for colorblind people or people with a CVD?

Prototype
This section describes the prototype and the used algorithm within this study.

OmniColor
The prototype of the Google Glass application to correct images according to the needs of people with colorblindness or people with a CVD is called OmniColor. It works asynchronous and not in real-time. The implementation is based on so called Immersions, which are basic design elements or more precisely representing canvases that contain other application components such as headers and images. However, Immersions offer the most custom experience. Live Cards are other basic design elements on which components can be drawn, but are more likely used for applications which need to be rendered several times in a second such as games. These UI techniques are suggested in the Google Glass developers design documentation [25]. To actually perform pixel based operations that are needed for the daltonization process, a standard library for computer vision called OpenCV is used. The application can be controlled either via touchpad gestures or voice control. Touchpad gestures conclude left and right swiping for choosing the correct CVD type, single finger tap for confirmation and a single finger swipe to go back. Thus, the process to correct an image with OmniColor can be described by the following steps: 1. Starting at the Home screen of Google Glass, one has to get to the application list by performing a single tap on the touchpad or use the voice command OK, Glass. 2. Next, The OmniColor application has to be opened by performing a single tap on the touchpad or use the voice command Omni-Color.
3. Now that the application is opened and initialized, the menu to select the desired CVD has to be confirmed by performing a single tap on the touchpad or via the voice command OK, Glass. 4. With the menu being opened, the desired CVD is chosen by using the forward and backward swipe gesture and accepted by performing a single tap on the touchpad or using the voice command according to what CVD should be chosen (e.g. Deuteranopia). 5. The next screen shows a prompt to take a picture by performing a single tap on the touchpad or use the voice command. 6. After a short space of time, the taken picture is shown to the user who has now the possibility to accept the image by performing a single tap on the touchpad or to get back by performing the swipe down gesture. 7. Since all computations are performed directly on Google Glass rather than use offloading to perform computational expensive tasks in the cloud or another device, the processing will take some time. The user gets a hint that the device is now calculating the result image according to the previous specified color vision deficiency. 8. Finally, after the result image has been computed, it is shown the user and stored on the device. The user can now get back and take another picture without having to choose the color vision deficiency another time. Figure 3 shows the screen flow of OmniColor and the color corrections of a sample picture. http://www.i-jim.org

Daltonization
OmniColor uses a slightly modified version on the standard daltonization algorithm. Although there are numerous ways to correct an image for colorblind people or people with a CVD, daltonization was chosen because of the fact that it acts by shifting colors, but attempts to preserve color and information. While confusing colors are corrected according to the different CVD types, other colors only change marginally and stay most widely unaltered. People with a normal color perception will only consider small changes when comparing the result image with the original one. The algorithm is also known for good performance in comparison to other color correcting methods, since each pixel has to be only processed once and can be computed by matrix multiplications rather than calculating integrals. This is an important design criterion for almost all mobile devices. Daltonization can be used for all three different CVD types, which is another big advantage. Tanuwidjaja et al. [1] also provided a daltonization mode in their Google Glass application Chroma. For simplicity reasons, values are rounded to two decimal places in this paper. The daltonization algorithm used by OmniColor could be described by the following steps: First the RGB values of the original image which are ranged between zero and 255 are transformed into the LMS color space. This is done by matrix multiplication as shown by equation (2).
Each different CVD type has its own color deficiency matrix. Those matrices are respectively S deut for deuteranomaly given by (3), S prot for protanomaly given by (4) and S trit for tritanomaly given by (5).
iJIM -Vol. 11, No. 5, 2017 In the next step, the CVD type is simulated by multiplying each pixel of the LMS converted original image with the matrix S x (according to which type is needed), which results in new LMS values (L sim , M sim , S sim ) shown in equation (6).
The simulated LMS values are transformed into the RGB color space by a multiplication with the inverse matrix of the LMS color conversion in equation 2. This results in an equation given by (7).
In this step, compensation values are calculated by subtracting the original's image RGB values with the RGB triple given by (8).
Now, the necessary shift is computed to make the color more visible for colorblind people, resulting in R shift , G shift and B shift given in (9).
Finally, the resulting matrix of the Daltonization algorithm is calculated by multiplying the original RGB values with the shifted RGB values, as given in (10). Figure 4 shows 3 samples of corrected images, one for each type of CVD.

Evaluation
The prototype was evaluated with a qualitative user test.

Method
To evaluate the OmniColor Glass application, a printed 17 plate Ishihara color test containing only numbers was performed. The Ishihara test containing 24 plates is usually only applied for people that are not able to read numbers. Therefore, the test containing 17 color plates was sufficient. Each participant is tested individually, which means that an appointment for place and time was arranged for each person. First, all participants got a short introduction to Google Glass including navigation concepts and input possibilities. Next, a questionnaire has to be filled out by the participants, primarily to get information about their color vision (normal, color vision deficiency, color blindness). After that, the concept of the Ishihara color plate test is described by the examiner. Related to this description, OmniColor is presented by the examiner -the participants have the possibility to watch a screencast on a standard Android phablet while an example color plate is processed by OmniColor. For this purpose, an introduction plate is used, which can easily be recognized for colorblind people and people with normal color vision. Afterwards the participants have the possibility to do the Ishihara color test twice: The first time without the use of Omni-Color, the second time with the use of OmniColor. There is no time limit for the test nor is the examiner allowed to help the test subject on confusing color plates. The participants are required to read the number of the actual color plate and report it to the examiner. The reported numbers are then recorded by the examiner. Actually, there were three possible options on each plate: After the first test without the use of OmniColor, the participants are instructed to put on Google Glass and do the test again with the help of OmniColor. However, since the image processing on each color plate take a non-negligible amount of time, the test is shortened by only analyzing four color plates with OmniColor (typically those where the participant had problems to recognize the number). The other plates are shown on a tablet, on which each color plate is already preprocessed by OmniColor for each color vision deficiency. Participants with a normal color vision can choose the correction method they'd like to see, but they were recommended to use the deuteranopia or protanopia mode, since the image only change marginally. After the two tests had been performed, the results where compared and discussed with the participants. Figure 5 shows the execution of the Ishihara color test with the use of Omni-Color.

Results
To verify the usefulness of OmniColor, 14 participants were recruited over a threeweek period. Five of the participants have a diagnosed CVD and are males. While four people with a CVD know their color blindness type (protanopia/protanomaly), one person was not able to give this information at the time the questionnaire was filled out. After the first round of the Ishihara color plate test was taken it could be determined, that the person had most likely protanomaly too. Hence, 3 of the CVD group are strong-while the other two are mild-protanopes. No one of the CVD group wears glasses; hence no concern about this limitation was expressed. The age of the participants ranges from 28 to 48. The nine people with normal color vision had no problem to pass the Ishihara color plate test -either with or without the use of Omni-Color. However, two participants had problems on certain plates to report the number. After the confusing plates had been photographed multiple times, the numbers could be rightly recognized. All of the five participants of CVD group where able to significantly improve on the Ishihara color test with the use of OmniColor. The three strong protanopes went from strong to mild, while one of the mild protanopes went to nearly normal color vision. The other mild protanope only achieved a minimal improvement with the use of OmniColor. However, all participants of the CVD group found the application useful to determine the number better or even faster. Table I shows the Ishihara color test results for the five protanopes. The users had to recognize 17 different numbers. Table I shows the correctly recognized numbers with and without OmniColor.

Discussion
The overall results had shown that the OmniColor Google Glass application could lead to an improvement for people with a CVD. However, during the test phase of OmniColor, certain limitations and problems came up that need to be considered for further development. First, although the Ishihara color test is a standard test to diagnose a CVD and what CVD type the test subject have, it does not cover up the daily activity challenges for colorblind individuals such as determining the rareness of meat while cooking or choosing confusing colored clothes. For this purpose, real life scenarios should be summarized as user stories, which could then be performed in an extended test with colorblind participants. A good approach is brought by Tanuwidjaja et al. [1]: While all test subjects performed a series of general tests, a subgroup of the participants performed also some specialized tests like recognizing colored band of resistors. The CVD group consisted of only five participants, which is rather small for a convincing result. Therefore, more colorblind people or people with a CVD should be tested. The search for colorblind participants was really challenging. This may be caused by not willing the participants to be recognized as colorblind or be mentioned as so in any work. Also, it would be interested to test even one tritanope with OmniColor. While the OmniColor Google Glass application can be controlled either via touchpad or voice commands, nearly all participants used the touchpad for navigation purposes. Only one participant who was already familiar with Google Glass also tried voice commands. During the development of the application, the voice navigation worked great. However, since each participant is tested individually on different environments with different noise levels, it could be determined that the voice recognition suffers from high background noises. Nevertheless, the touchpad navigation worked correctly for each participant. Most of the participants felt comfortable by wearing Google Glass. Those people that regularly wear glasses had problems to adjust Google Glass according to their needs and ended up with putting Google Glass in front of their regular glasses. While it is OK to do that for a couple of minutes, none of them would wear Google Glass for a longer period of time. Google need to provide a version of Google Glass with prescription lenses that can also be worn comfortably by wearer of glasses. As Google Glass was primarily designed to be a notification device rather than a device for running compute-intense applications, the performance of OmniColor was criticized by some participants. Although no real-time approach was chosen for OmniColor, since images are processed and no video stream, the computation of the daltonized image took too long for a smooth user experience. But not only the actual computation of the output image took too long, the recommended way from Google to use an observer method to wait for the taken image to be processed took about three to four seconds [26]. For this purpose, an own logic which uses the Android Camera API could be used to accelerate the capturing process. To improve the performance of the image correction, offloading could be performed. Thus, either cloud services or offloading devices connected via network could be used to perform compute intense operations. Ha et al. [27] presented a similar concept called cloudlets, which can be described as a cloud service next to the user. To guarantee reliability, a chain of computation possibilities where prepared: If no connection with a cloudlet can be established, the connection with a normal cloud service would be established. If that also fails, the users carried offloading devices (laptop) with them, which perform compute-intense operations. If even that fails, all computations had to be performed on Google Glass, which lead to a significant performance decrease. The authors also stated that the CPU frequency of the device does not provide the full theoretically computation power of 1008MHz. Because Google Glass tries to minimize heat development, several lower CPU frequencies are used according to the performance needs -Either 300, 600, 800 or 1008MHz. The highest frequency is only used for a short amount of time to boost compute-intense operations. Some participants claimed that Google Glass got very hot while using Omni-Color, which results in an uncomfortable user experience. Most likely, these were participants who do not wear glasses. The reason could be the fact that the temple stem of glasses act as a separator between Google Glass and the head. If the device gets too hot, the current application is interrupted and a warning message will be shown to the user that Glass has to cool down for a moment. Although this never happened on any test of OmniColor, the device still produced a significant amount of heat. Since all image correcting computations are directly performed on Google Glass, offloading may be interesting concept to overcome that issue [27]. The camera provided by Google Glass also brought some limitations, mostly because of the fact that colors deviate from the true colors. Digital cameras often come with the functionality to calibrate the camera. However, Google Glass does not offer the possibility to manipulate the factory settings of the camera such as white balancing. Also, the Ishihara color test was performed on different environments and light conditions. One participant had problems to recognize the numbers on certain plates. After presenting the pre-taken images on the Android phablet, the numbers could be recognized correctly. Hence, the results of OmniColor may vary under different light conditions.

Conclusion & Outlook
This paper presents a Google Glass application for people with color blindness or people with a CVD. The reason why Smart Glasses and not ordinary mobile devices as smart phones were the target device group was that Smart Glasses can be used hands-free. This is a big advantage to solve everyday challenges for color-blind per-sons. It demonstrates how new technical devices in combination with state of the art application development could improve the color perception of those people (RQ2). Although Google Glass had several limitations that came up while testing OmniColor, it shows the great potential of smart glasses for special purposes (RQ1). The application can be used to solve non time-critical use cases such as distinguish mellow from green bananas or to select appropriate colored clothes or identify the color of a map legend. Since the converted images are stored, the images could be used as accessory to help other color-blind persons. The algorithm itself works as intended, the hardware limitations of the device should not be considered as a big issue since the performance of the available hardware evolves constantly. Further work may include the improvement of the overall OmniColor performance. In a first step, the logic to take pictures with OmniColor directly via the Android camera API should be modified. In the next step, cloud based offloading possibilities could be used to offload computeintense operations. The Google Glass Explorer Project ended on January 19 th 2015 [28]. Google stated that the company will still support Glass at Work and will work on a newer, better version of smart glasses in an independent unit out of Google X research lab. The Google Glass Explorer Project acted as a beta phase for Google, in which problems could be identified and improved in further development. As a future work, ports to other platforms and devices such as Microsoft's HoloLens [29] mixed reality head mounted smart glasses would be very interesting.