Customized Smart Glasses for Needy Blind People

Visually challenged individuals are one of the most concerned people in the society in terms of independence and confidence. There is a class of needy blind people who are not able to afford the technical devices designed for them.  A personalized smart device designed using deep learning technologies will be of greater help to them. A brief discussion was made with them to make the design of this device more user specific. Requirements gathered from this meeting are processed using deep learning methods and are composed into a smart glass using Raspberry pi. This glass is customized for every user such that it detects presence of known people around them and also does priority specific object detection. The compact, stand alone and economically efficient design of this product is welcomed by the needy blind people with great enthusiasm. This product will thus effectively increase their morale and ability to roam outdoors notably.


Introduction
Visual impairment is the lessened ability of the human retina to visualize a scenario. Visual impairment in worse cases leads to complete blindness. There are different categories of blind people. The foremost objective of this product is to help these visually impaired people. Artificial Intelligence is creating an exemplary shift in today's world. Deep learning is an accurate expansion of artificial intelligence. Its compilation of methods that was designed to match the efficiency and accuracy of the human brain. This project will design a customized smart glass using deep learning software, embedded in a raspberry pi. A survey was made with the targeted set of customers to analyze their day to day requirements. A comparative analysis of the existing products and existing research proposals will be provided in this paper followed by our implementation details.

2
Existing Systems

Existing products
This section gives a comparative analysis of similar existing products available in the market.

Literature survey and analysis
The technically aided smart glasses as proposed in the papers [1], lack in meeting the actual requirements of blind people. Their software algorithms had issues with respect to speed and efficiency. Paper [2] proposes a design of smart glasses and a vibrator band to intimate the blind user for any forthcoming obstacle. It does not support face recognition module. Paper [3] proposes a design for low budget smart glass using object detection software. It does not have a face recognition integration in its software. Paper [4] and Paper [6] implements a glass type solution for educational and indoor environments respectively. They do not provide an all-time aid for visually challenged. Paper [5] proposes a smart glass that implements face detection algorithm with an auditory system. This glass will be integrated with a bio chip implanted in the user's cornea. This proposal is a long and expensive aid for visually challenged people. Paper [7] provides an application of a wearable device that informs the blind person about any imminent obstacle along with a GPS aided navigation system.
A profound research was made with existing systems and proposals. Prevailing systems did not satisfy the claims of the needy blind people on a whole. The novelty of our product lies in the design and implementation of an aggregate, affordable and comfortable solution that meets all their needs.

Novelty of our Work
There are different sets of blind people in the society and not everybody requires the same solution. We decided to contemplate on the needy blind people. We met them in person and interacted with them to check the novelty of our idea and to compare our idea with the devices that were aided by them from the government. There were sticks that vibrated in the presence of obstacles. This vibrated even for small obstacles such as small stones which created a panic in them as such small obstacles on there could be ignored. Another device that reads bus numbers of any approaching buses was provided to help them in their public transport commute. This device only read out the bus number and did not read the alphabets present along with the numbers. The bus numbers will be differentiated for routes only in correspondence to those alphabets.
These devices increased their inferiority in commuting in the society. No common man is ready to help such needy blind people, leaving their busy work schedule. Hence, we decided to analyze their requirements and concentrated on building a customized solution for this section of blind people.

Requirement Elicitation
The classes of blind people and their requirements change respectively. A classification of the expected customers of our product is shown in the flowchart below: We concentrated more on the needy blind people and tried to customize our product according to their requirements.   The following are the requirements gathered for further processing, after interviewing with them:

36
http://www.i-joe.org • They wanted a device that lets them know about the approaching objects.
• They suggested that a device that finds the presence of any known people to them would be of greater help to them while seeking help. • They also wanted a device that detects traffic lights while crossing the road.
• They were not ready to leave their stick to any of the new modern devices as the stick served as their right hand wherever they went. • They wanted a device that intimates them about the presence of stagnant.
• Water or animal wastes or potholes on the road well in advance.

Proposed System
We designed a smart glass that meets most of the above-mentioned requirements in an affordable and efficient way. The design and flow of our proposed system is shown below.
This smart glass will be embedded in a microprocessor called Raspberry pi. This has a camera interface. The designed smart glass will contain the camera interface and will have a bone conduction headset attached at its trailing end. The camera will feed real time video to the software module.
Customized object detection will be the base software module of our project. The three secondary modules namely traffic light detection, face recognition and the primary object detection will work in an iterative manner. All the modules may be generalized as object detection on whole. But our project prioritizes the outputs and calculates the direction of its presence. This will then be read out in the bone conduction headset in the user's desired language. The working of each module will be explained in further sections.

Base architecture
The designed software works on a deep learning architecture called Convolution Neural Network. The CNN parameters were used in customizing the YOLO algorithm for object detection as proposed in paper [13]. Version 3 of the YOLO algorithm is customized in our project. The underlying CNN architecture of YOLO contains 53 layers. These 53 layers contained 3*3 and 1*1 convolutional and residual layer in a specific pattern. An average pooling layer is placed at the tail of the constructed layers to preserve the extracted features, ensued by the fully connected layer. All these layers will cite features such as curves and colors of objects in the image. A softmax function at the tail of the neural network will interpret multiclass sigmoid possibilities of multiple classes.

Proposed algorithm
The proposed algorithm focuses on three main requirements gathered from the targeted set of people. This algorithm will run iteratively to detect traffic light, known faces and the general objects. Thus, the base YOLO algorithm is customized into three modules for appropriate prioritization. Output: Bone conduction speaker that speaks out the name and position of the appropriate object.

Data augmentation and data preprocessing
Data augmentation is the process of creating and processing the data as required. Object detection that includes detection of traffic light and other general objects is a process that involves deep learning techniques. Hence for this we used a pre-trained dataset called the Common Object in Context (COCO) dataset. Entire images were trained with complete images and features were extracted for further processing. This was then examined for its mean average precision value on the base YOLO algorithm used.
Face recognition is customized to detect known faces. This module uses machine learning methodologies. The number of known people may vary for every person. Augmenting the faces of known people to each user makes our product more unique. Around 50 images of each person were collected in a named folder and were named with the person's name. These images were trained and face embeddings were extracted. These face embeddings were again classified for similarity using Support Vector Machine classifier. Thus, similar faces were classified and were taken for further processing. Both these datasets were combined and augmented appropriately for further processing into respective modules.

Software implementation
Object detection module was implemented using the YOLOv3 algorithm. When the detected object is a traffic light, then it will instantly be taken to the speaker module as traffic lights are of greater priority when we move outdoors. When the detected object is a person, it will be prioritized again and instantly be taken to the face recognition module to check for the presence of a known person. When both traffic lights and faces are detected in the same frame then traffic lights are given greater priority in our algorithm.
After detecting objects, we customized the algorithm to analyze the position of the object's presence. The bounding boxes drawn on detected objects will be considered for position calculation. Camera positioning will be considered to append the name of the detected object with two position coordinates namely "top", "mid", "bottom" and "left", "right" and "center".

Hardware implementation
All the above designed software is embedded in a raspberry pi. It had very limited support for graphics processing. The designed software was embedded into Raspberry pi 4 model along with its camera interface. The frame processing efficiency using this processor was high when compared to its previous models.
A bone conduction headset was also designed to provide a safe auditory system. This headset will conduct sound vibrations through ear bones such that it reaches the cochlea of the inner ear. Since the blind user will be wearing the headset all the time, normal headsets will cause damage to their inner ear. By using this headset, they can also hear other sounds from surroundings along with this sound. This bone conduction headset is designed using piezo electric transducers.

Output Analysis
This section contains output screenshots and some analysis details of the developed project.

Testing and Experimental Analysis
This product is developed with a social cause. Hence the basic implementation of this smart glass was tested at various levels. The analysis of the testing outputs is summarized below.
• The designed smart glass is a combined effort of various modules. Hence these modules were tested for efficiency during development. The developed software model gave an accuracy of 90% during detection and classification of object detection. The prioritization of objects to be detected at appropriate intervals is also done perfectly with minimal GPU support. Face recognition models are also trained to recognize known faces of an individual. This module separately gave 75% accuracy. The safe bone conduction based auditory system also gave high efficiency on testing. • This product was designed purely on a social cause. Hence acceptance of the designed smart glass by the targeted audience matters most. They welcomed the idea and motivation behind this product with much enthusiasm. A nominal rate for this product was also provided by them and they wished to buy this product in the near future. Fig. 11. Bar graph survey of preference of our product over other existing products.

Future Enhancements
First stage of development of our product, concentrated only on object detection part of the analyzed requirements. But the blind people were not ready to leave their stick. They also wanted a device to guide them relating to the presence of stagnant water, animal wastes and potholes. Hence the immediate future enhancement of this smart glass will be integration with a smart stick that contains a sensor to detect stagnant water and potholes on the road. Future improvements of the designed product are briefly discussed below.
• Installation of GPS module for more location related information. • Notification based system, integrated with IOT for enhanced safety. Any safety or emergency calls or notifications can be done for more user friendliness of the product. • More fast and efficient processing of real time video frames can be done by implementing the processing software in cloud environments.

Conclusion
India is home to 20% of the total blind population in the world. Not all the blind people get benefited by government aids. We concentrated on a specific class of blind people and analyzed their specific requirements. We tried to implement a smart device that meets utmost of their requirements with high end processing technologies. The accuracy and testing of this product gave efficient results. This smart glass when deployed as a commercial product, will become more cost efficient and beneficial to the customers.