Smart Home Multi-Factor Authentication Using Face Recognition and One-Time Password on Smartphone

— Recently, the adoption of smart home technology has been on the rise and becoming a trend for home residents. The development of Internet-of-Things (IoT) technology drives the smart home authentication system with biometric systems such as facial recognition, fingerprint, and voice control techniques. In the context of homeowners, security is always the primary con-cern. However, conventional home security and the existing smart home security system have some limitations. These techniques use single-factor authentication, which provides limited protection for home security. Therefore, this project proposed a design for smart home multi-factor authentication using facial recognition and a one-time password sent to smartphones for a home security system. Rapid application development was the methodology for conducting this study. A usability evaluation suggested that the proposed smart home multi-fac-tor authentication is acceptable, but some usability issues can be improved in the future.


Introduction
At present, smart home technology is increasingly gaining popularity among homeowners due to the features offered by the system [1,2], including security protection and comfort. Moreover, it allows residents to remotely manage and control their houses [3,4] using mobile devices such as smartphones and tablets [1]. Nevertheless, home security protection such as door locks and surveillance cameras is always the main priority for homeowners to ensure the residents' privacy and safety from intrusions. However, unlike smart home systems, the conventional door locking system such as key and padlock on the grill door has low-security protection [5]. It is also inconvenient and inefficient because the resident needs to use a physical key for every identification process. Moreover, the system can be compromised easily and thus increase the rate of home burglaries through doors and windows. In addition, the careless attitude of residents who do not close the door and windows when nobody is at home is also a safety threat. Thanks to smart home technology that can address the problems in conventional door locking.
Home authentication systems are one of the components in smart home technology that controls the residents' entrance to the house [1]. It comprises the authentication mechanism and door security system, which only allows the access of authorised residents [6]. The advances in Internet-of-Things (IoT) and cloud computing technologies have increased the maturity and usability of smart home technology to a reasonable level. A smart home authentication system includes smart cards, biometrics, passwords, and radio frequency identification (RFID) sensor [7]. The use of a biometric-based door lock is highly reliable because it identifies an individual using their biological characteristics. These characteristics are unique for every people and cannot be duplicated easily. Further, the security of the smart home authentication system can be increased by applying more than one authentication technique, which is called multi-factor authentication. For example, the prevalent use of smartphone technology [1,8] can be utilised with biometric authentication to strengthen the security system. In addition, combining personal identification numbers (PINs) or passwords sent to residents' smartphones during the authentication process can provide higher security protection than the traditional lock.
Therefore, this study intends to propose and design a smart home multi-factor authentication system using face recognition and one-time password (OTP) sent to residents' smartphones. It is a multi-factor authentication system that can provide double protection to smart home residents by enhancing entrance control to their houses. First, the study identified the system's requirements and then developed a prototype to demonstrate the design. Finally, the prototype was tested for its usability. It works as a smart door system integrated with a microcontroller. Facial recognition technology is the primary authentication system, combined with OTP. The system is beneficial to home residents because it enhances home security to a large extent and improves the convenience for the smart home resident. Furthermore, it makes life easier as residents only need to scan the face and input the OTP for entering their house. Overall, the advancement in IoT on the smart home authentication system has impacted society in changing a better-quality lifestyle and improving the user experience on the home security system today.

Background and related studies
This section described the background of smart homes, biometric systems, and the OTP approach. Besides, the related studies investigated the implementation of face recognition systems in various areas. Smart home technology is also referred to as home automation, enabling the residents to manage their homes remotely through Internet-connected devices such as smartphones or tablets [1,9]. The concept of a smart home was introduced by Nikola Tesla in 1898 with the invention of remote controls [10]. The first smart automation system was developed in 1966 that allowed users to create shopping lists and control room temperature and home appliances [10]. Today, the smart home is integrated with the IoT and cloud computing technology that could increase security. Therefore, it can provide home residents with a comfortable, convenient, secure, and high-quality home environment. The general idea of a smart home system [6] is demonstrated in Figure 1.  1. The components of a smart home system [6] Multi-factor authentication systems use biometric features with other types of authentications methods such as PINs, passwords, OTPs, and smartcards [11]. Biometrics can be categorised into two types, which are physiological biometrics and behavioural biometrics. The physiological biometric include face recognition, fingerprint, hand gesture, and deoxyribonucleic acids (DNA). On the other hand, behavioural biometrics include signature and voice recognition [12]. Fingerprint, facial recognition, and voice recognition would be the most familiar biometrics in our daily lives beyond the application of smart homes. For example, the banking sector used fingerprint recognition to increase the transaction security on the automated teller machine (ATM). Moreover, the biometric system is used along with the PIN to form double protection of smartphones. Other than PINs, OTPs are also frequently used in multi-factor authentication. It is a set of numerical or alphanumerical codes generated for every transaction or authentication and valid only once [13]. For example, OTP is often used in Android apps. A server generates a random number of OTP and sends it to the user's smartphone via a short message service (SMS) or push notification [14]. Developers suggest six OTP rules for secure implementation. First, the system must guarantee randomness in generating an OTP for authentication [15]. Second, it must generate at least six digits of the OTP. Third, the system must limit the attempts for OTP validation. Next, OTP is only used once for every authentication process. Then, OTP should only be valid for a limited time. The sixth rule is that the OTP value can be renewed if it expires. Finally, the OTP should be more reliable than the password to protect against replay attacks [14].
Facial recognition is gaining attention as an authentication method in various domains, including smart home environments, due to efficient recognition devices and algorithms. For example, Sandar and Oo [16] proposed a face-recognition door lock system using Raspberry Pi and GSM modules. The authentication process started when the camera detected and recognised the face. Then, the user must enter the password, and the correct password will unlock the door. On the other hand, if the incorrect passwords were input more than three times, the GSM module sends the alert message to the admin user. It also used the Haar cascade classifier for face detection because of the high detection accuracy. Furthermore, the local binary pattern histogram in OpenCV was used for face recognition because it is more flexible to recognise the side face. It is an instance of multi-factor authentication that is more secure and suitable for homes, banks, or other public areas.
Manjunatha and Nagaraja [17] proposed a home security system using face recognition and an intruder detection system with automated alerts. The system captures an image of an intruder using a web camera and sends it to the homeowner through an email. At the same time, an SMS alert is sent to them. Utilising the intruder alert system with email and SMS makes the system more secure to protect the house from unauthorised people. The user can notice the intruder and react when they receive the notification immediately. Aishwarya [18] proposed an android based real-time smart door lock system with image detection and voice recognition. The system was integrated with IoT and used smartphones to control the door. First, the camera detects the face image and sends it to the user application for image verification. Then, the authorised voice command is used as an input to unlock the door. In addition, the OTP is generated and send to the registered mobile number for every registration process. In this system, the convolutional neural network of deep learning was implemented to train and recognise the images and the authorised user's voice. The multi-layered network increased the model accuracy during the recognition process. However, voice recognition would not be as necessary as the user can control the door using smartphones. Another possible way could be clicking on the smartphone notification to make the authentication process more convenient.
Face recognition and OTP have been implemented in the banking sector for ATM transactions [19]. First, the user inserted the ATM card into the machine. Then, the user can choose either face recognition or OTP mechanism for authentication. The advantage of the system is that the cardholder can assign other people to make a transaction on the ATM while necessary. In addition, iris recognition technology can be implemented to allow the system to work more accurately at night. Manish et al. {Manish, 2020, Card-Less ATM Transaction using Biometric and Face Recognition-A Review.} proposed a card-less ATM transaction with a biometric system. The user can start the transaction immediately with fingerprint recognition and the OTP approach without an ATM card. It provided more convenient and comfortable ways for the user to make a transaction. In addition, both studies by Singh et al. [19] and Manish et al. [20] aimed to prevent card fraud activities because only the authenticated person is allowed to make the transaction. Varshitha and Shivanand [21] proposed an Android-based voting system using face recognition and OTP. There are three steps of authentication for valid voters. The first step is to login into the application to verify users' personal information on the smartphone. Then, the face recognition process is executed to authenticate the voter's identity. The last step is that the voter received the OTP on the smartphones and entered the OTP. The voter is allowed to vote if the correct OTP is verified. The user status is changed to yes after voted. It indicated that one voter could vote only once and thus avoid the duplication of votes. The voters can vote using smartphones, which is more convenient and time-saving than the traditional systems.

Methodology of the study
The study adopted the rapid application development (RAD) [22,23] methodology. It consists of four phases requirement planning, user design, construction, and cutover, as demonstrated in Figure 2. The requirement planning involved gathering the information and identifying the user requirements to develop the multi-factor authentication for a smart home. The requirements for this system were gathered by analysing documents and conducting a survey using social media networks to random homeowners in Malaysia in January 2021. The result of the survey was analysed to understand the design and the functions that the system needs. Finally, the requirements were documented and demonstrated using a use case diagram, flow chart, and circuit diagram to construct a user design. The user design process is iterated and conducted in parallel with the construction of the prototype. The prototype construction process is also iterated to reflect changes in the user design [23]. The software and hardware architecture, including the programming language used and the database storage, were specified during the prototype construction phase. RAD emphasises the prototype's iterated development in obtaining the system's requirements so that the entire development process is faster and more flexible [23]. During the cutover phase, users were involved in evaluating the system functionalities and performance. A post-task questionnaire [23] containing the demographic information and the system usability scale (SUS) [24][25][26][27] were used to evaluate the usability of the proposed design. The description of the post-task questionnaire is summarised in Table 1. Selected respondents were also interviewed for further feedback and identifying issues. Finally, the user's issues were addressed to enhance the system's usability. Thus, it helps to improve the users' satisfaction and produces a more reliable multi-factor authentication for the smart home. Table 1. The content of the post-task questionnaire

Section Topic
A ▪ Demographic and background information on home security and face recognition technology today. ▪ Closed-ended question. ▪ The total number of questions is 8.

B
▪ After each task is demonstrated, the respondents self-rated their opinions using the System Usability Scale (SUS) [24][25][26]. ▪ A 5-point Likert Scale is used. ▪ The total number of questions is 13.

C
▪ Interview questions. ▪ The total number of questions is 5.

The design of smart home multi-factor authentication
The design and development of the smart home multi-factor authentication are described in this section. Two steps were involved in the requirement gathering process. The first step was analysing documents and information from Internet resources. The relevant contents were searched using the keywords and documented to construct the requirements in developing the smart home multi-factor authentication. The second step was conducting a survey on a social media platform like Facebook and WhatsApp. A survey questionnaire was created based on the information collected from the analysed documents in the first step of the requirement gathering process. A total of 100 respondents participated in this survey to provide their opinion regarding the face recognition system. Most of the respondents were students aged between 21 to 30. 57.7% of the respondents were familiar with face recognition technology, and 53.8% were satisfied with today's face recognition application. The survey result was then transformed into a list of requirements specifications, as shown in Table 2. The requirements consist of three major requirements, 'Configure', 'Register', and 'Authenticate'. The priority of the requirement is indicated by M (Mandatory), O (Optional), and D (Desirable). The system shall be able to verify the OTP input by the user. M 3.8 The system shall be able to notify if the user correctly inputs the OTP. M 3.9 The system shall be able to notify if incorrect OTP is input by the user. O

3.10
The system shall be able to display an alert message if incorrect OTP attempts more than three times.

O
The requirements are then translated to a use case diagram, one of the behavioural diagrams in UML, as shown in Figure 3. The diagram describes the interaction between the user and admin user with the smart home multi-factor authentication system. First, the admin user handles the 'Configure' use case. Meanwhile, the user and admin user execute the use case of 'Register' and 'Authenticate'. 'Authenticate' use case allows the system to recognise the user's face image and verify OTP. However, the process of face recognition and generate OTP must be executed before the verification of OTP. In addition, the user can receive an unsuccessful message during the failure of the face recognition and OTP verification process. The process of the smart home multi-factor authentication system is described in the flow chart in Figure 4. It allows the user to understand the operations of the system quickly.  The hardware components are configured in a casing, as shown in Figure 6. The white box indicates the door locking system, and the solenoid lock demonstrates the door lock in the resident's house.

Evaluation
Multi-factor authentication recognises authorised users according to their biometric characteristics and is combined with other authentication factors like smart cards, passwords, PINs or OTP [11]. Thus, it increases the system's security by making it difficult for criminals to steal identities or pretending to be someone else. However, usability has been the main challenge of multi-factor authentication. It is always the tradeoff of security and vice versa [11]. In other words, increasing the security of a system through authentication would also increase users' steps to authenticate themselves, reducing its usability. On the other hand, usability ensures users use the authentication system in the most efficient way [28]. Therefore, efficiency is essential for an authentication system's usability [29]. Kaur and Mustafa [29] suggested the following characteristics that can represent it (1) users' effort to authenticate themselves, (2) time to authenticate an authorised user, (3) memorability of the authentication methods and (4) learnability of using the authentication system.
In evaluating the usability of the multi-factor authentication using face recognition and OTP on a smartphone, the study qualitatively analysed the system based on these usability features.
1. Users' effort to authenticate themselves involves three steps, facing the camera to detect the face image, opening the text message on the smartphone, and keying the OTP on the keypad.

Time to authenticate an authorised user would vary depending on the quality of the
Internet connection for sending the OTP to the user's smartphone. Currently, it uses the cellular network to connect to the Internet. So, for example, users would receive the OTP on their smartphone within 3 seconds within an excellent GSM network connection. 3. Memorability of the authentication methods would be the most valuable part of the system as it does not require users to memorise another set of passwords or PINs. Therefore, there is no risk of a forgotten password or PIN. 4. Learnability of using the authentication system is very simple. The authentication system has an LCD screen that displays messages during the authentication process. First, it asks the authorised users to face the camera and then checks their phone for the OTP and finally key in the OTP.
The qualitative analysis of the multi-factor authentication using face recognition and OTP on a smartphone demonstrates that the proposed authentication system meets usability features. Nevertheless, it depends highly on the Internet connection to send the OTP to the authorised users' smartphones.
An evaluation was conducted to assess the system's usability and performance through a one-to-one session with 30 respondents. During the evaluation, the researchers met the respondents face-to-face or through the Zoom meeting platform. In addition, a post-task questionnaire (as described in Table 1) was distributed to the respondents at the end of the evaluation session.
The respondents consisted of 18 males and 12 females. 76.7% of them were between 21-30 years old. 60% of the respondents were students, 26.7% worked in the private sector, and 13.3% were self-employed. Regarding the type of door lock used in the respondent's house, 73.3% of them used the key and padlock on the grill door, 16.7% used the passwords or PINs, and the other 10% used the access card to unlock the door. In terms of attitude towards home security, 53.3% of the respondents were moderate conscious of their home security which they always lock the door. 43.3% were very conscious and always double-locked the door, and the other 3.33% of respondents were relaxed and sometimes forgot to lock the door. Next, most respondents used the face recognition application to unlock their smartphones or tablets and the temperature scanner at the shopping malls. Ten respondents never used the face recognition application in their daily life. A minority of the respondents had used the face recognition application, including unlocking the door, attendance system in school or workplace, airport boarding, and border check-in. Besides, 50% of the respondents believed that face recognition could provide better home protection. On the other hand, 26.7% of the respondents claimed that it is less secure than the other door locking system, and the other 23.3% of them were not sure regarding the security protection by the face recognition. In short, a few people doubted the security protection provided by facial recognition technology.
As mentioned in Section 3, SUS was the primary tool used to evaluate the usability of the proposed design. It used a five-point Likert scale, as shown in Table 3. The points were used in the calculation of the final SUS score. The respondents were divided into two groups according to their prior experience using face recognition technology in their daily lives. Group A comprised twenty respondents who had prior experience in using facial recognition technology. On the other hand, Group B comprised ten respondents with no experience of using facial recognition systems. The formula to calculate the final score of SUS [27] is defined in Equations (1-3).
where, N i = Number of respondents checked the scale S i = Points for each agreement Deduct one from the total score for each respondent for every odd-numbered question.
Final score = Total score -(1 × n respondents) (2) Deduct five from the total score for each respondent for every even-numbered question.
Final score = (5 × n respondents) -Total score The responses and scores of Group A and B are shown in Tables 4 and 5, respectively. The formula to calculate the SUS total score [27,30] can be referred to as the agreement chosen by the respondents with the corresponding point in Table 3. Finally, the complete result is shown in Table 6.     Table 7 illustrated the guideline for the SUS score in which the higher score indicates the system' better usability, and it is recommended to use. The overall SUS score for the group A respondent is 52.25, and group B respondents are 64.50. As referred to the SUS score guideline in Table 7, both scores were classified as poor and below the passing SUS score, 68. Thus, the scores reflected that the system is acceptable, but some usability issues can be improved in the future. Besides, the SUS score for the respondents in group B is greater than that in group A. It indicated that the respondents with zero experience on the facial recognition system are more satisfied with the system's usability than the respondents with facial recognition experience. However, the result might be inaccurate due to the different number of respondents in each group. The larger number of respondents in group A could have made the result more reliable and closer to the expected value. Thus, increasing the number of respondents for further analysis could reveal more information on the usability issue of the system.
Apart from that, 50% of the respondents agreed, and 23.3% strongly agreed that this smart home multi-factor authentication system would be an alternative to a conventional home locking system. However, 3.3% and 10% of the respondents strongly disagreed and disagreed that this system would replace the conventional system. The other 13.3% of them were not sure about this statement. In short, most respondents agreed that this system would be an alternative to traditional home locking systems. Furthermore, 93.3% of the respondents rated the positive comments (agree and strongly agree) that they were satisfied with the overall ease of use of the system. In comparison, there are 6.7% of the respondents rated neutral. Therefore, the conclusion is that the respondents were satisfied with the usability of this system. Moreover, 60% and 36.7% of the respondents agreed and strongly agreed that they were satisfied with the time to complete the process. Meanwhile, there were 3.33% of them rated neutral. Therefore, it proved that the respondents were satisfied with the convenience of the system.

Analysis of the interview
The interview session was transcribed and analysed based on the two groups of respondents as mentioned above. Overall, the respondents' experience with this system was excellent and enjoyable, but there would be some displeasure due to the online evaluation. It is also beneficial of experience sharing for the respondents who had never used face recognition technology. Besides, the respondents with zero experience of the facial recognition system were more concerned about the dependent factors that would affect the system's usability, such as the unstable internet connection, the electrical blackout issues, and the quality of the hardware components. For example, the Internet connection issue connecting the camera to the WiFi during the configuration step. For offline testing, the camera can quickly connect to the WiFi. However, it took longer to connect the camera with the WiFi while the online testing was conducted. The assumption can be made that the online meeting software had consumed a considerable amount of internet bandwidth and thus influenced the speed of the Internet. Meanwhile, the feedback provided by the respondents with experience with face recognition technology demonstrated that they were more concerned about the security issues and the improvement that can be made to this system.

Conclusion and future works
This research proposed a design of a smart home multi-factor authentication system using face recognition and OTP. The proposed design was implemented using an ESP32 camera, 3×4 keypad, 16×2 LCD and GSM module programmed on an Arduino microcontroller. Meanwhile, the relay module and the solenoid lock indicated the door lock in the house. The proposed design was tested to detect and recognise the face image on the camera. Then, an OTP sent to the user's smartphone is used to verify to unlock the door. The major limitation of the multi-factor authentication using face recognition and OTP on a smartphone is that it highly relies on the Internet connectivity used for the system. In the context of this study, it uses a GSM module for the Internet connection. However, it can be replaced with a home WiFi module for more reliable Internet connectivity in its actual implementation.
Nevertheless, the evaluation of the proposed design suggested that it can be further improved in several ways to expand the functionalities. First, it should allow several sample images taken during the enrollment process; hence, it could increase the classifier accuracy to identify and recognise the user's face. Second, a simple interface should be developed to allow a smooth enrollment process. It would allow the system to function more efficiently and provide explicit instruction to the user than a serial monitor. Next, an alert system using a suitable notification mechanism can be implemented when the camera detects an intruder's face image. Furthermore, an LED can also be added to indicate a successful face recognition process instead of relying on the message displayed on the LCD. Moreover, additional devices like lightning and tablet can be added to improve the system's user experience.