Multimodal Interaction System for Home Appliances Control

Authors

  • Hanif Fakhrurroja Bandung Institute of Technology
  • Carmadi Machbub Bandung Institute of Technology
  • Ary Setijadi Prihatmanto Bandung Institute of Technology
  • Ayu Purwarianti Bandung Institute of Technology

DOI:

https://doi.org/10.3991/ijim.v14i15.13563

Keywords:

multimodal interaction, speech recognition, gesture recognition, smartphone application, smart home.

Abstract


This paper proposes a way to control home appliances using a multimodal interaction system such as speech, gestures, and smartphone applications. The sensor to capture speech, in the Indonesian language, and gestures from users are Kinect v2. Speech recognition process with the Google Cloud Speech, gesture recognition process with the K-Means clustering, and dialogue system process with the finite state machine. Users can also use the smartphone application to remotely control home appliances through mobile devices such as tablets or smartphones that are connected directly to the real-time database. There are two output responses from this system, namely the audio response generator to provide feedback to the user through the sound of the computer speaker and also provide an action to control home appliances use Esp8266. The average level of accuracy testing of interaction using dialogue systems and gesture are 92.5% and 79,25%. Interaction using dialogue systems is better than gesture. Smartphone applications can control home appliances properly.

Author Biographies

Hanif Fakhrurroja, Bandung Institute of Technology

Hanif Fakhrurroja received the bachelor’s degree in Physics from the Universitas Padjadjaran (Unpad) in 2003 and the master’s degree in Informatics from Institut Teknologi Bandung (ITB) in 2010. He is currently pursuing the Ph.D. degree with the School of Electrical Engineering and Informatics, Institut Teknologi Bandung (ITB). Since 2006, he has been with the Indonesian Institute of Sciences as a Researcher. His research interests include Human-Machine Interaction and Intelligent Instrumentation Technology

Carmadi Machbub, Bandung Institute of Technology

Carmadi Machbub is received the bachelor’s degree in electrical engineering from the Institut Teknologi Bandung (ITB), in 1980, and the master’s degree (DEA) in control engineering and industrial informatics and the Ph.D. degree in engineering sciences majoring in control engineering and industrial informatics from Ecole Centrale de Nantes, in 1988 and 1991, respectively. He is currently a Professor and the Head of the Control and Computer Systems Research Division, School of Electrical Engineering and Informatics, ITB. His current research interests include machine perception, optimization, and control.

Ary Setijadi Prihatmanto, Bandung Institute of Technology

Ary Setijadi Prihatmanto graduated with B.E. and M.S. in Electrical Engineering at Institut Teknologi Bandung in 1995 and 1998, and received his PhD in Applied Informatics from Johannes Kepler University of Linz, Austria in 2006. He is an associate professor & lecturer of School of Electrical Engineering & Informatics, Institut Teknologi Bandung since 1997. He is also the president of Indonesia Digital Media Forum since 2009. His main interests are Human-Content Interaction, Computer Graphics & Mixed-Reality Application, Machine Learning & Intelligent System, Intelligent Robotics, and Cyber-Physical System.

Ayu Purwarianti, Bandung Institute of Technology

Ayu Purwarianti is a lecturer at Institut Teknologi Bandung, Indonesia since 2008. She was graduated from Institut Teknologi Bandung for her undergraduate and master degree. She received his doctoral degree at Toyohashi University of Technology in December 2007 with research topics of cross language question answering. She has interest in computational linguistics, especially for Indonesian language. She has written several publications in conferences and journals related with computational linguistics for Indonesian language. She also provides Indonesian natural language processing tools to be used by other researchers.

Downloads

Published

2020-09-11

How to Cite

Fakhrurroja, H., Machbub, C., Prihatmanto, A. S., & Purwarianti, A. (2020). Multimodal Interaction System for Home Appliances Control. International Journal of Interactive Mobile Technologies (iJIM), 14(15), pp. 44–67. https://doi.org/10.3991/ijim.v14i15.13563

Issue

Vol. 14 No. 15 (2020)

Section

Papers