Model-Based Quality Assessment of Internet of Things Software Applications: A Systematic Mapping Study

—The Internet of things (IoT) is growing at a fast pace owing to its vast applications in varied fields such as medicine, society, economy, and even the military. This growth cannot continue without establishing high quality. Over the past decade, interest in research for the quality assurance of IoT has gradually grown. However, the discipline is still evolving, and further research is required to investigate the various quality-related aspects. Although assessing the entire system is impractical, to assure the quality of IoT applications, various assessment levels are required. A well-known and established approach to mitigate this difficulty is to model the entire system or a few parts of it for the sake of assessment, which is known as model-based testing. To determine what has been achieved thus far and what is lacking in this direction, this paper presents an extensive study on the use of the model-based approach to assure the quality of IoT applications. The study systematically reviews papers published from 2009 (early publications on IoT) to 2019 that reported the explicit use of models to assess the quality aspects of IoT applications. As a result of an extensive search process, the paper presents the results of scanning and reviewing 390 published papers. Thus far, out of these, 54 studies used the model-based approach to assess at least one quality aspect of an IoT application. In addition to the several relevant research questions that have been addressed in this study, this paper also presents several new insights and approaches for future research.


Introduction
The Internet of things (IoT) is a grid of devices, vehicles, home appliances, or other items embedded with electronics, software, and sensors that enable these objects to connect and exchange data [1]. The number of applications that implement the concepts of IoT is increasing phenomenally. It is expected that the global market value of IoT will reach $7.1 Billion 2020 [2]. Owing to this rate of expansion, ensuring the quality of IoT applications before releasing them into the market is a must. Quality assurance (QA) is a systematic means to check whether a product or service meets the specified requirements and standards stipulated to achieve the desired level of satisfaction for stakeholders. Generally, stakeholders require an application to demonstrate several quality characteristics, such as performance, efficiency, security, or maintainability. These requirements are represented in the quality model. The quality model of software applications has been illustrated extensively in a series of standards within the ISO/IEC 25000, which is known as "Software Quality Requirements and Evaluation" (SQUARE) [3]. The ISO/IEC 25000 quality model covers eight characteristics with 31 sub-characteristics for ensuring software product quality. Figure 1 presents an overview of these characteristics. Although there are several analogies between conventional software and IoT software applications, multiple challenges may arise when the well-established methods of software testing and QA are applied to the IoT domain. For example, the distributed nature of the IoT system and unavailability of an implemented established system as a test bed is challenging. Motivated by the huge success of model-based testing (MBT) of conventional software applications, several studies have followed this approach to overcome the difficulties encountered in IoT software application testing. Currently, MBT is one of the promising approaches in this direction. Originally, MBT was a testing approach in which both the "test cases and expected results" were automatically derived from an abstract model of the system under test (SUT). MBT has been applied in the literature to test the functional and non-functional characteristics of a software product [4].
Unlike conventional software systems, research on the use of the MBT approach to assure the quality of IoT software applications is in an early stage because of the novelty of the IoT approach itself. However, there have been several attempts in this direction in the past decade. To this end, the goal of this study is to identify and evaluate the present status of model-based QA of IoT software applications. The well-known methodology of systematic literature study (SMS) is employed to gather data from papers published in the past decade and analyze them. Following this approach, the attempt is to answer several relevant research questions (RQs) to identify the current achievements, challenges, research gaps, and future directions.
The paper is organized as follows: the background and related works are described in Section II, the research method-ology is presented in Section 3, an analysis of the results is provided in Section 4, threats to the validity of the study are discussed in Section 5, and, finally, the conclusions are reported in Section 6.

Background and Related Work
During the past decade, significant effort has been spent in researching the field of IoT. One of the most significant areas of research is QA. A systematic mapping is the best means to survey the data gathered by researchers who have studied QA in IoT and identify the main challenges encountered.
An SMS aims to identify and collect relevant literature on a research topic to present the research qualitatively and quantitatively [5]. An SMS also gathers and evaluates the results of a research with the aim of providing a complete and detailed summary of the current literature relevant to the field of focus during a specific period. Based on literature [5], an SMS includes three primary stages. Figure 2 illustrates these stages in detail. The first stage is the planning stage, which produces a protocol for specifying the research questions and how to answer them. The second stage is the search stage, which identifies that strategy that will be adopted to select relevant published studies. The final stage is the review stage, which is used for extracting and synthesizing data.
This SMS focuses on the QA of IoT software applications. More specifically, the study focuses on how models have been used for the QA and testing of IoT software applications. Recently, several literature studies have been published concerning some aspects related to IoT quality. For example, Marwah et al. [6] have evaluated IoT implementation techniques by considering QA characteristics. Further, the study examined various techniques for implementing IoT. It concluded that most IoT techniques lack tools to support automation techniques. Foidl et al. [7] have outlined QA requirements of IoT into six categories environment, user, service level agreement, organization, security, and data management. Further, they attempted to describe the QA of IoT applications by subdividing it into four categories derived from the six QA requirement categories.

Fig. 2. Systematic Literature Study stages
Gary et al. [8] conducted one of the early studies and produced a systematic mapping of the quality for IoT applications. The study focused on quality of service (QoS) in IoT applications. This mapping study provided a view of the state of QoS approaches in IoT by assuming that QoS can be assured at different layers of the IoT architecture. However, the study focused on quality aspects from the perspective of QoS and high-level quality standards without going into the detailed methodologies of assuring various quality aspects. Later, Banerjee et al. [9] classified the quality data into two categories specification and conformance. The study also attempted to improve the quality of IoT by analyzing two daily life cases requiring a high level of quality. The study concluded that to ensure the quality of IoT application, three quality areas should be examined.
Concerning the quality of IoT applications, several map-ping and literature studies have been published recently. These studies focused on either the big picture of IoT quality or individual quality aspects. For example, Ahmed et al. [10] conducted an extensive SMS on most quality aspects covered by the ISO/IEC 25000 standard that are applicable to IoT applications. Lepekhin et al. [11] also addressed several quality aspects in their systematic mapping study on IoT challenges. Venceslau et al. [12] conducted a systematic mapping study on IoT semantic interoperability, which is a specific aspect of quality. Porras et al. [13] focused on security in his SMS, which is another aspect of IoT application.
Despite the various published studies on IoT quality, a gap exists in addressing specific methodologies to assess particular aspects of quality in IoT applications. A significant gap also exists in the literature on information about how to migrate quality assessment and testing methodologies to the IoT era. As mentioned earlier in this paper, MBT is a successful and well-established approach in conventional software testing. However, there is no SMS on how this approach has been used earlier within IoT software applications to assess some specific quality aspects. This study aims to fill these gaps and systematically study the MBT approaches used directly or indirectly in the literature to assess any quality aspect of IoT software applications. Research Methodology Following the well-known methodology of SMS [14][15][16], this study follows a threestage method to formulate relevant research questions (RQ), identify a search strategy, and specify the selection criteria. The following subsections illustrate these stages in detail.

Research questions
A set of relevant RQs is proposed and answering each one of them requires an effort to analyze the results based on our study methodology. The RQs were chosen with two main purposes to conduct a mapping of the published papers that proposed model-based QA and to review the content of the chosen papers in detail. Hence, this study entails more than just mapping publications. Considering this aim, the following RQs were chosen.

RQ2.1:
What are the quality assessment and testing levels addressed by the articles? RQ2.2: Which type of test techniques were proposed to generate the test cases? RQ2.3: What type of test bed was used to execute the QA test cases? RQ2.4: Which tools were used to generate the test cases for QA purposes? RQ2.5: What are the features of the testing tools? RQ2.6: What type of models were used in the QA process? RQ2.7: Which specific criteria of the ISO/IEC 25000 quality model have been addressed in the literature?

Search strategy
The study seeks to conduct an SMS on the model-based QA of IoT software applications. The search strategy is a systematical structural part of the SMS that utilizes keywords such as "Internet of Things," "model-based testing," and "quality assessment" for searching through the indexing databases. The search strategy also uses key terms from the search questions to obtain accurate results [17]. Here, the search scope should be identified to build an efficient search strategy that can return the highest number of related articles.
The search scope consists of more than one dimension, such as publication year and publication venue. The domain searched includes research studies that were published from 2009 to 2019. The search process was performed within four well-known electronic databases. The search items were journals, conferences, and workshop papers.

Search Method
To select primary studies, automatic and manual searches were used. Automatic search was performed by entering search strings into the search engines of electronic libraries. Manual search was performed by scanning and analyzing the articles after applying the inclusion and exclusion criteria. Well-known inclusion and exclusion practices from the literature [18] were used.

Identify relevant research
As mentioned previously, the automatic search process was performed on four electronic databases (see Table1), which are the most relevant libraries in computer science & engineering. These databases ensured high coverage of published research studies [19]. The selected libraries also provided support for text combinations via logical operators for automatic search. For automated search, the study used a combination of search strings to address the RQs. Within these search strings, more than one keyword was used. To be specific, the three keywords, "Internet of Things," "model-based testing," and "quality assessment" were used. To improve the accuracy of the queries, the study used the logical operators OR and AND to construct more meaningful strings for the search engine. The search strings used to search through the electronic databases and the respective number of outcomes of the search process are provided in Table 2. The total number of papers that appeared in the databases in the first stage was 4,570. As some studies could be included in more than one database, manual search was applied to identify duplicate studies and select the primary entries. As a result, 2,668 papers remained. IEEEXplore ("internet of things" OR" IoT") AND ("model-based testing" OR" MBT") (Internet of things applications" OR" IoT") AND ("quality assurance" OR" QA") 2,387 1,372 ACM ("internet of things" OR "IoT") AND ("model testing" OR "model software testing")("internet of things" OR "IoT applications") AND ("QA" OR "quality assurance")("internet of things" OR "IoT") AND ( "Quality" OR "Reliability" OR "Verification" OR "Validation" OR "Testware" OR "Testing Data" OR "Testbed" OR "Performance" OR "Security") 963 459 ScienceDirect ("Internet of Things" OR "IoT") AND ("Testing" OR "Quality" OR "QA" OR "Quality Assurance" OR "Reliability" OR "Verification" OR "Validation" OR "Testware" OR"Testing Data" OR "Testbed" OR "Performance" OR "Security" OR "Privacy") 435 380 SpringerLink ("Internet of Things" OR "IoT") AND ("Test" OR "model testing"OR"model software testing" OR "Testing" OR "Quality" OR "QA" OR "Quality Assurance" OR "Reliability" OR "Verification" OR "Validation" OR "Testware" OR "Testing Data" OR "Performance" OR "Security" OR "Privacy") 785 457 Total 4,570 2,668

Selection criteria
The papers collected using automatic search strings were included based on the titles, abstracts, and keywords of the papers in all electronic databases. As general search strings were used to ensure that any relevant paper was not ignored, the automatic search produced a large number of papers. Thus, the exclusion criteria aimed to reduce the number of papers based on their relevance. The study applies the inclusion and exclusion criteria on the selected papers.
• The following inclusion criteria were used:

IC1:
The papers have a direct relation to IoT software application. IC2: The papers which have been focusing on assessing and testing any quality aspect (based on ISO/IEC 25000 software quality model) of IoT software application. IC3: The papers which have been focusing on using model (i.e., model-based) for testing and assessing a specific quality aspect. IC4: The papers which have presented methodologies for evaluating any quality aspect of IoT software applications.

Data Availability Statement
This systematic study is reporting the list of final selected papers and the information assigned to each paper. This list is not attached here for the reason of space constraints. However, it is available online at https://docs.google.com/spreadsheets/SLSIoT

Analysis of Results
The data collected using the steps mentioned are analyzed to address the RQs of this study. The following subsections discuss the answer of each RQ in detail.

(RQ1.1 and RQ1.2) Ratio of publication growth
The papers selected over the past decade (2009-2019) were analyzed to determine the progress of published research. Figure 4 and Figure    In 2016, researchers started to investigate the QA of IoT software applications. To this end, the ratios of publications in 2016 and 2017 are 9% and 15%, respectively. In 2018, 15 papers were published, which is almost 28% of the total number of published papers, whereas, in 2019, a significant number of papers were published in this area (18 papers), which was approximately 33% of the total ratio. This increase in the publication ratio may be caused by several factors, such as the general increase of interest in IoT because of its wide range of applications and the necessity of QA for those IoT software applications that interface with life critical matters.

(RQ1.3 and RQ1.4) Geographical spread and nature of publications
Compilation of data extracted from authors' affiliations from the selected papers indicates that research activities in this field have been widespread. The contribution of countries was determined based on the authors' affiliations in each paper. Each country is counted once per paper, even if more than one author from the same country contributed to that paper. As it is clear from Figure 6, France and China were the most productive countries for publications in this area, with eight papers published by each of them. The USA and India rank after them with five and four published papers, respectively. Further, Italy, Brazil, Germany, Finland, and the UK also appeared in this ranking with three papers each. Another part of the publication analysis in this study is the publication type, such as conference, journal, and workshop. Figure 7 presents the results of this analysis. It is clear from Figure 8 and Table 4 that more than half of the selected papers (approximately 57%) were published in conference venues, whereas approximately 31% were published in journals and only 12% were published in workshops. The screening of papers published in conferences and other active fora in this research area is presented in Figure 8. As it is clear from the figure, the "IEEE World Forum on Internet of Things (WF-IoT)" published the highest number of papers in this research area.

(RQ1.5 and RQ1.6) The active author and his affiliation
As mentioned previously, this RQ aims to analyze the papers selected in this study to identify active authors and co-authors as well as their affiliations. The authors who have a large number of papers are Abbas Ahmad (three papers) and Bruno Legeard (two papers), and both of them were working in Easy Global Market; Fabrice Bouquet, affiliated to the FEMTO-ST Institute, and Elizabeta Fournere, working with Smart Testing Solutions and Services, with two published papers each, ranked second.

(RQ2.1) Testing levels
An essential RQ that was addressed in this study concerned the level of testing that has been applied for a quality aspect using the model-based approach. To address this question, the selected papers were scanned extensively to identify the level of testing used in the relevant published papers. The outcome of this analysis has been presented in Figure 9. It is clear from Figure 9 that most selected papers (approximately 27%) presented integration testing, approximately 24% employed system testing and approximately 18% presented unit testing. Further, approximately 11% introduced acceptance testing, whereas approximately 15 % did not address any specific level of testing.

(RQ2.2) Testing technique
A significant outcome of this study was the mapping of the model-based testing technique used within IoT software applications. An extensive analysis of the selected papers indicates that three approaches were used in the literature in which the test cases were generated. These approaches for test generation were based on a graphical model, mathematical model, and test bed. Among the selected papers, approximately 43% used a graphical model to generate test cases, approximately 18% used a mathematical model to generate test cases, and approximately 39% relied on an implemented model of the test bed to generate test cases. Figure 10 and Table 5 present the results of this analysis.

(RQ2.3) Test execution environment
It is also of interest to know which testing environment was used for model-based QA of IoT software applications. After scanning the selected papers, it is clear that either real devices or emulators were used as the test execution environment. It is also noticeable that a few published papers did not provide enough details of the test execution environment they used. Figure 11 presents a clear mapping of the test execution environments used in the selected published papers. It is clear from the figure that approximately 59% of the selected papers used emulators to generate and execute the test cases, whereas approximately 41% executed the test using real devices.

Fig. 11. Mapping of the test execution environment
During the detailed scanning and reading of the selected papers, several issues were identified regarding the test execution environment. These issues were particularly related to the authenticity of the emulator environment for generating and executing the test cases and the differences expected when running the tests on real IoT software applications in real devices. Table 6 summarizes these issues. Table 6. Issues found in the literature for testing of the software application within emulator and real IoT devices

Availability
The emulator is open and free software in most cases The use of real devices for testing is costly and it may raise the budget of the product and affects the delivery of the software product

Test case
The emulator is suitable for certain types of functional test case Executions The real devices deal with strict performance testing issues, in which it is difficult to get an indication by the emulator Battery state The emulator is not able to simulate the battery issues Real devices can inspect the performance of the battery

Memory issue
The memory storage level of the emulator tends to be far more than the real devices The memory level of the real devices tends to be far less than the emulators Test application In many cases, it is not possible to run all types of IoT software applications om emulators The real devices are by themselves support for the IoT applications

(RQ2.4 and RQ2.5) Model-based testing tools for IoT software applications
This RQ addresses the model-based test generation tools used for IoT software applications in the literature. It was noticeable during the detailed scanning of the selected studies that several published papers addressed the testing of IoT software applications without using any specific testing tools. It was clear that [22], [23], [35], [36], [40], [43], [45], [46], [48], [50], [53], [55], [69], [70] did not use any specific tools. Instead, the test cases were generated through empirical observation of the system during experimental studies. However, in 25 out of the 54 published papers, a testing generation or execution tool was used. Table 7 summarizes these tools and their features in detail. A modeling language with processes, protectors, channels, and simple variables that can check the model for standard and user-defined properties through a full or a partial investigation of the model's state space CloudSim [49] A framework for modeling and simulation of loud computing infrastructures and services.
Apache JMeter [72] An open-source Java-based tool that designed to load functional test behavior and measure performance.
Spec Explorer [20] A model-based testing tool from Microsoft that can generate the test automatically within visual studio or any other unit testing framework Wireshark [60] An open-source analyzer used to analyses software and communications protocol development Petri net [44], [64] A mathematical modeling language for the description of distributed systems. A discrete event simulator targeted at networking research to provide substantial support for simulation of TCP, routing, and multi cast protocols over wired and wireless networks TTCN-3 [25] Testing and Test Control Notation version 3 is a well-known testing language that used in conformance testing of communicating systems F-Interop [62] A compliance test tool for interoperability of implementation and to explore various testing schemes and configurations by interconnecting devices under tests with the server testing tools JUnit [26] A Java-based unit testing framework that been influential in the development of test-driven development
• The effect of latency parameter over throughput for the application. For example, when the latency is high, the system spends more time in idle status, which reduces the throughput that will in turn effect on the degree of performance of the system. Performance has been addressed in this form in [30], [33], [37], [56], [60], [67]. • The performance is the stability, response time, transmission rate, and also the accuracy of the system. Performance has been addressed in these forms in [45], [58].
Compatibility aspect: Generally, compatibility is the ability of two systems or more to work together efficiently. From the analysis of the selected papers, there were only five published papers that used modeling for compatibility testing. Compatibility appeared as a significant quality aspect of IoT software application because usually the application is comprised of more than two systems [22].
Usability aspect: Usability is the ease of use of the device or product. For the IoT software applications, usability is addressed as the degree to which a software can be used by users to reach quantified objectives with effectiveness and satisfaction. During the analysis of the selected papers in this study, only one paper found using modeling to assess usability of IoT software applications.
Reliability aspect: Reliability is the extent up to which a specific function of a software system can be executed continuously and efficiently without failing. It is considered a significant feature of IoT software applications because a high degree of reliability is required for critical applications that involve human lives [76]. During this study, only six published papers used modeling to assess the quality of an IoT software application.
Security aspect: Within the IoT context, security is the ability of a software to function correctly under malicious attacks [43]. Security plays a strong role in the IoT era. Several research papers have been published focusing on this area. However, during the analysis of the selected papers, it was seen that a few studies used modeling to assess the security of IoT software applications. Only 16 papers have been published in this area.
Maintainability aspect: Maintainability is the ability to restore the operational software state with ease and efficiency after a failure occurs. This is a significant feature for IoT software applications. Generally, it is an essential requirement of stakeholders in the software. During the analysis of the published papers in this study, only four studies were found to use modeling to assess and test the maintainability of IoT software applications.

Threats to Validity
Like any systematic mapping or literature study, the validity of this study can be questioned. This can be attributed to various aspects such as the searching method used for identifying the published papers, filtering of results, inclusion and exclusion criteria, and possible bias in the data extracted.
One of the probable shortcomings of the research method-ology is the mode of collection of data from the databases. This study searched through only four electronic databases (i.e., IEEE Xplore, ACM digital library, ScienceDirect, and SpringerLink) to identify the selected papers. The main rea-son is that these electronic databases were highly relevant to the study domain. This limitation was overcome by performing several rounds of manual and snowballing search within the lists of references of the selected papers.
The filtering of the papers during the search result may be another drawback because it may relate to the authors' selection and preferences. To overcome this drawback, the authors of the present paper collaborated and performed the filtering independently to obtain different views and achieve non-biased results. Along the same lines, yet another limitation might be the inclusion and exclusion criteria, i.e., the decision regarding which papers were to be included and which were to be excluded. To address this limitation, the research questions were designed clearly and the resultant search strings were formed carefully. The study used well-formed criteria from other similar studies.

Conclusion
In this study, data from 54 published papers that address model-based testing for IoT applications were extracted. The final set of papers was selected after careful screening processes to select the relevant studies. We based our study on the ISO/IEC 25000 software quality model as a guide for the selected quality characteristics. In addition to a comprehensive mapping study presented in the paper, a detailed literature study has also been presented to analyze the con-temporary critical aspects in this area. Thus, the paper forms a base for new opportunities of research in this direction. Several useful analyses of the MBT for IoT can be noted in the paper, for example, the test case generation process, the testing tools used, the models used, and the significant quality characteristics.