Evaluating Smartphones for Infrastructure Work Order Management

Infrastructure managers require timely and accurate state information to diagnose, prioritize, and repair the substantial infrastructure assets supporting modern society. Challenges in obtaining sufficient information can often be attributed to inadequate data collection procedures (phone calls, paper reports, etc.) or a general lack of knowledge or ability on the part of the reporting individual to accurately convey what is actually wrong with the facility. Fortunately, modern smart-phone technology offers the potential to improve maintenance work requests by providing better geolocation and problem description accuracy. An experiment simulating real-world maintenance requests was conducted comparing smart-phones with traditional verbal work order request systems. Usefulness and description accuracy ratios revealed smartphone systems generated more useful information regardless of submitter background or experience. However, interestingly the smart-phone applications did not improve asset geolocation and actually negatively impacted the ability of maintenance personnel to accurately relocate the asset needing service. Given the ubiquitous nature of smartphone technology, the potential exists to turn any citizen into an infrastructure sensor. This study takes a step toward understanding the benefits, as well as the limitations, of the smart-phone based work order submission systems. Keywords—mobile phone, application, work-order system, infrastructure


Introduction
Given various infrastructure assets seemingly fail on a daily basis, rapid identification and immediate reaction is the critical antidote toward guaranteeing sound facilities for continued operations [1]. To facilitate timely response, more efficient methods must be developed to quickly submit, diagnose, and respond to problems as they arise [2]. The advent of smart-phone technology, accessible to a majority of the population, has led to the development of smart-phone based work order management applications designed to take advantage of inherent advantages of smart phones [3]. The assumption is anyone carrying a smartphone can become an infrastructure sensor and report asset problems regardless of maintenance experience. Unfortunately, little research has directly explored the effectiveness of these applications in any rigorous way. Therefore, the following study proposes to answer two crucial questions about the technology: 1. Do smartphone-based mobile work order submission applications allow non-maintenance oriented individuals to transmit more useful information than traditional systems? 2. Will an increase in useful information enhance maintenance effectiveness in terms of problem location and description?

Background
Two key requirements for generating work orders include the ability to locate the problem (geolocation) and description accuracy. Evaluation of these two areas starts with customer submission and ends with how useful maintenance personnel find the information for diagnosing the type and extent of the repair. Both are discussed in the following sections.
Geolocation, essential for timely maintenance action, is the process of not only accurately reporting a specific geographic location but also relocating the asset at some point in the future. Without it maintenance craftsmen could never respond to repair the problem. Traditional work order systems typically consist of verbal or text reports and are frequently reduced to short text strings by a controller when inputting into a resource management system [4]. With little built-in capability to capture location information, it is not uncommon for maintenance personnel to experience subsequent geolocation issues, especially when the facility infrastructure is placed in a linear, or grid pattern. While it is well known that individuals have difficulty geographically describing just one item in the pattern, it becomes even more difficult when their perceived location is different that their actual location [5] [6] [7].
It is believed geolocation problems may be mitigated using smart-phones enabled with GPS. For some applications, quick and accurate geolocations were accomplished using GPS with minimal knowledge or effort on the part of the individual trying to determine a location [8] [9]. Unfortunately, under certain conditions, GPS effectiveness may be limited, such as indoors where GPS signals may be interfered with or lack sufficient accuracy to reveal which hallway or floor a problem exists [10]. Therefore, one objective of this study is to determine if GPS enabled smart-phones inherently provide better geolocation information than traditional work order submission systems.
After geolocation, object recognition and ability to describe accurately becomes the next necessary component to successful reporting. Being able to visually recognize an object and accurately describe it is the result of several factors. The first factor is an individual's past experiences with the object. Past experiences build visual representations of objects, which a person remembers when they see the object again. Representations of the object not only consist of a remembered image, they also carry additional information about the object, such as function and name. If the individual cannot determine the object as a whole, they resort to assigning the object into rough classes of objects they do remember. Utilizing representations based off past experiences enables individuals to visually recognize the object, and mentally prepare a description to be transmitted [11] [12]. Without experience with an object, individuals can only relay remembered representations that resemble the object and can lead to possible work order inaccuracies. A simple example of this would be calling an "A19 incandescent soft white interior frosted 60 Watt 120VAC medium base bulb" with the first class to occur to the individual, such as "light bulb." Clearly identifying the problem with the former description would allow immediate resolution as the maintenance personnel would arrive with the correct light bulb on the first visit. The latter, however, requires additional information to remedy the problem, possibly in the form of multiple visits or at a minimum time spent researching the lighting used in the reported area.
Options for mitigating a lack of object expertise are limited regarding facility infrastructure. One way is to educate all individuals to the visual recognition level of an expert, which isn't practical. Another, more preferable way, is to enable experts to be able to see what the individual is seeing and thus perform the visual recognition themselves without having to be there. An audio or text only media channel does not provide this capability. However, with the expanded communications inherent in modern smartphones, transmission of high quality visual information is possible. The addition of expanded channels for communication, especially visual, has been demonstrated to improve the effectiveness of communication [13]. Media richness theory describes a communications mediums ability to reproduce the information sent across it [14]. A visual channel which allows an expert to see exactly what the customer sees would be highly valuable to maintenance personnel and consistent with media richness theory would provide a much richer communication medium as compared to text or voice alone.
An underlying principle of media richness theory is the concept that information is only useful as long as it reduces uncertainty for the receiver [15] [14] [16]. As receivers of the reported information, maintenance craftsmen are the most appropriate evaluators regarding usefulness of information submitted, however usefulness for reducing uncertainty is largely dependent on what organizational language the information arrives in. Organizations and groups of specialties have their own unique organizational languages as a result of constant internal communication and natural adaptations for efficiency [17] [18]. Information that does not conform to unique organizational languages may have little impact on reducing uncertainty in the receiver.
Infrastructure customers frequently have a tough time overcoming organizational language barriers when communicating with maintenance personnel. Due to minimal facility experience, customers often are unable to communicate in the unique verbal/text-based language of maintenance personnel. With limited communication channels afforded by typical work order request systems, difficulties in communicating the exact nature of the problem result in frustrations customers and maintainers alike. Smartphones may provide a key to mitigating the language barrier and thus increasing problem description accuracy by providing a mechanism which allows both sides to understand each other's language. Utilizing the visual channel enabled by smartphones to send live images, customers may be able to transmit all of the information needed to maintenance craftsmen without knowing any additional information about the asset in question. Upon receiving the transmission, maintainers visually recognize the asset and reduce uncertainty about the problem.
To investigate whether or not smartphone technology is capable of overcoming geolocation, object recognition, and the lack of maintenance experience and subsequent language barrier obstacles, an experiment was conducted in order to test the following three hypotheses.
• Hypothesis 1 (H1): Smartphone-based mobile work order submission applications will provide more useful information than traditional verbal based submission systems. • Hypothesis 2 (H2): Smartphone-based mobile work order submission systems will allow non-maintenance personnel to submit information of equal description accuracy compared to individuals with a maintenance background. • Hypothesis 3 (H3): Smart-phone based systems will provide more accurate geolocation than traditional verbal based systems.
If smartphone based work order submission systems do support the hypotheses presented, the immediate effect of reducing communication errors, improving maintenance diagnoses, and reducing travel could be achieved, which would directly translate into saving time and money for the organization.

Methodology
To evaluate the effectiveness of a smartphone-based work order submission system, the authors conducted an experiment simulating common real-world maintenance scenarios. A smartphone application was created, modeled on commercial versions available today, to compare with an existing verbal capture system currently in use. Both tools gathered the three main items required for maintenance work orders: point of contact, location, and description of the problem. The primary difference of the smartphone-based application was that it allowed subjects to take a photo and simultaneously capture the phones last known GPS location. The traditional verbal work order system used a Google Voice voicemail account which prompted users to answer questions such as: what is your location and what is the infrastructure problem that you are trying to submit. The automated system conveniently standardized all questions for each subject and the authors were able to record every word uttered for future transcription and analysis.
The authors recruited maintenance craftsmen to serve as experts for the experiment. They were responsible for the primary task of generating a list of the key data required to diagnose and repair each maintenance scenario. After maintenance craftsmen determined what data was considered useful, work order requests placed by subjects were scored to determine the usefulness of the information for a work order (overall usefulness ratio) and accuracy of the description (description usefulness ratio). The authors then performed statistical analysis to evaluate performance differences between the two systems.

Subjects
Subjects (n = 40) were comprised of students, faculty, and family members, and were randomly assigned to a submission method (verbal or smartphone). Subjects completed a pre-experiment survey to determine facility maintenance experience, job background, education level, age, and gender. The backgrounds of all participants were examined to ensure there was a representative mix of organizational languages comparable to a typical maintenance organization. This distribution is provided in Figure 1. Civil engineers (with maintenance backgrounds) made up 50% of the subject pool while the balance of subjects came from the space, financial, instructor, logistics, contracting and computer science backgrounds. The corresponding diversity of organizational language allowed splitting the subjects into two groups (maintenance vs. nonmaintenance) for comparing smartphone work order submission vs. traditional verbal systems.

Analysis and Results
To test the study's hypotheses, the authors calculated description ratios and an overall usefullness ratio. The description ratio provides a normalized quantitative measure of how accurately subjects described the maintenance issue. The score was calculated by comparing an expert generated list of key description elements for each maintenance scenario with the subject's actual description of the maintenance problem. Every data point submitted that matched an expert identified key data value was scored as a pass (value = 1); if not then a failure was assigned (value = 0).
Once the description ratio was computed, a second ratio was calculated called the usefulness ratio. The usefulness ratio was used to incorporate the concept of geolocation in the study. As previously discussed, if a maintenance problem could not be located, any description provided would not be very useful. Geolocation data was converted to binary pass (value = 1)/fail (value = 0) criteria. Geolocation was scored as fail if the location data provided would not allow the expert to accurately locate the broken infrastructure item. The following example illustrates how a subject's response was captured and scored for a flickering light bulb using the mobile work order submission process. Figure 2 depicts data captured for the light bulb example.

Fig. 2. Work Order Submission Data Collection
Once the work order was submitted, the expert generated list of required data elements was used to score the submission. If either the text entered by the subject or the image captured by the smartphone provided one of the expert required data elements, a score of one was assigned. If a key data element was not received through the submission, a score of 0 was assigned. Figure 3 depicts the example scoring.

Fig. 3. Scoring Expert Key Data Elements
The Description Accuracy Ratio for this example was a result of the subject submitting six of the nine key data elements required by the experts. For Geolocation score, a "1" was recorded if either the GPS coordinates accurately returned the expert to the maintenance location, or the subject provided enough detail to rapidly locate the maintenance problem (i.e. building and room). If the problem could not be immediately relocated with the given data, a score of 0 was entered. In this example, the GPS location and the subject description proved insufficient to allow the expert to relocate the flickering light and therefore was scored as a 0.

Analysis of Usefulness Ratio
The authors conducted an independent samples t-test to test the hypothesis that the smartphone work order submission process (n =20) provided more useful information than the traditional verbal system (n = 20). Additionally the usefulness ratio was calculated by standardizing the Description Accuracy Ratio and Geolocation scores (to avoid overly weighting one of the scores), summing them, and then dividing by two. The test was significant (t(1148) = 7.42, p < .01), supporting research hypothesis 1. Subjects in the smartphone submission condition (M = .106, SD = .49) on average provided a higher usefulness ratio than subjects in the traditional verbal system (M = -.108, SD = .48). Figure 4 shows the distributions of the two groups.

Fig. 4. Usefulness Ration Analysis
This test supports the assertion that the smartphone-based work order submission application delivers information that is more useful; however, the question regarding its ability to mitigate organizational language barriers and communicate at the level of an expert regardless of experience remained.

Analysis of Description Accuracy Ratio by Subject Background
The authors conducted a second independent t-test to evaluate the hypothesis that a smartphone-based mobile work order submission system will allow non-maintenance personnel to submit information equal in accuracy to individuals with a maintenance background. To conduct this test, the subjects were grouped first by experience and then by description accuracy for each of the two submission methods. The first test evaluated description accuracy for the traditional verbal submission system among subjects with a civil engineering maintenance background and subjects from the other backgrounds. For the traditional work order submission system, the t-test indicated a significant difference (t(572) = 2.77, P < .05) between the two groups. The description ratio mean for civil engineering organization was 0.42 (sd = 0.25), and the mean for individuals outside of civil engineering organizations was 0.36 (sd = 0.26). Figure 5 shows the distributions of the two groups. An analysis of the second subset was then accomplished to determine if smartphonebased mobile work order submission technology allowed the individuals without a civil engineering background to describe the maintenance problem as well as individuals with a civil engineering background. The t-test indicated no significant difference between the two methods (t(577) = 1.51, P > 0.05), thus indicating that when utilizing the smartphone-based mobile work order submission technology there is an effect of mitigating organizational language barriers. The description ratio mean for individuals with a civil engineering background was 0.71 (sd = 0.23), and the mean for individuals outside of the civil engineering organization was 0.68 (sd = 0.25), thus support exists for hypothesis 2. Figure 6 shows the distribution of the two groups. It should be noted, however, that while there was no significant difference exists between groups in figure 6, the means for each group were much higher than subjects in the traditional work order submission condition shown in figure 5. The results indicate evidence exists that in addition to mitigating organizational language barriers, simply using the mobile submission method improves description accuracy for everyone, regardless of background

Analysis of Geolocation
Finally, the authors examined how the different submission methods affected geolocation. Geolocation was scored as a 1 for individuals providing information sufficient enough to allow an expert to find the problem, or a 0 for inaccurately defining the location. For the smartphone submission process, the phones' GPS location service was used in conjunction with subject descriptions to see if it had correctly captured their location, whereas with the traditional method, only the user's description was used. Surprisingly, smart phone GPS services for interior work orders generally determined the correct building, but beyond that was essentially useless in assisting in the geolocation of the work order, especially when the building contained multiple floors. Smartphones accurately provided geolocation only 65.46% of the time compared to the traditional method which had an accuracy rate of 71.45%. Figure 7 depicts the relative proportion of the responses that resulted in accurate geolocation by submission method.

Fig. 7. Geolocation by Submission Method
To evaluate the results, a Pearson's Chi-square test was calculated. Significant deviation between the two submission methods was found (χ 2 (1) = 4.79, p < 0.05). This significant difference can be observed when looking at the descriptive statistics listed in figure 7. The authors found that geolocation when using the smartphone method was poor (65%) compared to traditional submission users (71%). Hypothesis 3 suggested that GPS services inherent in the smartphone would help improve the geolocation of the maintenance issue, however the results indicate the exact opposite, thus no support exists for hypothesis 3. Further, it was interesting to note that the smartphone users actually performed significantly worse, despite the subjects retaining ability to describe the maintenance problems location similar to subjects in the traditional submission process condition. While impossible to determine the cause of the observed phenomena, the authors speculate that subjects using the smartphones may have relied too heavily upon the GPS services and felt less need to be descriptive when reporting the maintenance problem.

Conclusions
The results of this study indicate smartphone-based mobile work order submission systems may allow individuals to submit more useful problem descriptions regardless of how much familiarity the submitter has with civil engineering or maintenance procedures. The experiment results also imply craftsmen, when receiving more useful information tailored to their needs, can make better informed decisions and enable them to proceed to the job site only once with a clear idea of the work/parts required, thus eliminating unneeded effort. However, the results also indicate caution should be taken when employing GPS enabled smart-phone applications. Users should be trained on the limitations of the GPS services within buildings and encouraged to submit a high level of detail regarding geolocation of the reported problem within the problem description. These results clearly indicate while the potential benefits of smartphone asset management applications exists they should not be considered a "silver bullet" towards eliminated all problems with work order submission systems.