Online User Reviews as a Design Strategy for Global Communities: Contributions of the Open Device Labs C ase

—Grass-roots community movements related to design practices and open spaces have emerged to address different issues. This study is part of a comprehensive research, which aims to explain the Open Device Labs (ODLs) ecosystem. The ODLs are a grass-roots community movement that aims to de-mocratize cross-platform tests and evaluation on real devices. As a global community with 152 laboratories located in 35 countries, online user reviews play an essential role in helping the long-term prospects of the movement. From the Design perspective, this paper aims to answer the question: what can be learned about the ODL ecosystem from online user reviews? To answer this research question, we conducted a qualitative inductive analysis of n=217 user reviews posted on the community website, from 65 labs located in 12 countries. The results and categories presented here are a key contribution to understanding the ODL ecosystem, and ultimately to other global service communities.


Introduction
The Web, since near its beginning was intended to be universally accessible. Webbased interactive content based on specific design standards has become and remains possible because of global collaboration. The World Wide Web Consortium (W3C) founded in 1994 aims to develop protocols and guidelines for an open and accessible Web for everyone. All the aspects of the Web 2.0 paradigm, from its foundation through its evolution, the openness, the global reach, the architecture of participation, the harnessing of collective intelligence, led to a breadth of new interface design possibilities and challenges.
Fragmentation issues and concerns about it have been running side by side with the evolution of the Internet and the Web. The diversity of hardware and software makes people experience and visualize content differently. The development of smart mobile devices, and the mobile software industry, extended the challenges to the software development life-cycle. All these issues are addressed by different actors and communities working together in one way or another across the globe.
Our professional interest in the evaluation of mobile user interaction led us to explore gaps in this topic, related to the practices of Design community. Lab-based versus field-based evaluation is an issue already addressed by the Human-computer Interaction (HCI) research community. On the other hand, other topics such as the emergence of open lab grass-roots communities are recent and underexplored. These communities act locally and connect globally, presenting challenges from multiple approaches. Identifying and gathering data about these complex and multifaceted phenomena is as important as it is difficult.
In 2011, the demand for a way of facing browser fragmentation challenges led to the emergence of the Open Device Labs (ODLs). The ODLs are a movement to promote free access to physical labs equipped with a pool of mobile devices connected to the internet to Web, app, and game testing. This idea arose from Web developers contact at international Web conferences. After the opening of a few physical open labs in different countries, their online presence began to be established, and connections were made between them. Firstly, a list of ODLs was made. After that a Google Groups was set up, and then a community website was established, and this was the beginning of the online global network.
We have been conducting case study research on the Open Device Lab global community movement since 2015 [1]. The comprehensive study aims to explore, describe and explain the ODLs ecosystem. The first phase of analysis was conducted on the online documents that were available, the second on interviews with ODL managers and collaborators and the third phase on user reviews.
The previous phases of our research collected data on the ODL hosts perspective. This paper presents the first results of the ODLs guests perspective through online user reviews a key design strategy for helping in the running of the Open Device Lab community movement, and aims to explore what can be learned from the ODLs ecosystem.

Literature Review
The digitalisation of communications has gone through two phases a) the spread of the World Wide Web in the 1990s and b) the transformation of the Web itself passing from web 1.0 to web 2.0 in the 2000s; and the big bang of new mobile devices, from smartphones to tablets [2]. The gradual introduction of mobile phones in the 80's, its evolution and the Web 2.0 principles converge to create the mobile Web 2.0 phenomenon [3].

2.1
The web (2.0) The change in the perspective of understanding the Web as a service platform was the key aspect of the Web 2. The term Web 2.0 was coined by Darcy DiNucci [4], and Tim O'Reilly published the first initiative to define it and understand its implications for future generations of software summarising the core competencies of Web 2.0 companies as follows [5]: • Services, not packaged software, with cost-effective scalability, • Control over unique, hard-to-recreate data sources that get richer as more people use them, • Trusting users as co-developers, • Harnessing collective intelligence, • Leveraging the long tail through customer self-service, • Software above the level of a single device, • Lightweight user interfaces, development models, AND business models.
The Web 2.0 architecture of participation is a term coined by Tim O'Reilly to describe the nature of systems that are designed to encourage user contribution [6]. In this context, collective intelligence refers to Web-based software which help people share, collaborate and interact [7]. Systems designed to find out what people think about a product or service led to the increase of user-generated content phenomenon [8].
Many different disciplines comprise user engagement, recommender systems and related terms. Recommendation systems have been studied in diverse areas such as social-based recommender systems [9], user expertise through online reviews [10], expert opinion [11], customer reviews and automated recommendation systems [12], which are also deployed in industry for product and service recommendation [9].
Online customer reviews is defined by Ref. [12] as "peer-generated product evaluations posted on company or third-party websites" and the authors explain how retail websites use it to offer consumers the opportunity to post product reviews in the form of numerical star ratings (usually ranging from 1 to 5 stars) and open-ended customerauthored comments. Therefore, referring to future work, the study recommended an analysis of text reviews compared to the start rating to determine how the stars match the review content. We considered the authors recommendations and applied it to our research, which analyses service reviews, from current design perspectives. Moreover, this study presents the global community running challenges using user experience and service design concepts.

Service Design and User Experience
Interaction Design is the umbrella term for concepts like user interface design, user-centred design, and experience design, which designs experiences to support people communication and interaction in their everyday and working life [13].
User experience (UX) has different meanings and definitions and, in this study, we adopted the term as a measurement concept. UX goals are divided by Ref. [13] in desirable and undesirable aspects according to user feelings. UX is concerned about the user feeling, pleasure and satisfaction when interacting with a product or using a service related to the overall impression [13]. According to Ref. [14] "In user experience design multiple components must be designed: visuals, features and commands, copywriting, information architecture, and more. Not only should each component must be designed correctly, but they also be integrated to create a total user experience. Service design follows the same basic idea". In the context of online reviews, experience "is some quality that users gain over time, as they consume, rate, and review additional products" [10].
Service Design is concerned with making services better suit the user and customer needs examining all activities, infrastructure, communication, people, and material components, to improve the quality of service and the interactions between the provider of the service and its customers. It aims to suit the customer need while seeking to be sustainable for the service provider, and ideally being user-friendly and competitive within their market [15].
Service Design components According to Ref. [14] there are three main components of service design: Props -physical or digital artefacts, including products, used to perform the service successfully. It considers: • Physical space (storefront, teller window, conference room), • Digital environment through which the service is delivered (webpages, blogs, social media) • Objects and collateral (digital files and physical products) People -anyone who creates, uses, or is indirectly affected by the service (employees, customers, fellow customers encountered throughout the service, and partners).
Processes -workflows, procedures, or rules needed to perform the service successfully.
These components are divided into frontstage (channels, products, touchpoints, interfaces) and backstage (policies, technology, infrastructure, systems) depending on the customer perspective. Service Design is concerned to "meet the customer's needs in the most relevant way whilst remaining economic (or sustainable) for the service provider" [15].
Considering all the characteristics presented above Service Design was identified as an approach that meets the aspects of the Open Device Labs free service aspects and needs.

Open Device Labs (ODL)
The ODLs are a grass-roots community movement. They are a global voluntary initiative, which establishes physical device labs as a free service for mobile testing and evaluation on real devices. "In result, ODLs lead to an ultimate improvement of the web & app experience both for developers and for consumers" [16].
In April 2012, the initiative took-off with the first ODL in Brighton, UK. In August 2012, when there were eight ODLs, all in Europe, the first online list of ODLs was published [1]. In January 2013, the opendevicelab.com, a directory of a global movement, was published with 37 ODLs across 18 countries with over 500 devices accessible [17].
The high number of labs spread across so many countries brought up challenges in terms of global aspects for the people, located in Germany, who were running the global community movement. Users were reaching out to the ODLs runners and complaining about device labs, which were listed on the server, were not available or were not there anymore, for example. Thus, one of these key challenges was a) keeping one eye on all the labs and validate if they were set up according to the information listed on the community website b) validate if it was a proper service linked to the basic principles of the community c) to get an idea of how many people were using ODLs services. Therefore, the ODLs team implemented the online user reviews as a strategy to address this problem, namely, if it gave an opportunity to everybody who uses an open device lab to comment, rate, and share if the lab is found as open, if it is a nice place to go and if they provide what they propose to do.
It is important to mention the ODLs do not offer a testing service. This means they do not test a product for others. An ODL is a tool for the owner testing purpose, who shares it with the local community as an "open practice". They offer free access to device labs as a type of "self-service". The hosts use the device lab to test and evaluate their own products. Additionally, external people are invited to use the lab for their own purposes by appointment. It is these aspects that frequently cause hosts to consider that the ODLs are not a service, although the evidences presented below shows that for guests it is clear that ODLs are a service. Thus, in this study, in which the guests evaluate their overall satisfaction, we will use the term service as a central category.
The OpenDeviceLab.com The online directory serves to three major goals (ODL 2013): • Help people to locate the right Open Device Lab for the job • Explain and promote the Open Device Lab movement, and • Attract contributors and sponsors to help and donate to ODLs.
The fast growth of ODLs across the globe resulted in 157 ODLs located in 35 countries being registered in March 2016. The online user reviews form has 4 areas: 1 -user identification 2 -ODL status situation 3 -rate overall satisfaction (1-5 stars) 4 -Comments (see Figure 1.and Figure 2)  Research Method Based on the global research approach, this study was designed to address the main research question: what can be learned about the ODLs ecosystem from the online user reviews?

Sampling and study population
At the time we collected the data, there were 152 labs registered on the webpage community. We worked on a homogeneous sample without concern for statistical generalisability [18]. We classified the ODL's users in two main categories: hosts and guests. The hosts are the organisations and / or institutions (companies, co-working or education institutes) which host the lab. The guests are external people who make use of the lab for free (individuals or companies such as freelance designers and developers, students, and web agencies). This study is based on the Open Device Lab global movement guest users.

Data collection
From the 152 labs, we have selected and collected data from the labs that had user reviews published. We collected all the data available online from 7 th to 9 th May 2017 and have been updating this until 8 th January 2018. The collected sample represents a total of n=217 reviews from 65 labs located in 12 countries: Australia, Belgium, Brazil, Finland, Germany, India, Poland, Spain, Thailand, United Kingdom, United States, and Uruguay. All data was captured with the original layout as selectable text in pdf files (Figure 3).

Data analysis
We conducted a general inductive approach for qualitative analysis [19] using Atlas.ti. This approach provides a systematic set of procedures to: • Condense raw textual data into a brief, summary format • Establish clear links between the evaluation or research objectives and the summary findings derived from the raw data • Develop a framework of the underlying structure of experiences or processes that are evident in the raw data.
In the analysis processes, first, we performed a pilot study analysing the data per lab reviews, but there were labs with 20 reviews and others with only one. Therefore, for the final analysis, we decided to consider each review as a document and not each lab. We imported the 217 pdf documents, user reviews, to the software. We searched for data not written in English, translated and reviewed it.
We conducted the primary-cycles coding using descriptive and in vivo methods [20] in which we first gathered "what's going here" [21] and defined the specific research questions to answer the main RQ: RQ1: What is the ODLs guest users' rate and comment about? RQ2: What is being tested by guest users? RQ3: What is the overall satisfaction of the ODLs guest user experience? RQ4: What are the ODLs guest users' profiles? RQ5: According to the ODLs guest user, what are the good and bad practices? In addition to the first-cycle coding, we created a first model which was later refined in the second-cycle coding based on the disciplinary and theoretical concepts [21]; Design lens presented in the previous section. The In vivo method was essential to identify similar technical concepts described differently by participants like web, web-based, web app.

Results
The results are explained in the following sections according to each research question, and the hierarchical category system in Figure 6.

RQ1: What is the ODLs guest users' rate and comment about?
In terms of numbers, the global results of the user rating about overall satisfaction show that the majority of them had a pleasant and satisfactory experience; rating it with five stars, n=186. A few labs received four stars, n=18. There was no lab rated with three stars, n=0, one lab received two stars, n=1, and a few received one star, n=12 (see Table 1).
These numbers are related to 65 ODLs located in 12 countries, Australia, Belgium, Brazil, Finland, Germany, India, Poland, Spain, Thailand, United Kingdom, United States, and Uruguay. The UK and Germany have the higher number of user reviews published. The results of the qualitative analysis related to these numbers are presented below, according to each category. There was no ODL rated with three stars, and considering the inductive analysis approach, there was no coding for this case. The final discussion is presented at the end of the results sections.

URW -R-1S
The labs rated with one star, n=12, are all almost found as closed, n=11, a proper and justified rate; e.g. "It has closed".
Only one lab, n=1, was rated with one star and found open. The user's comment about his/her experience started with the positive aspects of the space, but the primary purpose of the ODLs, the devices were considered negative: "Great digital workspace: Nice and quiet with free vending machine coffee, but the device lab has a couple of odd chinese knockoffs and a cheap htc all with more security cables than an iPhone 6 on the outside of a phone shop on a council estate. Don't waste your time trying to find this space... That'll teach me trying to do some work whilst on a weekend away!"

URW -R-2S
The lab, n=1, which received two stars, was found open, in terms of status, and the user comment was not much help; the guest just wrote: "Good'.

URW -R4S
The labs which received four stars, n=18, were all found to be open and had comments about positive experiences, detailed in the following paragraphs; e.g. "All devices I pre-booked were available, apart from one Samsung which was broken".

URW -5S
The labs rated with five stars, n=186, were almost all found as open but there were two reported as closed. In one of these cases, it seems the user selected the wrong choice in terms of status in the form. First, because it was rated with five stars and second because the comment says "Thanks". The other case was about a user who was in an ODL, which was moving to another place, so even if this user had a five-star experience, the lab would be closed.
After explaining the relationship between the ODLs rating and their status briefly, we present an example of the final qualitative categories according to its data analysis process cycles (see Figure 4 and Figure 5).

STATUS -Refers to open or closed labs.
• Open -Refers to the ODLs found open.
• Closed -Refers to the ODLs found closed. Although reporting an ODL as closed depends of different situations: Fact -Refers to an unquestionable source like directly from the ODL. Based on conclusion -Refers to different reasons (see Figure 6).

SERVICE -Refers to the guest user comments about the ODL service.
• Props -Is about the venue, the devices, the lab as space, software, Wi-Fi, cable and extra facilities like drinks (coffee, tea, water etc.). • People -Refers to the comments about the ODLs teams, which are usually positive and more about being friendly, helpful and experts on testing. Therefore, it is not about going there and just using the infrastructure. It is also about knowledge exchange, which was classified as a benefit in a subcategory. • Processes -Refers to booking system, communication aspects and devices arrangement. • User experience -Refers to the guest user impression about the ODLs service experience. • Testing experience -Refers to the guest comment specifically about the testing experience at the ODL. • Benefits -Refers to the guest user comments about the service benefits.
TESTING -Refers to technical issues about the testing field in general, not related to the ODLs services such as methods, purposes, and tested products.
The main themes presented above and all the lower-level categories, which answer all the specific research questions, are presented in the hierarchical category system ( Figure 6).

Discussion
Status is simple and handy information for both hosts and guests. First, there is the fact that when a guest decides to go to the ODLs website and register their opinion, they are primarily contributing to their local community, helping to verify if an ODL is open or closed. It is key information for the locals and the people running the global movement. Validating if an ODL is open or closed around the world would be invaluable for a small team or an individual. The closed labs are more challenging to verify because there are cases where the ODL is listed but is not available anymore. There are also the ones who are working but not updating their online presence or the ones who are in standby or in a similar situation. Therefore, a lab reported as closed by a user depends on many different conclusions.
Validating the ODLs status as open or closed is the primary contribution of the online user reviews, which allows the guest to complain about inconsistences with the information published on the community website. The rating star system is an objective evidence about the guest overall satisfaction with the service. Besides specific exceptions, ODLs rated with one star are clearly related to closed labs and labs rated with two to five stars are related to open labs.
Considering the ODLs are a voluntary movement and offer a free service, it is remarkable to observe in the results of the open-ended comments how demanding the guest users are. The comments help to understand the guest user experience; the desirable and undesirable aspects; what was positive and what was negative about it to guide recommendation about good, bad and key practices. Moreover, the comments help to understand what guest value more about an ODL, what kind of products they are testing and in which way, what are their profile, and issues needing to be improved.
Service helped us to understand, for example, the user preferences about the devices organisation. There are some labs, which do not have the devices on a stand and ready to go; the devices sometimes are in boxes. When we were seeking for good practices on device management, it was not clear what the best options were. The user review evidence shows that to the guests it is high valued to get the requested devices charged and ready to go when they get to the lab, even if it is a free service.
Considering all the themes evaluated by the guests, the overall satisfaction is positive. The findings confirm that a representative pool of devices is fundamental. At the same time, the people, a friendly and knowledgeable team, provide high added value to the ODLs service. The booking and structure arrangement for the testing performance are reported as very satisfactory. The ODL hosts could also share information about ODLs processes. As a community movement, we believe examples of a service design plan would help the entire community.
The help of the ODL team and the opportunity of knowledge exchange and expertise sharing related to testing issues during the time spent at the lab is not something promoted online, but the user reviews findings show the hosts are of service and this is something highly valued by the guests. There were cases of helping first visitors and first time testing on multiple devices. The evidence shows the guests value getting help from the hosts and at the same time, they value their privacy to conduct the tests.
We previously knew for what purpose the ODLs implemented the online user reviews. Thus, analysing these data, we could learn key aspects of the ODL ecosystem from its global users. The ODLs guest user overall satisfaction is related to open status, free access to a representative pool of mobile devices ready to go, in a place which combines privacy with knowledge and expertise sharing. To sum up, we could gather global information about: • ODLs status -the foreseen categories open or closed and unexpected ones like temporarily closed. • The overall guest level of satisfaction about the ODLs, and with what is related to, both common and special cases. • Indications about testing approaches and tested products.
• Hints on guest profile.
• Information about positive and negative service, testing practices, and ODLs benefits.
In conclusion, we may say the user review is essential both to the long-term of the local and global community. The results can be used as the base to create another version for the user reviews rate and comments in a semi-structured way based on the categories we found; not to replace the current open-ended text field but as an option for users who would be willing to contribute through a more informative and structured form. It would be a way to gather more and comparable data from the ODLs and improve the quality of service, the interactions between the hosts and the guests, and ultimately the user experience.

Limitations and Conclusions
This study is part of global research, which aims to explain the Open Device Labs ecosystem through a qualitative case study. This paper presents the main themes from the ODLs online user reviews (rating and open-ended comments) related to an evaluation of service user experience. An inductive analysis was conducted based on 217 reviews from 65 labs located in 12 countries. Online user reviews are a useful strategy both for the hosts and for the guests. The sample is limited in number, but considering the nature of the study and the open-ended and spontaneous comments, the findings are crucial to the primary research. It is the first step to understand the ODLs guest experience; without it, we would only have the hosts' perspective, not the guests.
The user reviews added a clear perspective of the ODLs as a service, a different view to previous data analysis, which led to select the Service Design as a proper approach to improve ODLs service user experience, for employees and guests. As a final contribution, the categories and qualitative data may help to build service design strategies for labs and communities in general.

Future Work
Primarily, we will use this study results to present the data triangulation of the comprehensive research on the Open Device Lab community ecosystem. Second, the results will be used to work on semi-structured user reviews and service design strategy. Additionally, we still have much to learn about the ODLs due to its openness nature, global reach and there are few studies about this. The identified gaps in this study provide opportunities for further research focused on service design and user experience.