Chatbots for Brand Representation in Comparison with Traditional Websites

martin.ebner@tugraz.at Abstract —Chatbots have gained enormous popularity in recent years. IT gi-ants such as Microsoft, Google and Facebook have taken an interest in automated conversations. Messaging apps like WhatsApp and Facebook Messenger are playing an increasingly important role in smartphone usage and communication in general perfect conditions for chatbots. This paper provides an introduction of recent chatbot development in general and in customer environments. As part of this work, a chatbot was developed for the Austrian IT company CodeFluegel GmbH. The chatbot, named Theodore, provides information about CodeFluegel via Facebook Messenger and webchat, like the existing company website. The design process and implementation of the chatbot as well as architectural considerations are explained throughout this document. In a user study, participants perform typical tasks with the website and the chatbot. The usage of both platforms is evaluated in order to identify advantages and disadvantages as well as differences compared to the other technology and to draw conclusions. Findings to this study indicate promising prospects for chatbots as alternative touchpoints for customers and others and a way to replace and enhance traditional


Introduction
Chatbots and conversational interfaces have become increasingly popular in recent years. At the developer conference Build 2016, Microsoft's CEO Satya Nadella boldly proclaimed that "Bots are the new apps" [1]. Several factors have nurtured this trend. On the one hand, messaging applications like WhatsApp and Facebook Messenger are attracting billions of users (see Table 1) and occupying a large portion of a user's screen time [2]. On the other hand, major technology companies like IBM, Microsoft, Google, Facebook and Amazon have taken an interest in providing development tools and natural language processing centered around chatbots, resulting in easier development and wider acceptance and adaption of conversational interfaces. [3] Since the invention of the Internet, companies have been using websites as means to get in touch with potential customers, employees and generally interested parties. This paper aims to evaluate the feasibility and usability of chatbots in such a context, given the immense popularity of previously mentioned messenger and social media platforms. For this evaluation, a chatbot was developed for the Austrian IT company CodeFlügel GmbH 1 . The chatbot provides information about CodeFlügel via Facebook Messenger and webchat, similar to the existing company website. The design process as well as architectural considerations and details of the implementation are explained throughout this document.
For the evaluation itself, a user study is conducted where participants perform typical tasks with the website and the chatbot. The usage of both platforms is evaluated in order to identify advantages and disadvantages as well as differences compared to the other technology. Based on these evaluations, conclusions will further be drawn concerning, for example, differences in speed and user satisfaction. Findings to this study indicate promising prospects for chatbots as alternative touchpoints for customers and others and a way to replace and enhance traditional websites. Table 1. Monthly active users of popular messaging platforms with bot support according to [4], [5] and Telegram 2 .

Related Work
Chatbots provide an advantage compared to traditional mobile apps. Apps are prone to "app fatigue", which describes the tiredness of users to install new apps. Other driving factors of bots are the growth of messaging apps and social media, as well as the support of large companies like Facebook and Microsoft in the recent years [3] [6]. With these factors in mind, this chapter focuses on some case studies around the development and research of chatbots and conversational interfaces.
Shawar et al. showed a chatbot answering Frequently Asked Questions (FAQ) for the School of Computing 3 (University of Leeds). The bot used pre-processed data of the school's online FAQ on their website and pattern matching to map user input to questions and retrieve the correct answers. Users "found it a novel and interesting way to access the FAQ using natural language questions". Compared to a Google search, users were able to find a relevant answer more often and the majority of them preferred using the chatbot. Driving factors were the chatbots ability to provide direct answers to the question if there was only one matching answer and fewer links in general, resulting in less search time and better overview. [7] Almost 50% of Internet users in the U.S. use social media to contact customer service. According to recent studies, more than two thirds of users expect an answer within an hour, but it usually takes over six times as long to get a response. Scaling can be a problem with high numbers of requests, which chatbots could solve. Xu et al. built a chatbot for customer service on Twitter, as a feasibility study on such software. They used deep learning algorithms and trained their bot with nearly a million Twitter conversations from more than 60 different companies and brands. One of their key findings was that more than 40% of customer requests are only emotional and not looking for any kind of information or help, possibly due to the nature of social media. Their customer service bot showed results close to a human when handling such emotional requests, while informational requests seem to require more complex training. [8]  Another example for bots in customer interaction is WAH Nails 5 . Sharmadean Reid, the founder of the company, said that 30% of reservations were already handled by their chatbot only two months after its launch and plans to enhance it further using AI were already underway [9]. While this is only a single example, chatbots have the potential for significant savings in such applications, where human agents could be replaced, or new services created. According to an infographic (see Figure 1) by business magazine Business Insider 6 , potential annual savings in customer service in the United States of America alone amount to $23 billion US dollars 7 .
Beriault-Poirier, Tep, and Sénécal conducted an exploratory user study similar to the one shown in this paper. The goal of the study was to learn more about the user experience provided by chatbots from different brands on Facebook Messenger, namely food and beverage company Whole Foods 8 , clothing line Tommy Hilfiger 9 and travel search site Skyscanner 10 . Users were asked to complete two tasks for each brand (six in total) one on their website and one with their Facebook Messenger chatbot. Chatbots and websites were then evaluated from 1 to 7 in terms of usefulness, ease of use, ease of learning and satisfaction. User experience scores and abandonment rates were favouring the companies' websites. Analysis of the participants' facial expressions, however, showed that chatbots generated more positive emotions. Participants liked the chatbots' ability to deliver quick and precise answers and also showed a willingness to use chatbots in the future. [10] Chatbots for customer interaction could also impose new problems. Heckmann and Kraus mention that depending on the specific use case, personal data must be handled carefully, and usage of such data has to be disclosed. Commercial usage could also come with the requirement to inform users about those commercial intentionsfor example, disclosing advertisements. Especially with the recent introduction of the General Data Protection Regulation (GDPR) 11 , companies must take special care of protecting personal data in and outside of the European Union (EU). [11] While bots can help in building a brand and customer relation, a bot's personality is also able to do harm. As seen with Microsoft's Twitter-bot Tay (see [12]), machine learning and AI could lead to unforeseen behaviour, which in turn could potentially damage a brand or company [13]. Brands and developers also have to keep in mind that a chatbot's presentation and persona can result in unwanted perception and emotions, especially in regard to the uncanny valley effect 12 [14] [15].  12 Described as "the feeling of eeriness and discomfort towards a given medium or technology that frequently appears in various kinds of human-machine interactions" by [17].

Theodore, A Chatbot
As part of this case study, a chatbot called Theodore was created. The chatbot's main use is to represent a company, namely the Austrian-based software developer CodeFlügel GmbH. CodeFlügel is located in Graz, Austria and specializes in developing augmented reality mobile applications. Virtual reality, traditional apps, web apps and custom projects are also part of their offered services. CodeFlügel has about twenty employees consisting of software developers, sales staff and other people from the operative business such as a human resources manager, marketing personnel and the chief executive officer.
Like most companies, CodeFlügel uses a website 13  CodeFlügel also engages in social media platforms, one of which is Facebook 14 . Facebook offers the possibility to directly contact a company via their own Facebook Messenger 15 . Messenger is a chat and messaging platform providing APIs to build third-party tools and more importantly chatbots.
Theodore was designed to reproduce most of the website's features and to work with Facebook Messenger as well as a custom webchat, which could be embedded on a website. Existing APIs and content should be (re-)used as much as possible. For example, if a user asks the chatbot "What does your company do?", it should answer with a proper explanation about the company's products and services.

Target audience
In order to design a company chatbot, it is imperative to recognize and understand the target audience of CodeFlügel. The website users can be divided into three groups: Clients: A client or potential client is someone interested in purchasing a product or service from the company typically a mobile application or a website. Some of them might have a very specific request or idea, while others are interested in consultation and guidance. They most likely want to know what the company is offering, their level of expertise in specific technologies and how to contact them. The technological expertise of the customers varies greatly and ranges from beginners to experts. For clients, the chatbot would not only offer information, but also function as a demonstration of the company's previous work on chatbots. Potential employees: Potential employees are usually students with little job experience, but a technological background or less likely people interested in marketing, sales or human resources. They could use the website or chatbot for general information about the company or to inform themselves about job openings or projects the company has worked on.
Blog readers and everyone else: This group of users is probably not looking for anything specific about the company, but rather for one of the varying blog posts or general information about the technology used at the company. They most likely found the website / chatbot via a search engine such as Google 16 or a social media channel such as Facebook.
The chatbot provides little benefit for such users, besides interacting with the user and possibly generating interest and publicity.

Dialog design
With such a target audience in mind, the chatbot's requirements and a feature set were defined in consultation with CodeFlügel. The chatbot should feature almost all of the website's content. Since the interaction is different from browsing a website, some additional design decisions have to be made. For example, a help function and a fallback dialog help to enhance the user experience. User input is limited to text input and quick reply buttons, while the output can incorporate media and specific layouts as well.
Greeting: Theodore must be able to greet users in order to help them understand that they are chatting with the company chatbot of CodeFlügel. Thus, the chatbot first introduces himself and then shows some examples about what it can help the user with.
About the company: The chatbot should be able to provide a short explanation about what CodeFlügel does and which services they offer, given an input like "What does CodeFlügel do?" (German: "Was macht CodeFlügel eigentlich?").
About products and service: Products and services should be explained by Theodore. Furthermore, the chatbot should provide information about completed projects.

Contact information:
The most important contact information should be displayed by the chatbot. This includes the office address, phone number and email address. A link to Google Maps 17 is provided to help users in locating the office.
Open positions: Theodore should inform users about open positions and provide a way to contact CodeFlügel. In case of no vacancies, an information about speculative applications and where to apply is shown.
Newsletter subscription: CodeFlügel uses Mailchimp 18 as a service to handle their newsletter. The chatbot should provide a way to subscribe to the newsletter by accessing the Mailchimp API 19 .
Blog: Blog posts are published regularly by CodeFlügel. The website uses Wordpress 20 as its content management system. Theodore should output the most recent blog posts by accessing the website's REST API 21 .
Social media links: CodeFlügel's social media handles should be shown and linked to their respective platforms, if a user asks for this information. This includes Facebook, Instagram 22 and Twitter 23 .
Help: Another important aspect in designing a chatbot is to provide help, if a user gets stuck or does not know what to do. Consequently, if a user asks for help, a short description about the chatbot's abilities should be shown.
Fallback: The chatbot is made for a specific domain which defines the capabilities. If a user input is outside of this domain or the Natural Language Understanding (NLU) engine is unable to detect the user's intent, some kind of fallback should be triggered. For example, Theodore is not designed to announce the weather. Thus, if asked whether it is going to rain today, the chatbot should tell the user that he could not understand what the user wants or that it does not know an answer to the specific request. If this happens several times, the chatbot should present a way to get in contact with a real person.
Theodore should feature informal language and a friendly attitude. The chatbot should make clear that it is a program and not pretend to be a real person, which should help to avoid the uncanny valley effect mentioned in Related Work.

Architecture and implementation
Since CodeFlügel maintains a Facebook page as well as a website, the chatbot should be usable on both platforms. Facebook Messenger is used for the first, while a custom webchat is used for the second. Messenger already provides an existing messaging platform and chat experience. Only the backend needs to be developed for this platform. A webchat, however, also needs its own User Interface (UI) and ways to communicate with the backend.
The same backend is used for both platforms. Facebook provides an API to communicate with Messenger as well as webhooks to handle incoming messages and actions. The webchat uses web-sockets to offer a fast and responsive chat experience.
In order to streamline the development process and to increase the maintainability of the chatbot, Facebook Messenger's message format (in JavaScript Object Notation, short JSON) is used for both platforms and its UI components are recreated for the webchat. These components include simple text, Messenger's Generic, List, Button and Media Templates. Typing indicators to simulate the behaviour of typing messages and quick reply buttons (see Figure 2) were implemented as well. This also makes the different platforms more consistent and increases usability as users are presented with a familiar chat experience.
Instead of using a bot development framework to start the development, a custom implementation was chosen. While such frameworks provide functions and templates for quickly achieving first results, they tend to only support subsets of the features offered by some bot platforms. Google's Dialogflow 24 (previously known as API.AI) was chosen as the NLU service because it shows promising development with features such as versioning and environments. It also offers a concept which is easy to use and understand and, therefore, enables fast prototyping. Another important aspect working in Dialogflow's favor is their free Standard Edition, which comes with a sufficient quota of text requests.
Dialogflow's Messenger and webchat integrations, however, provide only little room for customization and lack support for advanced features such as message receipts (Facebook Messenger) and UI templates (webchat). Therefore, the chatbot uses a dedicated backend to handle incoming messages. On the user side, a client application (webchat or Facebook Messenger) is used. In the case of the webchat, communication happens directly between the server and the client. When using Facebook Messenger, client requests are first sent to Facebook's server before reaching Theodore's backend and vice versa.  Figure 3 shows the chronological structure of the communication between a user and the chatbot on the webchat. First, the client sends the user's input to the backend. The backend then sends the request to a NLU service, which extracts an intent and possible parameter to get to know what the user wants. If necessary, additional APIs are visited. For example, the latest blog posts are retrieved from the website using the Wordpress REST API. Subsequently, a proper response is created. After all this processing, the response is sent back to the client and presented to the user.
The chatbot has to access different APIs in order to provide all required features. Wordpress's REST API is used to retrieve blog posts as well as open positions from the website. Mailchimp's REST API is used to subscribe users to the newsletter, if they want to.
Other content and responses such as product and service descriptions are stored in files or in Dialogflow's intent handling.
Theodore is written from scratch using Node.js 25 (version 8), Socket.IO 26 and Dialogflow. The backend handles connections from and to both Facebook Messenger and a custom webchat, written in HTML and TypeScript with the popular frontend framwork Angular 27 . Communication is done with normal HTTP requests and websockets.
The chatbot was built with modularity in mind, in order to easily adapt to different needs and provide an extendable base for other projects. Adapters for other bot platforms, NLU services and analytics like Chatbase 28 as well as API connectors and additional features can be added via extensions. Currently, Theodore features scripts for Facebook Messenger, a webchat via Socket.IO and a complete adapter for Dialogflow.

Case Study
In order to see how the chatbot performs and whether users accept it as an alternative to the website of CodeFlügel 29 , a user study was conducted. Tests took place from November 6th to December 22nd, 2018 with individual participants. Users were chosen based on a specific user classification. The goal was to only introduce participants which are relevant to the company's website (and chatbot respectively) for example, potential employees. Those users were asked to perform several common tasks on the company website and with a corresponding chatbot. Timings of those tasks and user feedback were recorded to get quantitative (speed) and qualitative (acceptance, dis-& advantages) results and see what tasks are better suited for the chatbot compared to the website.

Setup
User selection: Participants were chosen to reflect the user base of the company's website. The relevant user groups have been explained in 3.1 Target Audience.
A total of twenty (20) users were selected based on the previously outlined classification. This approach makes it possible to better adapt the selection of test subjects to the actual users of the website and to create more appropriate tasks. At least five participants per user type were among them. The age ranged from 20 to 33, while the average user was 27.7 years old. 60% of the users had a bachelor or master's degree while another 25% were still working on their bachelor (with 20% working on their master's degree). 70% had a technical background (mostly IT), either through work or education.
Preparation: At the beginning of the test session, users were greeted and given a summary about the project as well as about the user study and the upcoming steps. Then they were asked to fill out a consent form and a questionnaire about their background, knowledge in different technologies such as chat applications, augmented reality and programming as well as their expectations of a company's website and chatbot. The goal was to collect enough background information about the users to be able to find possible connections between the use of the chatbot or the website and the different backgrounds.
Setup and tasks: The test setup consisted of a workspace with a computer, two monitors, a keyboard and a mouse. The participants were placed in front of the computer while the facilitator was sitting to their left, a little further back, to observe the test person's actions and measure the time of each task. The right monitor showed instructions for the tasks, while the left one was used to perform the tasks. The actions on the left monitor were captured with the screen capturing software OBS Studio 30 .  Goal: User finds job page/list. 7 Task: Find a way to try Augmented Reality (AR) for yourself.
Goal: User finds demo app. 8 Task: Find at least one Augmented Reality (AR) project created by CodeFlügel.
Goal: User names at least one (1) Augmented Reality (AR) project. 9 Task: Find out what Augmented Reality (AR) actually is.
Goal: User opens AR explanation.
A list of nine typical tasks was defined (see Table 2), taking into account the various user groups and the target audience of the company's website. The list included things such as looking for the latest blog post, searching for job openings and finding at least one completed augmented reality project of the company. First, the tasks had to be performed on the company's website. In order to increase the quality of feedback and observations, users were asked to verbalize their thoughts, similar to so called thinkingaloud-tests. Each task instruction had to be read out loud by the participants. After that, a timer was started by the facilitator. The user had to clearly state the solution to the given task and, if it was the right one, the timer was stopped. When all nine tasks were completed, the same tasks had to be performed using the chatbot instead of the website.
Feedback and interviews: After the completion of the tasks, another questionnaire had to be filled out. The questionnaire was there to gather the participant's feedback, recognize potential flaws in the chatbot design and measure the acceptability of the company chatbot. An interview was then conducted to clear up any potential ambiguities and obtain better-quality feedback.

Results
When asked what users expect from a chatbot and website of a company, the results were pretty close. Information about Products & Services and Contact Details were expected by all users from the website, while 70% (Contact Details) to 85% (Products & Services) expected the same from the chatbot. References, Jobs and Blog & Articles only amounted to 20% -55% of the users' expectations.
In terms of speed of the task completion, 80% of participants were faster using the chatbot. The average difference in time needed for all tasks was 2 minutes and 54 seconds in favor of the chatbot. The average completion time of all tasks was 3m 38s with the chatbot and 6m 33s with the website, while the averages for one task were 24 seconds (chatbot) and 43 seconds (website).

Fig. 4. Average completion time per task for both platforms
Six out of the nine tasks were completed faster with the chatbot, although two of those six were only completed about six seconds faster. Tasks involving the website were faster or equally fast when a corresponding and self-explanatory (e.g. Blog) menu entry existed, while tasks seeking specific information not immediately accessible by the main menu (tasks 1, 7, 8 and 9) where significantly faster achieved with the chatbot. This is similar to what [10] concluded in their research.
In terms of perceived completion time, 80% of test users thought they were faster using the chatbot. 75% correctly assessed the chatbot as being faster, while one of the participants thought completion time was faster, but the opposite was true. Another one thought their website tasks were completed faster, but in reality, the chatbot run was the quicker one. 15% of the test users believed to have been equally quick with both platform tasks while actually being faster with the website.
Although the chatbot matched every participant's expectations, seven out of the 20 users said the input-to-intent matching could be improved, as the chatbot did not understand every message and reformulating the request was necessary. 80% of the users found the chatbot more entertaining than the website vice versa, only 10% were more entertained using the website. 40% deemed the chatbot more modern, while none said anything similar about the website. The chatbot was also described as fast by 50%, while only 10% used this adjective for the website.
60% of the participants said they preferred a chatbot with informal communication. The other 40% did not care about whether the chatbot approached the conversation formally or informally. This pretty much confirms what [16] published in their study. [17] showed that humans tend to use shorter messages when conversing with chatbots. Our study revealed that 50% of users relied on a combination of whole sentences and simple keywords to communicate with the chatbot. 30% used only sentences, while 20% based their messages solely on keywords.
Half the users thought the website and chatbot were equally appealing to them. The website was favored by 25% of the participants, partially due to being used to websites or due to being able to explore the content more freely. [10] suspected similar reasons for favoring websites over chatbots in their study. The other 25% found the chatbot more appealing. Reasons to favor the chatbot were instantaneous answers to specific questions without the hassle of clicking through menus and searching through pages, as well as its interactive nature, which resulted in more fun for some of the participants.
Overall, the company chatbot was received positively and 85% of participants could see themselves using more chatbots in the future.
Despite extensive background checks of the participants, no correlation between the time needed to complete the tasks and the level of IT knowledge was observed.

Conclusion
In order to compare the usage of chatbots to traditional websites, a specific company chatbot was created and a case study was conducted. Typical use cases of a company chatbot were worked out in order to compare such a chatbot with the existing company website and measure its performance.
The user study showed that chatbots are accepted by users. The users provided positive feedback about the chatbot, only some had minor issues with the matching of their queries and the size of the chat window. However, this could be improved easily by slightly changing the web chat design and providing more training data to the chatbot and the respective NLU engine. The study also revealed that it is important to train for whole sentences as well as key words, as both are commonly used by users. This was also discovered by [17]. The chatbot was faster than the website, especially when it comes to finding specific information about the company, and the users perceived it as more entertaining. Thus, it could potentially help companies in reaching new or younger audiences and in improving specific parts of their websites, such as the FAQs.

6
Future Perspectives Building on this study, further testing of the impact of UI improvements can be done. Natural Language Processing and matching of user inputs to intents also bear potential to increase the acceptance and enhance the user experience. In general, more chatbots could be tested and more users from different backgrounds and age groups could provide more valuable insight.
Another aspect to investigate further is the effect of different chatbot personas on the user experience and the influence of brand perception. One could also evaluate the possibilities of distributing and marketing bots. Do chatbot "marketplaces" provide a similar exposure to mobile app stores such as Google Play and Apple App Store?
Conversion rates and other Key Performance Indicators (KPIs) are things that could be explored in more detail as well. These analytics could show whether chatbots are economically feasible and here to stay.