Influence of Design Elements of Mobile Push Notifications on Mobile App User Interactions

atilla.wohllebe@gmail.com Abstract —With the increasing relevance of mobile apps for companies, push notifications to address app users are also becoming more important. While the acceptance factors of push notifications have already been extensively researched, the effect of the different design elements on user interaction by opening the mobile app is still completely unexplored. Based on existing scientific findings from related fields, especially banner advertising and e-mail marketing, the authors first develop hypotheses on the effect of title, button and image on user interaction with push notifications. In several experiments the hypotheses are tested using the example of a mobile shopping app. The results are evaluated using Chi-square test and Cramer’s V. While the use of a title seems to have a positive effect on interaction rates, the hypotheses on the positive effect of but tons and images on interaction rates have to be


Introduction
Smartphones and mobile apps are the most important ways globally to access the Internet: More than half of all Internet traffic is generated via mobile devices [1], while at the same time around 90 percent of Europeans are connected to the Internet -and the global trend is still rising [2]. From the user's point of view, mobile apps are among the most important functionalities of smartphones and now exist for practically all areas of life, such as shopping, communication, travel, education, health or entertainment [3][4][5][6][7].
One of the core functionalities of mobile apps is the so-called push notifications. The notifications, which are displayed on the home screen or the lock screen of a smartphone, regularly provide app users with current content, e.g. offers or news [8]. At the same time, the push notifications also serve to bring existing app users back to the app again. Commercial expectations of mobile apps from companies are high [9]. Especially in the retail sector, mobile apps are becoming increasingly relevant [10]. Push notifications play an important role in meeting these expectations [11].
The impact of push notifications on mobile app usage in the broadest sense has been studied frequently in the past from different perspectives, such as the influence of content or personalization [12], the use of location-based data [13] the influence of frequency [8], [14].
A gap in previous research on push notifications is the use of different design elements that are specifically possible for this type of advertising: for example, a title, an image or a button can be added to the actual text of a push notification [15], [16]. However, there seems to be practically no knowledge about how the design elements specific to this form of advertising influence user engagement with the notifications [17], [18].
In line with the S-O-R model, push notifications from mobile apps can be understood as a stimulus that acts on an organism -its recipient -and causes a reaction, for example the opening of an app [19]. Whether and how the organism reacts to the stimulus also depends on the attitude the organism, i.e. the recipient, develops towards the message just received [20].
According to the Attitude Toward an Ad Model, this attitude depends not only on the advertiser and the mood of the recipient, but also on the nature of the ad itself [20]. The consumer's desire for relevant information on the one hand and the curiosity aroused by a lack of information on the other hand are opposed: On the one hand, many findings indicate that relevant, even personalized information is desired [21][22][23]. On the other hand, the information gap theory suggests that a lack of information could increase curiosity and therefore lead to higher interaction rates [24]. This paper therefore examines the question of how the use of titles, images and buttons as design elements in push notifications affects user engagement with the notifications. In the literature review, existing relevant scientific findings on push notifications are presented first. Furthermore, existing literature on the design of banner ads (e.g. [25][26][27][28]) and on the design of subject lines in e-mail marketing (e.g. [29][30][31]) will be reviewed.
The hypotheses subsequently formed on this basis are then examined by means of several A/B test experiments, which has already proven to be a successful research method in similar contexts [32]. This is followed by the summary of the findings and the conclusion.

2
Literature review

Perception and impact of push notifications
Push notifications are an instrument of app marketing and can be considered an important driver for app usage [33]. By sending them, app publishers can expect virtually instant attention from their existing user base and corresponding app openings [34]. Content is the decisive factor for user response to the notifications. The literature constantly emphasizes the importance of content relevance from a user's perspective [35][36][37].
However, any push notification must also be considered as an interruption and thus carries the risk of disturbing users [38]. Such interferences can lead to frustration and even to the uninstallation of an app by the user [8], [39]. In this respect, the relevance of the content repeatedly must be emphasized.
The perceived usefulness is considered to be the driver of the relevance of push notifications. In the shopping environment, for example, this is driven by coupons, discounts and other price advantages [35], [40]. In addition, the importance of personalized content adapted to the behavior and socio-demographics of the respective consumer is emphasized [41] and also plays a role in the context of mobile apps [42]. A certain entertainment factor can also be seen as a driver for the interaction of app users with push notifications [43].

Design of banner ads
Since the previous findings on the influence of the design elements of push notifications are only very limited, the effect of the design in banner advertising is also considered, because this advertising channel has similar possibilities, at least in its design.
A/B testing is a method that is repeatedly used in the context of banner advertising to examine the effect of different design elements and aspects [32]. Basically, it can be stated that, in addition to the content, which has already been identified as decisive for performance in the context of push notifications, the color design and animation of banners also have an effect on performance in terms of clicks. However, this effect can also vary depending on the context [28].
Larger banners seem to have a clearly positive effect on the click rate [25], [27], [44]. Generally speaking, the amount of information displayed on the banner also has a positive effect on the click rate [44], [45]. Surprisingly, more text on a banner also seems to have a positive effect on user interaction, contrary to the expectations of the authors one study [44]. In the context of search engine advertisements, a study analyzing 57 million impressions and 185,000 clicks reveals that click-through rate is positively influenced by brand and price while promotions and questions have a negative effect [45].
The effect of animations is controversial, with some studies identifying them as a positive influencing factor [25], [27] and some as a negative influencing factor [44]. Still other surveys do not find a clearly positive influence [46], [47].
Especially relevant seems to be the picture in the user perception. This is seen by internet users even before the logo and the text on the banner as a study using eye-tracking technology shows [26].
In contrast, there is no homogeneous picture with regard to the use of suggestive prompts for action (often also called "call-to-action" or "action phrase"): Some sources identify these as having a negative effect on consumer interaction [44], while other, albeit much older sources identify them as positive [47].

Subject line design in e-mail marketing
In addition to the effect of the design in banner advertising, the effect of the design in e-mail marketing is examined in the following. The functionality of the advertising channel is similar in that it also addresses consumers who have already come to terms with the company's brand by registering for the newsletter (in the case of the app: by installing the app). Visually, however, advertising via e-mail newsletters is only comparable to push notifications to a limited extent. In this respect, the literature presented in the following focuses primarily on findings regarding subject lines. Subject lines can be seen as comparable to push notifications as the length of the text is similar.
In general, personalized, content-related subject lines are particularly suitable for achieving higher opening rates, as can be determined using A/B testing [30]. Other sources also emphasize the relevance of high information content in straightforward subject lines [29]. The use of emotions is also identified as a positive driver of open rates in email marketing messages based on an A/B test evaluation of an email message sent to 1,409,963 recipients [31].
With regard to the length of subject lines, contradictory findings are repeatedly found in research. It has been established that short subject lines do not have a significantly better effect than longer ones. Furthermore the research results of the same study indicate a slightly positive effect for custom and content-specific subject lines [30]. In some cases, there is evidence that very short subject lines achieve even worse results than longer ones [29].
Based on the Information Gap Theory [24], other research suggests that an empty subject line should actually achieve better interaction rates because it arouses the curiosity of the recipient. In fact, this assumption cannot be confirmed by the authors [48]. Nevertheless, other sources, contrary to the results presented above [29], [30], [48], find that short [31] or completely empty subject lines [49] perform better than longer subject lines.
These largely contradictory results of different studies show how significant the differences between theoretical and practical or experimental findings are [50].

Development of hypotheses
Regarding the effect of title, button and image, one hypothesis is made for each design element.
Since a title allows to display more information with a push notification and thus the advertisement appears larger [25][26][27], [44], [45] it is assumed that H1: A title has a positive effect on the probability that users tap a push notification. With regard to the use of a button, a positive effect on interaction rates is also assumed, since the button makes the push notification appear larger and acts as a callto-action [25], [27], [44], [47]. It is therefore stated that

H2: A button has a positive effect on the probability that users tap a push notification.
A picture also has a positive effect on the size of the push notification as an advertisement, especially since it plays a particularly important role in the user's perception process [25][26][27], [44]. A positive effect is also assumed here: H3: An image has a positive effect on the probability that users tap a push notification. Figure 1 shows the different options to include the respective design elements in a push notification. Empirical analysis

Experimental setup and data collection
The established hypotheses on the effect of title, image and button on the performance of push notifications are to be tested by using A/B Testing. This method is used regularly in similar experimental setups [30], [51]. Two push notifications with only one element differing are sent to two different, randomly selected target groups. By using the Chi-square test, the extent to which the observed frequencies differ from each other is examined. The null hypothesis assumes that the variables are statistically independent of each other [52]. Furthermore, the strength of the correlation is also measured by means of Cramer's V [53]. A value of 0.1 is considered weak, 0.3 as medium and a value of 0.5 or greater as strong correlation [54], [55].
To increase the reliability of the results, the effect of each design element is measured not only on one message, but on five messages. Thus, a total of five messages are sent three times in two variants each, i.e. 30 messages in total. The push notifications are sent per variant to 7,500 users of a mobile shopping app. Success is measured in each case by tapping the notification.

Statistical analysis and results
For each hypothesis it is shown per message how many push interactions were generated without the tested element and with the tested element. In addition, the Chi-square value and Cramer's V-value and the interpretation with respect to the hypothesis are shown. The following applies to all data: Each variant of each message was sent to a total of 7,500 users. Each Chi-square test is performed with c 2 crit = 3.84, df = 1, (1-α) = .95. Table 1 shows the results of the five A/B tests in which push notifications without and with titles were sent to 7,500 recipients each. In all cases, the Chi-square value determined is above the critical value, so that in all cases there is a significant difference. H1 is accordingly regarded as confirmed.
Using Cramer's V, the strength of the effect is determined. In all cases, this is below 0.1000, so that, despite confirmation of the hypothesis, only a weakly positive effect of a title on the openings can be assumed. Table 2 shows the results of the A/B tests regarding the button. In three out of five cases the openings with buttons are higher than those without buttons, but at the same time no significant difference can be detected. In two cases the openings without button are even higher than the openings with button. In one case this difference is even significant. H2 is rejected completely, but at least no negative effect of a button on the interaction rates can be assumed.
The results for the effect of an image as a design element of push notifications are shown in Table 3. While H3 is rejected in three cases, it can be confirmed in two cases. In these two cases, however, Cramer's V is only in the weak range. In summary, H3 is rejected accordingly. No clear evidence can be found that the presence of an image has a clearly positive effect on interaction rates. It must be noted that only the presence of an image was tested. Results may differ based on which specific image is used.

Summary and discussion
After reviewing existing research results on the effect of push notification design, related literature from the fields of banner advertising and e-mail marketing was considered. The developed hypotheses assume that the design elements title, button and image have a positive effect on the interaction rates. The hypotheses were empirically tested in several experiments. Only for the title such a positive correlation could be shown, whereas the strength of the effect is only weakly positive. Table 4 summarizes the hypotheses and their results.
The results make it clear that the already mentioned, partly significant differences between theoretical derivation and practical or experimental experience in e-mail marketing are also likely to be present in the area of mobile push notifications [50].
With regard to limitations, it must be noted that the results may vary depending on the context, e.g. shopping, news or messaging [28]. The results of this paper are therefore mainly related to mobile shopping apps. The metric is also a limitation in that the results can only be related to direct opening of push notifications, but not to indirect opening of push notifications [36]. Probably the most important limitation is that here only the presence of the mentioned elements was tested, but not the concrete design. It would be conceivable that there are differences in the effect on the probability of interaction, depending on how a title is concretely formulated or which picture is concretely chosen.

Conclusion and further research
This research fills an important research gap in the investigation of the effect of push notification with regard to the different design elements. Similar issues have already been extensively researched in other marketing channels and have been highlighted accordingly in this paper.
The results of the experiments carried out here to test the hypotheses put forward have a high practical relevance for marketing via mobile shopping apps. While a title Table 4. Summarized results of hypothesis testing

Hypothesis Results
H1: A title has a positive effect on the probability that users tap a push notification. Confirmed H2: A button has a positive effect on the probability that users tap a push notification. Rejected H3: An image has a positive effect on the probability that users tap a push notification. Rejected obviously has a positive effect on the probability of interaction, marketers should critically examine the effort required to create images for push notifications and also question the use of buttons. This paper focuses exclusively on the existence of the different design elements. For further research, the authors propose to examine in particular how concrete design variants of the different design elements affect interaction rates. It is conceivable, for example, that titles have different effects depending on their length, tonality, or content.