Usability in 4D AR: Visualising Multi-temporal Real-time Geo-data in Augmented Reality Environments

Even though advantages of 3D visualisation of multi-temporal geo-data versus 2D approaches have been widely proven, the particular pertaining challenge of realtime visualisation of geo-data in mobile Digital Earth applications has not been thoroughly tackled so far. In the emerging field of Augmented Reality (AR), research needs comprise finding the optimal information density, the interplay between orientation data in the background and other information layers, using the appropriate graphical variables for display, or selecting real-time base data with adequate quality and suitable spatial accuracy. In this paper we present a concept for integrating real-time data into 4D (three spatial dimensions plus time) AR environments, i.e., data with “high” spatial and temporal variations. We focus on three research challenges: 1.) high-performance integration of real-time data into AR; 2.) usability design in terms of displaying spatio-temporal developments and the interaction with the application; and 3.) design considerations regarding reality vs. virtuality, visualisation complexity and information density. We validated our approach in a prototypical application and extracted several limitations and future research areas including natural feature recognition, the cross-connection of (oftentimes monolithic) AR interface developments and well-established cartographic principles, or fostering the understanding of the temporal context in dynamic 4D Augmented Reality


INTRODUCTION
In the emerging field of Digital Earth research, 3D visualisation of multi-temporal geo-data potentially has considerable advantages versus 2D approaches in effectively conveying spatial content [1]. Thus, threedimensional representations of spatial structures and processes have recently gained significant importance. However, a particular pertaining challenge in portraying 3D geo-data is the development of algorithms that allow for real-time visualisation of geo-data for time-critical applications. This is particularly true for rendering realtime information in mobile map-based geo-apps.
A specific unsolved methodological and technological issue for these mobile applications is the integration of additional external 3D real-time information into map-based visualisation systems. Here, challenges comprise finding the optimal information density, not covering the orientation data (i.e., the camera image) in the background with other information layers, using the appropriate graphical variables for display (cp. [2]), or finding and aggregating real-time base data that is suitable for the use case at hand.
Consequently, mobile users often suffer from a lack of data or non-optimal visual presentation. On the one hand, this means that systems purely reflecting the real world are oftentimes not sufficient and usable. On the other hand, entirely virtual systems are too much disconnected from the physical world for most purposes as orientation and navigation in physical space require a complex transfer effort to interpret information from a virtual environment and act accordingly in the physical environment.
Augmented Reality (AR) seems to be a suitable approach to tackle these issues. The concept of Augmented Reality is basically defined in the realityvirtuality continuum [3] as a system with three distinct properties [4]: 1.) the virtual reality and the reality are interlinked; 2.) user interaction happens in real time; and 3.) real and virtual objects are related to each other in three-dimensional space. Yet so far, research in the area of AR widely focussed on the integration of static data (i.e., data with no "high" spatial and temporal variations) such as Wikipedia entries [5], mountain peaks [6], or points of interest (POI) like restaurants [7].
Thus, the presented approach targets two use cases (even though it is not limited to them as shown in the methodology section) involving time-varying data: mountaineering (support of mountaineers in their shortterm decisions on whether to continue their tour or to turn around to avoid dangerous situations) and tourism (support of tourists' short-term decisions in their visiting schedule). As AR aspects 1.) and 3.), which have been mentioned in the previous paragraph, are inherently true for the given use cases, this paper focuses on the real-time aspect -not solely from an interaction viewpoint, but also from a real-time data perspective. Hence, this paper presents an approach to integrate real-time data for spacetime varying phenomena into Augmented Reality environments. AR has been chosen as the underlying visualisation concept as quasi-realistic 4D rendering has considerable advantages over traditional 2D maps in PAPER USABILITY IN 4D AR: VISUALISING MULTI-TEMPORAL REAL-TIME GEO-DATA IN AUGMENTED REALITY ENVIRONMENTS conveying the dynamic nature of environmental processes as shown in [8].
The research questions for the presented approach are defined as follows: • Which usability criteria need to be particularly considered with AR visualisations and which advantages do these visualisations have over other approaches? • How can dynamic geo-objects be integrated into AR environments in a quasi-realistic fashion while preserving high performance in the visualisation and interaction? • What are the main challenges in visualising multitemporal real-time geo-data on mobile handsets with a particular focus on usability in AR environments, and how can they be addressed to comply with demands for supporting time-critical decisions? In brief, the goals of our research are in the areas of real-time data integration, visualisation of high-resolution spatio-temporal data on standard mobile devices (smartphones and tablet PCs), definition of usability criteria and design guidelines for real-time 4D AR, and proving the practical feasibility of our approach through a prototypical realisation. Consequently -in contrast to numerous previous research efforts -our research does not focus on natural feature recognition, object detection, landscape extraction, improving location or positioning accuracy, image-based structure analysis, and the development of new physical see-through AR interfaces.
The paper at hand is structured as follows: This introduction is followed by a description of related work in the areas of AR applications and 4D visualisations including a clear identification of pending research gaps. Then, our methodology for visualising spatio-temporal real-time geo-data in AR environments is presented, followed by a description of the validation through a prototypical implementation. Finally, we discuss our approach (advantages, limitations and future research avenues) and end the paper with a number of key conclusions.

II. RELATED WORK
First realisations of the extended representation of surroundings for mobile applications go back to [9]. With the head-mounted three-dimensional display described therein, simple wireframe drawings in the direct surroundings of the user could be represented in real time. However, the term 'Augmented Reality' was primarily coined only some years later in [10], in which AR was mainly targeted at saving graphic performance compared to the originally planned virtual reality.
According to [11], the underlying functionality of AR systems can be divided into two categories: computer vision-based AR (using markers) and location-based AR. The position of the virtual additional content in computer vision-based approaches can be detected by means of natural features in existing, conventional images or artificial markers such as, for example, previously placed markers in the real world [12]. In many applications, however, the previous placing of markers is not always feasible or associated particularly for larger areas with high costs.
With natural feature recognition it is possible to detect and track almost any existing image, but the target images on which to place an augmentation have to be supplied to the system beforehand [13]. However, particularly when aiming for world-wide applicability, it would be too expensive to store fine-grained 3D models on the phone or to transfer them on demand. Wagner et al. state that feature detection is computationally complex for mobile devices [14]. Furthermore, the authors mention that feature detection algorithms always require a radical trade-off between memory usage, speed and robustness. This is also confirmed by Schmalstieg et al. [13] who identified one of the main problems being "that mobile phones have limited processing power, while computer vision algorithms typically perform heavy computations". Finally, Almer et al. state that a highly accurate position of the mobile and its orientation are necessary to perform high-quality feature recognition, which is oftentimes not the case [15]. This especially applies to urban environments where positioning accuracy is compromised by the built-up structure and by the need for feature matching over short distances, making the identification of real-world objects even more challenging. More, the authors acknowledge that feature detection has only been proven under optimal conditions in terms of good illumination conditions, moderate changes of viewpoints with regard to reference images, high contrast textures and images, etc. This is a particular issue for time-varying geographic features such as flooding areas, snow cover, human crowds, or cloud cover.
The location-based AR on the other hand positions virtual AR content based on measurements obtained from sensors such as GPS, compass and motion sensors. Since potential inaccuracies of the individual sensors add up, the positioning accuracy of this location-based approach is usually lower. There are some current research efforts such as the one presented by Ventura and Höllerer [16] that improve the positioning accuracy, but they often require explicit user input, special sensors or additional information.
However, for many applications GPS accuracy is sufficient and provides world-wide coverage so that many AR applications nowadays use the locationbased approach. Some of today's most common and least expensive AR platforms are mobile devices in the form of smartphones and tablet computers. An early implementation of the marker-based tracking approach was described as a thin client AR phone by Assad et al. [17]. Further marker-based approaches can be found in [18] and [19]. First implementations of location-based approaches for smartphones were developed as part of the MARA project [20].
Newer location-based approaches for self-developed platforms or comparably high-performance mobile computers than simple smartphones are already more advanced in their development. In SiteLens [21], for instance, geo-data at a defined point of time are visualised by colouring several shapes such as spheres, cylinders, and smoke. An impression of a time series visualisation is presented in HYDROSYS [22], by combining video footage with a required digital terrain model, a similar approach as developed by [15]. Another approach for visualising time-oriented data in AR for selected regions of interest in a segmented scene is given in [23]. The essential shortcomings of these approaches are threefold: 1.) no real-time data are integrated; 2.) temporal PAPER USABILITY IN 4D AR: VISUALISING MULTI-TEMPORAL REAL-TIME GEO-DATA IN AUGMENTED REALITY ENVIRONMENTS navigation is deficient (through clicking one of six buttons for discrete steps in [23]); 3.) they are mainly just a modification of existing visualisation approaches without new concepts for presentation of geographic information. The same applies to the approach by Golparvar-Fard and Ham [24]. Furthermore, "AR Weather" [25], a promising AR-based weather application, differs from our approach in that it does not account for real-time data integration, but just simulated data. More, it mainly presents a system implementation description without distinct and scientifically derived design decisions, usability is not considered, and the user interface -a standalone system including a helmet-mounted see-through display -is not suitable for everyday usage.
Consequently, data visualisation of a time series in most previous AR approaches was either not suitable for timecritical decisions on most common smartphones or limited to static and time-invariant geo-data, which are oftentimes regionally restricted [5]- [7]. A 3D real-time visualisation in AR running continuously over a period of time has not yet been developed. This is mostly due to hardware limitations of mobile devices, but also to the special requirements for the geo-data to be visualised. Furthermore, data needs to be available in a high temporal resolution for time series visualisations. In addition, the information should also be available in a high spatial resolution so that the direct surroundings of the user can be accurately extended with virtual content for a locationbased service at any place in the world.
In conclusion, the research gaps of previous AR approaches for today's most common mobile devices in the form of smartphones and tablet computers can be identified as follows: 1.) no integration of time-varying real-time geo-data in high spatial and temporal resolutions; 2.) no clear design and usability guidelines and distinct design decisions; 3.) lacking real-time ability; 4.) little to no support for 4D data representations; 5.) no or non-satisfactory temporal navigation possibilities; and 6.) strong focus on marker-based methods, which are not suitable for many dynamic AR applications.

III. CONCEPT FOR 4D VISUALISATION OF GEO-DATA
IN AR This section describes the developed concept to integrate multi-temporal geo-data in Augmented Reality environments. First, we set the scene for our research by laying out the requirements of two distinct use cases. Then, we present a thorough analysis of usability criteria that need to be considered, and finally end the section with specific design considerations for the development of realtime AR applications integrating spatio-temporal geodata.

A. Use Case Description
The necessity of integrating up-to-date time-varying data into Augmented Reality (AR) environments arises from the need for short-term decision support in near real time as required by a number of time-critical application areas. This clearly distinguishes the concept presented in this paper from previous approaches as these have mainly focused on displaying static data or data with low spatiotemporal variability. For the presented approach we extract the essential requirements from two distinct use cases, but it is not limited to them.
The first use case revolves around mountaineering. Mountaineers are oftentimes facing critical situations due to quickly changing weather conditions that require fast decisions. Mountaineers often run into danger due to rain, snowfall, dropping temperatures or darkness setting in. This is mostly the result of lacking information about the current weather and its short-term development. The presented AR-based approach aims to support mountaineers in their short-term decisions on whether to continue their tour or to turn around to avoid dangerous situations.
The second use case focuses on the area of tourism. Tourists strolling around in an unknown city are often not familiar with the local or regional climate and the distances they need to overcome. Similarly to the mountaineering use case, the presented AR-based approach aims to support tourists' short-term decisions in their visiting schedule -e.g., whether they should continue their tour visiting another monument or look for shelter from the quickly approaching rainfall.
Apart from these two use cases, the presented approach is also transferrable to other application scenarios including public transport (Which buses will leave at a certain track and are there free seats available?), energy system monitoring (How much power do my solar power plants generate and how much will they produce in the next hour considering the weather forecast?), or event management (assessing crowd movements by visualising densities of geo-social media data). Although these application scenarios could be realised using the presented concept, the major restricting factor is the scarce availability of real-time data in these areas.
While it would be possible for the described use cases to provide the basic information for the decision process through more traditional user interfaces, we argue that the usability, efficiency and effectiveness of the presented approach is distinctively higher and will result in a noticeably better quality of the decision and ultimately superior user experience. We will motivate this statement over the course of the following paragraphs.

B. Usability
Usability is an essentially important aspect for Augmented Reality applications even more than for other mobile applications. The visualisation of data using Augmented Reality methods offers the opportunity to considerably reduce the cognitive effort needed by the user to gather, interpret and understand the presented information in a spatial context. This is due to the fact that AR removes the abstract representation on a map or in the form of plain text and instead displays the information in direct relation to the perceived real objects and locations. However, this statement only holds true if the AR visualisations are intuitive and strike the right balance between richness of the provided data and information overload, between necessary detail and simplicity.
Usability generally constitutes a quality criterion that describes how simply and intuitively user interfaces can be interacted with [26] Usability is defined in the ISO 9241-11 norm as the "extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" [27]. Nielsen's original definition contained three more aspects as a part of system PAPER USABILITY IN 4D AR: VISUALISING MULTI-TEMPORAL REAL-TIME GEO-DATA IN AUGMENTED REALITY ENVIRONMENTS acceptability: learnability, memorability, and errors. The following paragraphs lay out for each parameter 1.) a general description of the criterion and 2.) how the concept presented in this paper accounts for the parameter.
Users: This parameter describes users or user groups in terms of their pre-knowledge, experience, usage context (physical, technical and social), age, etc. In the presented concept, we derived the typical user for our application from a typical smartphone user as determined in a comprehensive telephone interview based study involving 2,277 persons [28]. The choice of this study builds on the fact that 42% of mobile phone users own a smartphone, where 87% of all smartphone users use their smartphone to access the Internet or email. Our typical user runs the application independently of their physical location (urban vs. rural), as the application is intended to serve different user groups ranging from tourists in urban areas to mountaineers in highly rural areas. More, the application is basically used independently of the user's education level. It focuses on the age group between 18 and 45 years, in which approx. 50% of all persons own a smartphone. The mentioned report [28] clearly shows that smartphone usage is independent of gender in section on "demographic differences in smartphone Internet use".
Goals: This parameter defines which tasks a user performs with an application and which goals shall be achieved. For the presented real-time AR use cases, the goal is to get an idea about the future development of the weather situation in terms of precipitation, cloud cover, wind and sunshine. In addition to current conditions, the user shall be able to retrieve short-term forecasts and a time series visualisation of previous conditions at the current and at distant locations. These goals have been implicitly derived from the use case description above.
Context: This parameter defines the usage context composed of the physical context (physical environment, degree of mobility, etc.), the technical context (type of device, device capabilities, used software, etc.) and the social context (distractors, interaction with other persons, public or private environment, etc.). For the presented real-time AR use cases, users will typically be outside in public space and move around at low velocity, e.g., walking speed. They use a standard smartphone (e.g., 3.5-5" display with multi-touch capabilities, 1.4GHz duo-core processor, 8GB of internal memory, 8 megapixel camera, 3G data connection). The user is usually in company of one or more persons sharing the interest of retrieving information of the current weather situation and its shortterm development.
Learnability: Learnability indicates how easy it is for users to perform the defined tasks the first time they use an application. For the concept presented in this paper we decided to give the user a short visual introduction into the functionality of the application. Here, it is crucially important to find the optimal trade-off between a comprehensive textual introduction (too much textual information can potentially overburden the user and result in fading interest) and conveying the necessary information needed for the user to operate the application with confidence. Thus, it was decided to use illustrations, which are enriched with short textual hints, to point the users to the essential core functions, as shown in Figure 1. Efficiency and Effectiveness: These parameters state how quickly (in terms of time and effort) and to what degree (how effectively) users can perform a certain task once they have established proficiency in using the application. As the real-time AR use cases focus on timecritical decision support, the user interface's design has to be as simple as possible to be comprehensible in very short time. The AR view on a mobile provides an ideal representation for the user to quickly grasp and understand tempo-spatial data related to the current location and context as opposed to a small section of a map. Thus, the major part of the viewport contains the current camera image including the virtual, dynamic geo-objects. A pseudo-realistic (close to reality, but easily distinguishable from it) illustration of the geo-objects makes them stand out and easy to identify. The relevant environmental phenomena (e.g., air temperature, cloud cover, relative humidity, rainfall, snowfall, wind, etc.) are represented in a generalised manner to optimise information density on the screen. More detailed information about particular measurement values, further information or object settings need to be available on demand with a simple click on a specific object. This results in the central advantage that users do not have to interact with the application via complex hierarchical menus to accomplish their central tasks. If a user is interested in more detailed information, the click on a specific object reveals different types of diagrams (line diagrams and bar charts) to clearly present the temporal development of the phenomenon's value over a certain period of time. Figure 2 shows the time series graph of five weather-related parameters (UV index, felt temperature, wind chill, heat index and temperature).
Furthermore, the presented concept foresees a caching mechanism in order to pre-load weather conditions in the adjacent raster cells around the current position (in all directions). This results in a smooth experience while moving through physical space without having to wait for the according location-dependent data. Additionally to the weather conditions in the immediate surroundings, a user can also choose to view the weather in other places using a separate map interface. In order to increase performance in displaying weather-related phenomena -particularly on low-end devices with smaller displays -users have the possibility to show and hide specific weather variables.  Memorability: Memorability defines how easy it is for users to re-establish their proficiency when they re-use an application after not having used it for a certain period. Generally, we expect users to download the application and use it for a specific period (e.g. a holiday trip or a ski tour), and then approximately once every other week. To account for this usage pattern, the presented concept displays icons that are easier to remember than text-based menus. Furthermore, our concept uses context-based menus to maximise intuitive handling of augmented information; for instance, detailed information and properties for the sun object are presented to the user when interacting with the augmented sun geo-object.
Errors: This parameter indicates how many errors users make and how these errors affect usage of an application, i.e., how users recover from these errors. In the presented concept, we focus on the creation of an error-tolerant application which is designed to prevent errors caused by the accidental interaction. One strategy to avoid incorrect user actions is to use distinctive and clearly identifiable icons for all buttons. Additionally, the icons are big enough so that they can be easily clicked. This aspect is seemingly trivial, but it has been neglected by a large number of mobile applications even though it is especially important in the described mobile context where clicking precision is compromised when the user is moving around in outdoor environments. Furthermore, the interface design aims at limiting users' choices to "correct" ones by providing sub-selected criteria and helper tools. For instance, inputting time and date to view the temporal development of a phenomenon is done via a time slider and a visual calendar to prevent users from making invalid actions. If, despite these measures, a user inadvertently performs an undesired action, it can be reversed easily at any time without having to restart the application. Furthermore, we also consider situations that are mistakenly perceived by the user as an error. These often occur during longer waiting times when either the application does not respond immediately to any user input or the user does not get instant feedback on their interaction. To prevent this behaviour, all images are precached when starting the application in order to decrease loading times and to enable a smoother interaction flow in the application. In addition to that, all history charts are replaced by a short note on less powerful mobile devices to avoid delayed appearance, since their computation is time-consuming. For example, in displaying detailed information about a specific weather variable, the user initially only receives the textual information together with a note that the history chart will appear momentarily.
Satisfaction: Finally, satisfaction describes how subjectively pleasant it is for users to use the application. Therefore, the presented concept strongly focuses on fluent transitions of discrete weather conditions in subsequent time steps. Besides the visually appealing presentation it conveys the impression of continuously developing weather conditions and a sense of coherent temporal context as opposed to a discrete sequence of chronological steps. Here, particularly the display of geoobjects is highly resource-consuming. Thus, as many calculations as possible are performed either in the background or during the initialisation process in an asynchronous fashion when starting the application in order to minimise loading times while using the application, and overly complex animations and large graphics are avoided. Another method to optimise a user's subjective satisfaction is to display further information about the weather variables on request only. This enhances a user's ability to interpret a value for a specific phenomenon in case of insufficient background knowledge.
The above paragraphs contain a basic description of the single parameters and their use in the presented real-time AR approach. More detailed information on the definition of the criteria can be found in the ISO 9241-11 norm [27] and in [26]. The description of our considerations in the design phase of the concrete application of the concept for validation purposes is descriptive and has not been validated in a user study as the goal of this paper is not a usability evaluation, but the demonstration of the opportunities of the integration of real-time data into AR environments.

C. Design Considerations
Apart from the usability criteria described in the previous sub-section, several additional design decisions were taken to optimise the practicability and the applicability of the concept.
First of all, the question of representing spatio-temporal phenomena in 4D is a prevailing challenge. As mentioned in the introduction, the advantages of 3D representations of real-world phenomena are the basis for the design of a 4D interface (3D + time) for presenting environmental phenomena in an interactive spatio-temporal AR application. According to Jobst and Germanichs [1], the optimised design of the user interface is a central quality aspect in mobile applications. This requirement is particularly important for mobile geo-apps, which offer new and more dynamic ways of presenting geospatial information. Thus, the presented approach accounts for dynamic interaction, scale-based data sub-selection, graphical presentation of geographic features, and ondemand data visualisation [29] in a 4D AR environment. PAPER USABILITY IN 4D AR: VISUALISING MULTI-TEMPORAL REAL-TIME GEO-DATA IN AUGMENTED REALITY ENVIRONMENTS Furthermore, the approach's ability to integrate realtime data is critical to its success. As previous approaches have not yet dealt with real-time data, there were no references in terms of related design considerations and conceptual strategies. We tried to account for challenges like limited connectivity, small screen sizes and varying brightness by developing caching mechanisms, efficient data rendering, and optimising information density -as described above.
Another challenge that needs to be thoroughly tackled is that mobile users often suffer from a lack of data or non-optimal presentation. On the one hand, this means that systems purely reflecting the real world are oftentimes not sufficient and usable. On the other hand, entirely virtual systems are too much disconnected from the physical world for most purposes as orientation and navigation in the physical world requires a complex transfer effort to interpret information from a virtual environment and act accordingly in the physical environment. Consequently, the presented approach tries to find a suitable compromise between providing sufficient information density to support users' short-term decisions and leaving enough "non-augmented" space reflecting the real world in order to support orientation and navigation. The figures in the section on "Validation -Prototypical Implementation" show how this compromise is achieved.
Finally, the related issue of the influence of the visualisation complexity on the user, varying from simple abstraction to photorealistic representation of the spatiotemporal phenomenon, with regards to the comprehensibility of a spatio-temporal development has not been extensively studied. Most mobile users find a pseudo-photorealistic illustration of environmental phenomena that is closely related to the objects' physical appearance more helpful because it conveys a more realistic and more efficiently comprehensible view on a real-world situation. Thus, the presented concept proposes a pseudo-photorealistic representation of these phenomena, communicating the actual environmental processes and weather-related developments in a way that is easy to understand and interpret. This decision has been taken in accordance with the most recent research outputs [8].

IV. VALIDATION -PROTOTYPICAL IMPLEMENTATION
In order to prove our concept's practicability, we created a prototypical implementation for validation purposes that focuses on the real-time visualisation of multi-temporal geo-data. For the actual realisation we used the ARchitect engine of the Wikitude AR browser [5] as the AR environment. It is based on established web technologies such as HTML5, JavaScript, and CSS, and supports all possibilities of a mobile web browser for the visualisation of HTML content. Furthermore, the AR engine offers the advantage that even complex AR applications can be implemented without being particularly dependent on to the underlying operating system or the device-specific hardware. Therefore, the AR content can be directly used for other mobile platforms, operating systems and browser technologies as independent applications without having to resort to external display applications.
The basic user interface of the developed real-time weather application is shown in Figure 3. The upper part of the interface (framed in red) contains an information bar in the form of a head-up display that includes the most commonly needed data (air temperature, precipitation, risk of precipitation, wind speed and wind direction) at a glance. Further information and charts for specific weather variables can be faded in by clicking on the information bar. General settings for the selection of weather variables or the place for which the weather is to be displayed (the user's current or a distant location) can be chosen here as well. The lower part of the application interface (framed in orange) contains a slider for choosing the desired time within the series and a play/stop button. With these interaction tools, users can either display the weather for any freely selected point of time or the animated change of weather at any chosen time. The main part of the prototypical application (framed in green) is reserved for the camera image setting the scene for the visualisation of the various weather variables which are integrated as dynamic AR geo-objects in the image of the real environment. All weather objects can be displayed as time series visualisations with dynamic geospatial positions; i.e., a weather front is not a simple screen overlay, but it is virtually moving "towards" the user in the 4D AR interface. The mapping from the real-world position to the virtual position in the three spatial dimensions of the AR interface is calculated in the AR engine by using viewport parameters derived from the position and orientation sensors. The visualisation strategies for these variables are explained in more detail in the following paragraphs. Cloud cover: The visualisation of the cloud cover is composed of four layers that can be dynamically combined according to the actual weather situation. The top layer, which represents a completely cloudless sky, is kept in a light blue colour. The second level visualises a cloud-covered sky and contains a Cascading Style Sheets (CSS) sprite composed of successive Perlin noise states [30]. The delayed display of the individual Perlin noise states simulates the movement of the clouds. The intermediate stages of a cloud-free and a fully overcast PAPER USABILITY IN 4D AR: VISUALISING MULTI-TEMPORAL REAL-TIME GEO-DATA IN AUGMENTED REALITY ENVIRONMENTS sky are controlled by the opacity of the second level. For instance, in case of lightly covered skies, the high transparency of the second layer lets the light blue colour of the top layer shine through the clouds in some places conveying a quasi-realistic impression. The third layer is kept in a dark grey colour and its opacity regulates the brightness of the sky with its visible clouds. This visualisation strategy has been chosen to make the sky appear darker in "bad" weather than in "good" weather regardless of the degree of cloudiness. The fourth and lowest layer visualises the individual clouds using different pseudo-photorealistic graphics that move depending on the speed and direction of the wind. The visualisation of the sky (clear sky on the left and cloud cover on the right) is shown in Figure 4.
Precipitation: Figure 5 illustrates the visualisation of precipitation including rainfall and snowfall. The upper part of the figure shows precipitation depending on the temperature variable through raindrops (left) or snowflakes (right) that fall from the clouds. The precipitation quantity is indicated through the fall velocity and number of raindrops or snowflakes respectively, while the probability of precipitation is reflected by the graphics' opacity. To make strong precipitation (heavy rainfalls and snowfalls) clearly distinguishable from light precipitation (light drizzle), an animated precipitation front is displayed instead of individual raindrops or snowflakes. The lower part of Figure 5 illustrates heavy rainfall (left) and heavy snowfall (right).

Sun and Moon:
The prototypical implementation also illustrates the current position of the sun and the moon. As shown in Figure 7, the position of the sun is visualised by a graphic element which is embedded in the camera image of the real environment. The sun's visibility is indicated through the opacity of the graphics. In case the sun's position is outside of the current viewport, a direction indicator in the form of a yellow arrow points in the direction of the sun. In order to provide visual information about the sun's path of the current day, small yellow circles are included to show the position of the sun at each full hour and half hour. Like this, users cannot only see the sun's current position, but also view its future positions, e.g. to assess shadowing effects for a certain location at different times during the day. Furthermore, we integrated a navigation menu for the sun, which can be faded in and out by clicking the sun. It allows users either to display further information about the sun of the current day or to obtain information about the sun's course at a specific date. The visualisation of the moon is implemented analogue to that of the sun by using different graphics and colours which also reflect the current phase of the moon. Figure 7 illustrates the visualisation of the sun (left), the indicator arrow to the sun's position (middle), and the navigation menu for the moon (right).
Wind: As shown in Figure 6, we visualise the wind variable through the display of waved arrows that move through the viewport. Their direction and speed depends on the actual current direction and speed of the wind.

V. DISCUSSION AND FUTURE RESEARCH AVENUES
This section discusses the presented approach and the produced results, points out its limitations and pin-points future research avenues that need to be tackled in the area of 4D real-time AR. The research questions mentioned in the introduction (usability criteria, integration of dynamic geo-objects, real-time visualisation of multi-temporal geodata) are implicitly answered in the following subsections.

A. Advantages of Multi-temporal Real-time Geo-data in Augmented Reality
One of the main advantages of the presented real-time AR approach over conventional methods is the intuitive visualisation of real-time geo-data. This helps people to easily grasp the presented information and to minimise the transfer effort from the digital representation to the physical state compared to traditional map-based interfaces.
This is of particular importance as processing timevarying geo-data and interpreting them in the current spatio-temporal context is a cognitively challenging activity.
Eliminating conventional, abstract representations, i.e. on a map, and directly overlaying the information on the live camera images makes it easier to gather, process and understand the presented information, and ultimately draw conclusions based on them. This is specifically important for mobile usage scenarios where a user's attention span is typically limited to less than 30 seconds [31]. For instance by trend, people find it easier to get an understanding of a forecasted wind direction and intensity when visualised using animated arrows (s. Figure  6) [32] as opposed to the raw data indicating a cardinal direction of SW with 20 km/h. In some cases only an AR representation may be able to convey the information in a way that it can be clearly interpreted in a specific usage context. For instance, given the data of the sun's solar altitude over the course of a day, the user wants to know at which time the sun will be obstructed by a mountain or building. Our AR-based approach seems to be well-suited for conveying such information correlations through an easy-to-use interface.
Another aspect that underpins the presented real-time AR concept is that mobile devices such as smartphones and tablets have recently gained enormous traction. According to International Data Corporation (IDC) reports, tablet shipments grew by 80% in 2012 and are forecasted to surpass the number of desktop PC shipments in this year already [33]. The report states that the main reasons for the huge popularity of these devices are the simple user interface, much better usability of the operating system, and the accompanying applications when compared to traditional personal computers. For this new generation of innovative devices and user interface concepts, our AR-based approach is well-suited to leverage this trend with regard to data visualisation by providing an intuitive method for presenting and visualising time-varying geo-spatial information in a way that is easy to understand for the target audience, even more so for high-resolution real-time data.
Another development that underlines the necessity of a real-time AR-based approach is the quick rise of the novel market segment of Augmented Reality glasses that has recently received a lot of attention. On these AR glasses, the presentation of information happens via an AR overlay in a natural manner by directly augmenting the field of vision. At the same time AR glasses provide the unique opportunity to present information in a completely selfactuated fashion at any suitable time depending on the user's context. This development clearly goes beyond traditional usage patterns forcing the user to consciously make the decision to request information and pull their smartphone out of the pocket to retrieve the desired information. For the concept presented in this paper, AR glasses are a useful means to overlay, for instance, the names of mountain peaks when glancing over the horizon (helping the mountaineer to decide which trail branch to take by overlaying navigation instructions) or displaying the live time table of a bus stop to support personalised urban transport.

B. Limitations and Challenges
After having pointed out the advantages of our approach for integrating real-time data into 4D Augmented Reality environments in the previous subsection, this sub-section discusses the limitations of the presented concept.
A significant limitation is the need for a high-quality data connection for the permanent transfer of real-time data. This can be a particular issue for the mountaineering use case described above, as connectivity is not always given in remote sites like mountains. Furthermore, PAPER USABILITY IN 4D AR: VISUALISING MULTI-TEMPORAL REAL-TIME GEO-DATA IN AUGMENTED REALITY ENVIRONMENTS roaming in other countries limits the use of the application due to high prices for data transfer services. We believe that this challenge will be solved as well on legal (federally regulated roaming fees as already introduced by the EU) as on technical levels (even better network coverage, e.g. using new technologies like Long Term Evolution (LTE), also known as 4G). This is underpinned by recent discussions in the European Commission (EC) to abolish roaming fees within all member states of the EU, as proposed by EC Vice President Neelie Kroes in May 2013.
A related issue is the need for a permanent data connection as real-time data need to be fetched continuously. This makes it hard to make this concept work for an offline version of the application. One way to handle this limitation is to develop caching algorithms that download data in advance (e.g., weather forecasts) or even try to anticipate cell phone connectivity using coverage maps of mobile networks.
More, we need to tackle limitations regarding the complexity of interacting with the application. On the one hand, this results from the fact that the user is mobile, thus moving around continuously which can make user input on the go tedious and error-prone. On the other hand, the small screen size severely limits the amount of data that can be displayed. These issues will probably be solved within the next years as other forms of user interaction such as natural language based, gesture-based, braincomputer interfaces, etc. will arise.
Finally, there are some limitations that apply to location-based services (LBS) in general -which are in turn naturally valid for AR applications in particular. These include position accuracy which does not negatively impact the illustration of weather conditions but may impede the visualisation of information on a small scale like navigation instructions or bus time tables. For outdoor usage scenarios legibility of the display is deteriorated in direct sunlight which needs to be taken into account by using high contrast colour schemes. Depending on the use case and amount of geo data that need to be processed, resource constraints regarding CPU performance or available memory may be limiting factors.

C. Future Research Avenues
One of the absolutely most pressing research questions in the area of Augmented Reality is the development of methods for natural feature recognition. This results from the traditional problem of AR environments that augmented information is automatically placed in front of the abstraction of the real world, which oftentimes causes a user's perception that the augmented objects are somewhat intrusive and appear non-realistic because their position is not well aligned with the real world. Even though some of these issues have been solved for technological high-performance environments, a prevailing major challenge in mobile AR systems is their resource-constrained nature that restricts feature recognition by object-based image analysis (OBIA) techniques in real time. Furthermore, natural features such as trees, rivers, mountains, forests or also urban structures are difficult to identify using image processing methods as their morphology is not homogeneous and it is subject to constant change. Oftentimes structural information about the environment is not available so that visual interactions between virtual and real object cannot be realistically represented, e.g., the occlusion of a cloud when moving behind a skyscraper.
A related problem that has not been tackled in our research so far is dynamic lighting (shadows, etc.) to convey a more realistic impression of the augmented information. This issue goes hand in hand with the realistic presentation of 3D elements like conveying a real-looking perspective, embedding them into the map environment, and making them suitable for interaction.
Another central question for AR applications is how we can foster the understanding of the temporal context in dynamic 4D AR environments; in other words: How can a viewer's perception of a spatio-temporal development be supported in the most efficient manner for advanced cognition? [34] In case of the presented research on environmental data we proposed and integrated animations conveying the temporal changes of data as well as complementary diagrams (histograms, statistics, etc.) to visualise dynamic developments over time. Yet, we have not scientifically proven how those diagrams affect the users' understanding of a variable's temporal development.
A tightly coupled research avenue is to develop methods for optimised temporal generalisation. In previous approaches, this has only been achieved on a perapplication basis as the temporal granularity essentially depends on the presented data, the application and the goals or tasks the user wants to achieve. For example, a 15-minute aggregation interval might be suitable to support weather-related decisions whereas in the application area of urban public transportation, a 1-minute interval would be necessary. This problem seems to be of generic nature and needs to be parameterised to be solvable.
A fundamental issue is the cross-connection of (oftentimes monolithic) AR interface developments and well-established cartographic principles. Although a number of possible graphical variations have been identified for 4D maps, a true mapping to the thematic expressivity is still missing. The visual variables defined by Bertin for 2D cartography [2] are a subset of all thinkable and technically modifiable graphical variables, i.e., the subset that allows reliable mappings from data to visual depiction. Current literature often mixes the concepts with the result of diluting both. Especially in computer screen 3D environments, for instance, size, colour, texture and lighting effects need to interact to provide illusions of three-dimensional depth. This is related to the profound inverse optics problem as it is also encountered in 2D maps [35], which stringently limits the expressive capabilities of many graphical modifications to actually achieve dual coding [36] such as varying graphical variables like colour, shape, intensity or the zvalue extrusion.
Another development, which opens up a number of research questions, is the quick rise of the novel market segment of Augmented Reality glasses that has recently received a lot of attention. As mentioned in sub-section V.A, these AR glasses overlay augmented content on the glasses in front of the eye thus directly augmenting the field of vision. In this area, a central research question will be to find new ways to combine AR glasses with specifically tailored presentation concepts of multitemporal real-time geo-data in Augmented Reality that PAPER USABILITY IN 4D AR: VISUALISING MULTI-TEMPORAL REAL-TIME GEO-DATA IN AUGMENTED REALITY ENVIRONMENTS can assist and speed up the user's information retrieval and decision making process.
Furthermore, there are a number of open questions including additional interaction possibilities (e.g., eyetracking based or brain-computer interfaces), the consistent definition of graphical variables for 3D and 4D, optimised dynamic colouring strategies (cp. dual coding as mentioned above) and the development of guidelines for user experience design in 3D/4D cartographic applications.

VI. CONCLUSION
3D visualisation of multi-temporal geo-data potentially has considerable advantages over 2D approaches in effectively conveying spatial content [1]. Yet, a particular pertaining challenge in portraying 3D geo-data is the development of algorithms that allow for real-time visualisation of geo-data in mobile Augmented Reality (AR) systems. Here, challenges comprise finding the optimal information density, not covering the orientation data (i.e., the camera image) in the background with other information layers, using the "right" graphical variables for display [2] and finding suitable real-time base data.
Consequently, mobile users often suffer from a lack of data or non-optimal presentation. On the one hand, this means that systems purely reflecting the real world are oftentimes not sufficient and usable. On the other hand, entirely virtual systems are too much disconnected from the physical world for most purposes as orientation and navigation in the physical world require a complex transfer effort to interpret information from a virtual environment and act accordingly in the physical environment. Thus, AR has been chosen as the underlying visualisation concept as quasi-realistic 4D rendering has considerable advantages over traditional 2D maps in conveying the dynamic nature of environmental processes as shown in [8].
From previous research approaches on AR for today's most common mobile devices, we extracted the following research gaps: 1.) no integration of time-varying real-time geo-data in high spatial and temporal resolutions; 2.) no clear design and usability guidelines and distinct design decisions; 3.) lacking real-time ability; 4.) little to no support for 4D data representations; 5.) no or nonsatisfactory temporal navigation possibilities; and 6.) strong focus on marker-based methods, which are not suitable for many dynamic AR applications; In this paper we present a concept for integrating realtime data into AR environments -in contrast to previous approaches that solely focussed on the integration of static data (i.e., data with no "high" spatial and temporal variations). The presented approach targets two use cases (even though it is not limited to them as shown in the methodology section): mountaineering (support mountaineers in their short-term decisions on whether to continue their tour or to turn around to avoid dangerous situations) and tourism (support tourists' short-term decisions in their visiting schedule).
The presented concept mainly focuses on three areas: 1.) the high-performance integration of real-time data into AR environments; 2.) usability in the design in terms of displaying spatio-temporal developments and the interaction with the application; and 3.) design considerations accounting for the stress field between reality and virtuality, reduced visualisation complexity, and suitable information density.
Although our approach still shows some limitations (s. sub-section "Limitations and Challenges") including the need for continuous data transfer, lacking ability for offline operation, the trade-off between functionality and the complexity of an application, or insufficient positioning accuracy, we believe that it is a significant step towards the realisation of the vision of "situational awareness" -that is still computationally intensive and thus at best conditionally solved for mobile devicesthrough the use of AR technology.