Applying Multimedia and Virtual Reality for Learning Environments

Most of the tools and languages for modeling Virtual Reality environments, such as VRML, X3D, Java3D, etc. do not provide means of describing the synchronized presentation of multimedia content inside these environments. Multimedia has demonstrated its capabilities of motivating users and capturing their attention, which are important characteristics when we want to provide a higher degree of immersion and learning capabilities inside Virtual Reality applications. This paper presents a robust and generic solution for the integrated presentation of different kinds of media objects inside virtual environments based on the Graphical Engine OGRE and how this solution can be applied broadly for providing customizable multimedia and virtual learning environments.


INTRODUCTION
Multimedia has been applied in different domains as a helpful tool for a fast comprehension about a subject being presented.Indeed, it has been proved that human is more receptive to new information and construct easier cognitive models if this information is presented in different modalities [1].The integration of multimedia content inside Virtual Environments (VEs) is a promising and interesting trend in the development of Virtual Reality (VR) applications.Multimedia captivates users´ attention inside the VE enhancing interaction, promoting user´s interest, facilitating learning and improving user´s immersion.Some important issues must be considered for the integration of multimedia content inside a VE such as the specification of the temporal and logical synchronization of different media objects (with at least one audio or video) to be rendered inside the 3D environment, and determining which events (e.g., user interactions) will be applied for the communication between the 2D/3D worlds.Unfortunately, most of the existing languages for describing 3D environments (such as VRML [2], X3D [3] or Java3D [4]) are monomedia and non-interactive since they support only the presentation of isolated media objects without any synchronization relations among them.One exception to this is MPEG-4 which by means of BIFS allows the creation of rich 2D/3D graphical scenarios with synchronized multimedia [5].However, the authoring of the MPEG-4 BIFS is still too complex and intuitive tools and approaches are still lacking.
Many VR systems have been proposed in the literature addressing different application domains: e-learning [6], [7], [8], collaboration among workgroups [9], augmented collaborative spaces [10], multimodal VR applications [11], among others.The rapid prototyping, modeling and authoring of VEs has been a major concern to many authors, as presented in [12], [13], and [14].Although, most of the systems propose the development of VEs, few of them explore the presentation of integrated multimedia content inside VEs [15].
This paper presents a solution to provide the integration of multimedia content inside a VE based on the Graphical Engine OGRE [16].The API implemented is called OGRE-Multimedia, and can be applied to any VR application to allow their customization with multimedia content.OGRE (Object-Oriented Graphics Rendering Engine) is a scene-oriented, flexible 3D engine written in C++ designed to make it easier and more intuitive for developers to produce applications using hardware-accelerated 3D graphics.When comparing OGRE with other existing languages and approaches for describing virtual worlds, we decided to adopt this platform based on its design quality, flexibility and clear documentation.
This paper is organized as follows: Section 2 presents a solution to customize a VE with multimedia presentations; Section 3 presents the main architecture of the API developed; Section 4 illustrates a Multimedia and Virtual Reality application, and; Finally, Section 5 presents some conclusions.

II. CUSTOMIZING MULTIMEDIA PRESENTATION INSIDE
A VIRTUAL ENVIRONMENT The main goal of our work is to propose and develop a solution for providing the presentation of multimedia content within an OGRE´s virtual environment (VE) for facilitating the design of learning environments.This multimedia content is related to: the output of any embedded multimedia player (such as RealPlayer [17], GRiNs [18], etc.); a Flash executable content [19], or; a web-browser content.The main idea is to present the multimedia content as textures over any 3D object inside a virtual environment.
When proposing the integration of multimedia content inside a VE, we had to come up with a customized solution to cope with the need for specifying synchronization relations among the media objects, supporting user interactions with these objects, ensuring interoperability of multimedia players, and mapping 2D objects into the 3D world.Unfortunately, the existing languages and models for describing multimedia presentation such as SMIL [20] do not support the description of three-dimensional channels, that is, the specification of the x, y and z coordinates for the presentation of the multimedia content inside the 3D environment.For this reason, the solution relied on the proposal of a simpler XML-based meta-language for de-scribing multimedia documents to be presented inside virtual environments, so-called the meta-multimedia document.
The meta-multimedia document was strongly inspired on the syntax of SMIL and can be applied as a multimedia authoring language where users can customize the virtual environment and describe what is going to be presented, where and when they will be presented.Briefly, it describes all the components of the multimedia presentation and their temporal and logical synchronization.The particularity about this presentation is that all the media objects (multimedia documents, flash, web-browsers, and primitive media objects such as video, image, text, audio, etc.) are synchronized and rendered anywhere inside the virtual environment.The interpretation and coordination of this document presentation inside the VE is done by the API developed for OGRE, the OGRE-Multimedia.
As presented in Figure 1, the structure of the metamultimedia document is composed of four main elements: panel, trigger, eventHandler and event.
Each panel element describes a presentation panel for media objects inside the virtual environment.The container called panels is a set of the panel objects that must be rendered inside a VE.
Each trigger element characterizes an object inside the VE which controls the activation and deactivation of a multimedia presentation.The container called triggers is a set of all the trigger objects that will be used to control the multimedia presentations inside a VE.
Each EventHandler element characterizes how the presentation of the media objects associated with a given trigger will be controlled (e.g., start their presentation when the user clicks on the trigger or when he approximates it).The container called eventHandlers is a set of all multimedia presentation described by all the elements eventHandler.
Each event element characterizes how and when the presentation of each media object of a given eventHandler will be carried out.
The structure of the meta-multimedia document was defined to make the process of authoring the multimedia document easier and intuitive.We consider the metamultimedia document as the key-solution for the integration of multimedia content inside VEs.Indeed, with this document, the author of the application is able to customize his virtual environment with new multimedia content without changing a single line of his code.

III. OGRE-MULTIMEDIA: INTEGRATING MULTIMEDIA
WITHIN VIRTUAL ENVIRONMENTS Most of the libraries available for creating Virtual Reality applications do not have appropriate APIs for the integration of multimedia content inside a VE.Some languages and platforms such as VRML, X3D, Java3D and OGRE, provide only APIs for the presentation of single media objects (such as video or audio) without integrating or synchronizing these objects.The solution proposed with OGRE-Multimedia is to provide an API to integrate the presentation of different multimedia objects (rendered by different plug-ins or APIs) around the definition of the meta-multimedia document.In this sense, APPLYING MULTIMEDIA AND VIRTUAL REALITY FOR LEARNING ENVIRONMENTS the meta-multimedia document describes all the synchronization relations among all the components (media objects, Flash presentation, web-browsers, etc.) of the document.Taking advantage of the OGRE´s component-based architecture, this API can be easily instantiated and integrated with the remaining available library.This section presents the main architecture of OGRE-Multimedia The architecture of OGRE-Multimedia describes the integration of the meta-multimedia document with the Virtual Environment, which are supported by the implemented software modules and some existing APIs.This architecture is depicted in Figure 2. The architecture of OGRE-Multimedia is composed of four main components: • External modules, which are represented by those APIs developed by other projects, or which were already provided by the OGRE´s library, such as (i) TinyXML [21] (XML syntactic analyzer parser), (ii) OIS [22] (Interactions management), ( 3) OgreAL [23] (Presentation of audio objects), ( 4) DevIL [24] (Presentation of images inside VEs), ( 5) Navi [25] (Presentation and interaction of a web browser inside the VE), and ( 6) OGRE graphic engine which is the main module of the system being responsible for creating, managing and updating the tri-dimensional model.• Elementary module, which describes the nonfunctional components of the architecture used as a support for the application (Meta-multimedia document and Virtual Environment).
• System startup, which is in charge to set up the presentation of multimedia content inside VEs.• System Update, which is in charge to control and update the multimedia presentation inside the VE according to possible user interactions.
The implemented prototype is further discussed on the next section.
IV. THE PROTOTYPE IMPLEMENTED This section illustrates the prototype implemented by the presentation of an OGRE-Multimedia application.This application applied a virtual world (also called map) for the presentation of multimedia content in an exhibition style.This environment can be applied for educational or art-exhibit purposes, for instance.OGRE-Multimedia was easily integrated to the OGRE application, where the methods of OGRE-Multimedia were invoked to enable the presentation of multimedia content previously defined on the meta-multimedia document, called "MMDocument.mmc".In the case of this VR application, the virtual world describes an open-wide area, where in front of each wall there is a sensitive column which can be activated by the user's click or by his proximity (depending on its previous configuration on the metamultimedia document) in order to trigger the multimedia presentation on the wall (Figure 3a).The multimedia presentation is launched after the user interacts (by clicking) with the sensitive column (Figure 3c).When the media objects start to be presented, their level of transparency is gradually changed producing the effect of fading-in.As we can see in Figure 3(d), a webbrowser is also presented as one of the multimedia textures.This browser is rendered by the API Navi which allows the user to navigate on the Web.Figures 3(d), 3(e) and 3(f) present an example of this navigation.
All the media objects presented inside the VE are synchronized and managed by OGRE-Multimedia which keeps pace of each presentation starting and interrupting all the media objects according to their previous configuration on the meta-multimedia document.OGRE-Multimedia enables the multimedia presentation inside the VE making of it a more realistic environment and, above all, keeping the user´s focus.

V. CONCLUSIONS
This paper presented the development of an API for the presentation of integrated multimedia content inside Virtual Environments based on the Graphical Engine OGRE, called OGRE-Multimedia.The integrated presentation of multimedia content inside VEs relied on the proposal of an XML-based representation to describe all the media objects to be presented and their synchronization relations, the meta-multimedia document.OGRE-Multimedia can be applied straightforward in different OGRE Virtual Reality applications since it is the result of an open-architecture where different APIs were applied in conjunction to provide the presentation of different kinds of media objects including the traditional images, audio, video, animations, etc., and also multimedia documents such as FLASH, and webbrowsers as well.Indeed, the combination of Multimedia and Virtual Reality can be successfully applied to the design of robust learning environments where students feel more comfortable and have their focus inside the VE, definitely improving their feeling of immersion and facilitating knowledge transfer.

Figure 1 .
Figure 1.Example of a meta-multimedia document

Figure 3 (
b) illustrates the interactive column in the virtual environment.This column (which is represented by a trigger element on the meta-multimedia document) is able to launch a presentation by a user´s click.