MIXED REALITY SYSTEMS

Currently one of the most challenging aspects of human computer interaction design is the integration of physical and digital worlds in a single environment. This fusion involves the development of "Mixed Reality Systems”, including various technologies from the domains of augmented and virtual reality. In my talk I will present related concepts and discuss lessons learned from our own research and prototype development. Our recent work involves the use of mixed reality (as opposed to ‘pure’ virtual reality) techniques to support seamless collaborative work between remote and hands-on laboratories.


INTRODUCTION
Mixed reality comprises concepts and technologies to produce innovative user interfaces where physical and virtual objects co-exist in the same computer-based environment.Milgram and Kishino (1994) defined a Mixed Reality as being "... anywhere between the extrema of the virtuality continuum", where this continuum extends from the completely real to the completely virtual environment with Augmented Reality (AR) and Augmented Virtuality (AV) in between (Fig. 1).
Fig. 1 Virtuality continuum (Milgram & Kishino 1994) The Mixed Reality approach brings the virtual world of computers into the physical world of ordinary job and then to enhance them with the functionality of virtual objects: they can make use of computer generated data or simulation models while continuing their tasks, instead of constantly returning to a stationary computer or keyboard.Moreover, mixed reality systems support social presence in collaborative remote environments, as it can enable co-located and distributed users to interact in distributed virtual spaces while viewing or even manipulating real world objects at the same time.This includes aspects of natural communication that serve as mediators for mutual understanding: physical object handling, eye contact, and facial play.3D modeling, motion tracking, haptics, simulation, rendering and display techniques are core elements of Mixed Reality applications.Early developments were mostly based on blending computer generated virtual worlds or simulations with real-life video.Later on, interfaces where developed, which sense and generate real-life data being exchanged between virtual objects and their physical counterparts (Ohta & Tamura 1999).

INTERFACING REAL AND VIRTUAL OBJECTS
Based on a series of case studies we have developed an interface for mixed reality, supporting a unified view on the interaction between real and virtual systems.The general idea bases on Bond graph modeling (Paynter 1961), which describes dynamic system links between various sub-systems as energy-(or power-) connections.Power is a product of effort (e) and flow (f).
Effort (e) stands for force, pressure, voltage, etc., and flow f stands for velocity, flow of a fluid, current, etc.If an appropriate interface is provided to sense effort and flow at one side of the intersection and generate effort and flow at the other side and vice versa, two subsystems (real or virtual) can be connected in a way, which preserves the behavior of a comparable connected real system.Fig. 2 describes it in general: Effort and flow are sensed (or generated) providing voltage and current (or effort and flow in the opposite), an analog-digital converter provides digital information for the software (or an digital-analog converter converts digital information in analog signals to drive a generating mechanism for effort and flow to the real world).This interface generates or dissipates energy (or power).The power, provided through the real system, has to be dissipated, because the virtual continuation with software needs nearly negligible power.In the opposite direction, the digital information provided through the software, has to generate the power necessary to connect the real system.Power (effort × flow) can be bi-directionally transferred from a real energy level into a virtual signal/information-level and vice versa.Real systems, virtual systems and links can be described by bond graphs and analyzed according to classical control theory (Bruns 2005).The concept allows the splitting of dynamic systems into "pieces" as long as effort and flow are preserved at either side of the intersection to maintain a dynamic equilibrium.

CONCLUSIONS
Presently some research effort is being done to foster collaborative engineering between remote sites (Müller & Erbe 2005).One of the problems is to seamlessly connect the real to the virtual world.The environment envisaged should support users to work with real and virtual, isolated or embedded systems, local or distributed components, thus allowing a continuous shift between various degrees of abstraction and various levels of distributed collaboration.At the moment this seems to be a quite visionary approach.But future laboratories will benefit from further developments in computer simulation technology, mobile computing, sensor/actuator devices, integrated equipment supporting hardware-in theloop functionality, and enabling connections to be made between real-life phenomena and their virtual representation or continuation.

Fig 2 .
Fig 2. Interfacing real and virtual sub-systems