Short Paper Augmented Chemical Reactions: 3d Interaction Methods for Chemistry Augmented Chemical Reactions: 3d Interaction Methods for Chemistry

—Supporting chemistry students in learning and researchers in developing and understanding new chemical molecules is a task that is not that easy. Computer applications try to support the users by visualizing chemical properties and spatial relations. Thus far, there mostly exist applications that are controlled by using ordinary input devices as mice and keyboards. But these input devices have one problem: they always try to map a lower degree of freedom to 6-dimensional movements for the location and the orientation of the virtual molecules. Augmented Chemical Reactions is an application that uses Augmented Reality to visualize and interact with the virtual molecules in a direct way. With the introduced 3D interaction methods, the work of students and researchers is tried to be simplified to concentrate on the actual task.


INTRODUCTION
The understanding of the geometry of chemical structures is a crucial part for understanding chemistry.This applies especially for chemistry students which cannot yet imagine the spatial structures of molecules.One reason for this may lie in the fact that teachers normally use two dimensional representations of molecules in their course materials as slides or on the blackboard and books.Understanding chemistry relies also on the understanding of the 3D structures of molecules and their parts.Certain behaviors only can be explained by knowing the spatial extent of the structures.Without this understanding chemistry students may have problems.To explain and understand why a special chemical reaction performs in a specific manner, the arrangements of the atoms in the molecules have to be known.For example, some atoms could be surrounded by others, blocking the binding process to another atom of another molecule because the repulsive forces of the surrounding atoms are stronger than the attracting forces of the surrounded binding partner.When the students know the special structure and the dynamicity of the molecules, they better know why a chemical reaction behaves in this special way.
There already exist several applications which visualize molecules in a 3dimensional manner.But those applications mostly have a difficult user interface especially when it comes to the manipulation of the rotation and positioning of the molecules while inspecting the molecules.They make use of the keyboard and the mouse to position and rotate the virtual molecules.As the movement of a computer mouse is restricted to two dimensions it has to be mapped to the 3-dimensional rotation and the 3dimensional placement of the virtual molecules.As this is not always intuitive, a lot of attention from the user goes into the mental mapping process.Augmented Chemical Reactions tries to circumvent this problem by introducing a direct manipulation user interface which is easy to use without the need to map a lower degree of freedom to the needed 6 dimensions of freedom.With the help of Augmented Reality it is made possible for the users to interact in the 3D space with tracked hand held devices to control the position and orientation of the virtual objects in an intuitive way (Figure 1).
In an evaluation of a German gymnasium [1] we have shown that a direct user interface for manipulating the position and orientation of virtual objects help to understand the special relations of molecules better than by using the conventional user interfaces with mice and keyboards.To achieve this, we built an application that visualizes the molecules which can be controlled by a direct manipulation user interface and also can show the dynamical behaviors of the molecules when they come close to each other.
As this user interface also needs to supply some kind of system control, gestures have been developed for selections and confirmations.

II. AUGMENTED CHEMICAL REACTIONS
Augmented Chemistry [2] and Augmented Chemical Reactions [3], [4] have been introduced to combine the visualization of complex structures and the advantages of direct manipulating physical objects.Both systems use Augmented Reality to visualize and a marker tracker to SHORT PAPER AUGMENTED CHEMICAL REACTIONS: 3D INTERACTION METHODS FOR CHEMISTRY deliver a direct manipulation 3D user interface to control the position and orientation of the virtual objects.
Augmented Reality is a technique that melts together reality with virtual information or objects in real time and interactively.This generates the impression as if the virtual objects are part of the real physical environment and can be interacted with.The application Augmented Chemical Reactions uses one or more markers or cubes with a black and white pattern on it which can be tracked by a marker tracker [5] similar to the AR-toolkit [6].This marker tracker captures images from a webcam and calculates the position and orientation of the patterns relative to the camera by using the size and deformation of the patterns in the camera image.
In a typical use case, the user sits in front of a table holding such marker cubes in his/her hand and looks at a monitor on top of the table.The webcam is mounted on the end of a microphone stand as close as possible to the users' eyes (Figure 2).By moving and rotation the markers in the field of view of the camera, the virtual molecules that are associated with the markers can be moved and rotated accordingly.
When controlling two molecules with two hand held markers at the same time the problem of performing system control tasks arises.The user can not trigger system commands while at the same time interacting with the molecules.As a possible solution gestures have been introduced to support the tasks of selecting and confirming bonds between the controlled molecules.
When there are several possible bonds that can be created between the two molecules (Figure 3), the user should be able to select the desired one out of the bunch of possible bonds.There have been developed and tested two methods to do the selection task.One method uses a shaking gesture to cycle through the available binding partners on the respective molecules, whereas the other method always selects the possible bond that forms the shortest bond.As both methods have their drawbacks, a combination of both methods was developed.The possible binding partners of each molecule were subdivided into several groups which could be activated by shaking the molecule.After selecting this subgroup of possible binding partners, the proximity method can be used to finally select the desired possible bond [7].
For confirming the selected possible bond a waiting method was developed that introduces the metaphor for gluing parts together by just holding the respective parts as still as possible [8].
Normal gestures are at the moment implemented with fixed rules and values for detecting the different gestures.This could become a problem when there are different kinds of users.For one group of users the values and thresholds could be too sensitive whereas for the other group of users this could be too insensitive at the same time.To come over this problem, we investigated on a gesture recognition algorithm that adapts to the movements of the users.When the system is too sensitive, the recognition algorithm would trigger too early.If this is the case, the user can aboard the triggering process.The algorithm recognizes this behavior and tunes the thresholds to be more insensitive.
When it is the other way round that the algorithm is too insensitive, and the gesture is not recognized, the system realizes this because the user tries to perform the nearly recognized gesture over and over without success and tunes the thresholds more sensitive.
This way the recognition algorithm tunes itself automatically and adapts to each user.
To be as flexible as possible, the application can associate any molecule that is stored in a Protein Database file (*.pdb) with the physical markers (Figure 4).Even more, the application supports various displaying methods as   SHORT PAPER AUGMENTED CHEMICAL REACTIONS: 3D INTERACTION METHODS FOR CHEMISTRY stereo on two screens, stereo head-mounted displays and ambient occlusion.It also can be used as a front-end to a molecular dynamics simulation application.
With this simulation running in the background, it is also possible to show the dynamic behavior of the molecules to the users while simultaneously interaction with the molecules.For this purpose the minimal dynamics simulation application Mindy [9] was used and reprogrammed to be a server application that can be controlled by Augmented Reality via network.To control such a molecular dynamics simulation application some redesign was necessary.Normally the simulation is started with the necessary molecule definition files and output in each computation step the new positions of the atoms in the molecules.As there normally exist two molecules in this application they would fly apart due to the repulsive forces.Augmented Chemical Reaction thus fixes one atom of each molecule to its designated marker.The position of these atoms is then also fixed in the simulation.To be possible to move around and rotate the molecules during the simulation the change of the real markers during one computational step have to be calculated and applied after the new positions arrive from the simulation to the respective molecules.The new positions of the molecules and the atoms in the molecules are then sent to the simulation application which itself continues to compute the new positions with this new positional data.
The Augmented Chemical Reactions application has a good responsiveness on an average PC.The system uses Windows® and a DirectX® 10 enabled graphics card.
In a user study with high-school students [1], there was mainly positive feedback about the application.The students really liked to interact with the system and would like to use it in class.
This application also won several awards for the best demo at the 2nd Experiment@ International Conference [10] as well as best student paper at the 5 th International Conference on Computer Supported Education [11] III. CONCLUSION The use of a direct manipulation 3D user interface brings the benefit of a better understanding of the spatial structures of chemical molecules.Thus it can help students as well as researchers to understand several chemical properties better.A steerable chemical dynamics simulation that was also integrated in the application further helps to understand the internals of chemical reactions better.
Several gestures had to be introduced and evaluated not to lose the benefits of a direct two handed interaction with the system.The most crucial parts of system control in this application are the selection and confirmation of possible bonds between the controlled virtual molecules.For the selection of a possible bond a combination of the shaking and the proximity based gestures was the preferred method.For the confirmation a waiting gesture turned out to be the most usable and precise method.We also developed a self-tuning gesture recognition algorithm to cover all the different behaviors of the users.

Figure 1 .
Figure 1.A hand held device is tracked by the Augmented Chemical Reactions application and visualizes a virtual molecule on top of it.

Figure 2 .
Figure 2. Computer setup for the Augmented Chemical Reactions application, using a webcam a physical cube on a handle and a monitor.

Figure 3 .
Figure 3. Between the two user-controlled virtual molecules exist multiple possible bonds.Those possible bonds where the two molecules could bind are visualized just for showing the amount of possible bonds as transparent cylinders.

Figure 4 .
Figure 4.The user can dynamically assign a molecule description file with a physical marker.