An Interactive Mixed Reality Ray Tracing Rendering Mobile Application of Medical Data in Minimally Invasive Surgeries

—Visualization of patient’s anatomy is the most important pre-op-eration process in surgeries; minimally invasive surgeries are among these types of medical operations that counts totally on medical visualization before operating on a patient. However, medicine has a problem in visualizing patients’ through looking through multiple slices of scans, trying to understand the three-dimensional (3D) anatomical structure of patients. With Mixed Reality (MR) the developments in medicine visualization will become much easier and creates a better environment for surgeries. This will help reduce the excessive effort and time spent by surgeons to locate where the problem lies with patients without looking through multiple of two-dimensional (2D) slices, but to see patients’ bodies in 3D in front of them augmented in their reality, and to interact with it what-ever pleases them. Moreover, this will reduce the number of scans that doctors will ask their patient’s for, which will result in less harmful x-ray dosages for both the patient and the radiologist. Biomedical development in medical visualization is an active research topic as it provides the physicians with required devices for clinically feasible way for diagnosis, follow-up and take decisions in different disease life line. Current clinical imaging facility can provide a 3D imaging that can be used to guide different interventional procedures. The main challenge is how to map the information presented in the digital image with the real object. This is commonly implemented by mental processing that requires skills from the medical doctor. This paper contributes to this problem by providing a mixed reality system to merge the digital image of the patient anatomy with the patient visual image. Anatomical image obtained from Computed Tomography (CT) or Magnetic Resonance Imaging (MRI) is mapped over the patient body using virtual reality (VR) head-mounted device (HMD).


Introduction
The current growth in medicine and technology should proceed at the same level.Furthermore, medicine should take advantage in the speedily development in technology.One of these significantly important subjects of medical applications using new technologies are the visualization of human anatomy [1,2].Interventional radiology procedures using imaging guidance such as CT/MRI does not meet complete surgeon's satisfaction.In current procedures, radiologists must scan the patient from different positions.Thereafter, surgeons and radiologists must investigate the scanned images to better locate the problem.Consequently, doctors and patients are exposed to heavy radiation.However, some types of medical imaging systems provide a series of scans that can be viewed as a 3D model using appropriate software that can guide interventional clinical procedures.
Augmented reality (AR) was used to provide a solution to this medical problem for a long time [3].The core challenge in these applications is to offer real-time accurate representation of human organs that is enough for surgeon to proceed with clinical procedure [4].The developed system should consider the data acquisition device (e.g.video camera, human eye, etc.), the data registration that maps the digital data to be mapped with the real visualized object, and finally, the motion handling and calibration.In this research, we consider the following scenario.A patient is undergoing a minimally invasive surgery where the surgeon needs to process some procedure like injunction of medication or biopsy.We also assume that a CT or MRI image of the patient is available.However, it is difficult for the surgeon to map the 3D anatomical image to the real patient.This happens frequently when the surgeon lack experience in similar procedures.It requires mental process to imaging how the 2D slices presented on the screen are represented on the patient in surgery room.The target of the developed system is to map the anatomical image over the patient real body for easy to comfortable process.Our developed system should be a one-step forward to solve the problem of visualizing human bodies.
Volume rendering of 3D image data of patient's multiple slices is the revolution in imaging human body [5].A voxel is the 3D equivalent to a pixel and are the smallest element in a 3D object [6].Voxels are used to build 3D objects, mostly used in computer graphical applications like computer games, but also used to render a volume.Applications on volume rendering have taken a large part in interventional and minimally invasive surgeries over the past couple of years.Before the volume rendering concept is used, there were other techniques that concentrate on visualization via surface shading.It transforms the volumetric data into geometric primitives then screen the pixels.It is good in representing the object but not the best for visualization.
When it comes to volume rendering, the technique displays the information inside the object volume, it is a direct display, the technique transforms volumetric data to screen pixels directly, and it uses transparency to see through volumes.The presented study aims at the development of MR software that will be used in minimally invasive surgeries and interval procedures.Several Groups and associations formed of researchers and scientists have also work in this same field of technology, but none of which went on developing such software in MR, neglecting its vast importance in medicine, some of these works are mentioned in Section 3.3.Our goal is to reduce the heavy load of scans visualization, as well as saving time and effort.This procedure will be much cheaper than the previous methods.Our system requires only a smartphone and a MR ready headset.Mixed Reality is the combination of Virtual Reality (VR) and AR [7].This combination brings together the real world and the digital one in one reality [8,9].It allows the users to interact with both physical and virtual items, making it more practical than previous methods.

2
Materials and Methods

System overview
The introduced system visualizes medical images (CT, MRI) as a 3D object.First, we use the developed software to visualize the medical images by using volume rendering ray-casting technique.The term volume rendering is used to describe techniques which allow the visualization of 3D data.Volume rendering is a technique for visualizing sampled functions of three spatial dimensions by computing 2D projections of a colored semitransparent volume.The technique works as follows: Step 1: • Trace from each pixel a ray into object space.
• Compute and accumulate color/opacity value along the ray in the process ofpixel compositing.
• Assign the obtained value to the pixel.
Figures 1 and 2 illustrates the process of Ray-marching and compositing of pixels.[11,12], where c is the color of pixel, and ∝ (alpha) refers to the opacity.
Step 2: In this step, we use compositing technique named alpha blending, i.e. the iterative computation of discretized volume integral.Figure 2 illustrates how alpha blending works while each ray goes through the object on its direction.
The developed software runs on Samsung Gear VR headsets and using its passthrough camera feature.It will enable the software to augment the 3D object of the medical scans in real world space.Interaction with the augmented object will be performed using the Gear VR controller.Users could manipulate the viewed 3D object generated from the medical images sliced by hiding parts of the object or view it in different ways with some GUI features to help the user to interact with it more easily, such as: • Increasing visibility • Increasing and Decreasing Opacity • Clipping (removing parts of the object) on the X, Y and Z axes.

• Rotation and Translation
Briefly, the considered scenario may be summarized as follows: • First, obtain volumetric medical data Digital Imaging and Communications in Medicine (DICOM) [13], or RAW file format.• Preprocess the data to the best possible lossless form of useable data.
• The data are stored afterwards on a smartphone then mount it on a VR headset that has a pass-through camera feature.
• The software will render the preprocessed data as a 3D object into reality using AR technology through the virtual reality headset.• User will interact with the 3D object via Gear VR controller.

Paper-An Interactive Mixed Reality Ray Tracing Rendering Mobile Application of Medical Data…
Using this scenario, surgeons and radiologists will be able to see the scanned slices of the patient as a real 3D object in front of them and will be able to interact with it through a controller.They have the capability to zoom in or out or even to make parts of the object transparent as well as clipping parts of it.

Image acquisition
CT or MRI scanners first scan the patient.Afterwards, measured data is sent to an online archive to be stored and registered.Thereafter, the data has to be sent to the smartphone via wireless communication for processing and visualization.In this study, we have used CT data provided from Suez Canal University Hospital with blind patient information.An example of the image slices is shown in Fig. 3.Moreover, we have used some online free available CT data to confirm the validity of the proposed method using different resources.

3
Results and Discussion

Samples and results
In this section, we demonstrate results obtained from experimental study using the developed system.In Fig. 3, a sample of abdominal CT slices obtained from single image from the CT data used in this experiment.These images demonstrate the anatomical structural of human internal organs.The proposed method is implemented using volume image shown in Fig. 3 and the surface of the patient's body as a 3D object viewed from three different angles is shown in Fig. 4. The volume rendering displays the 3D object with focus on the surface only.It is not possible to view internal structures with this visualization setup.The internal structures can be viewed with three different angles after decreasing the opacity value as shown in Fig. 5. Several organs can be viewed with higher quality.Spinal cord, liver and kidneys can be viewed accurately in 3D structures.Figure 6 is showing only half of the rendered object viewed from three different angles using the first dataset.The same experiment is repeated for the second dataset and results are shown in Figs.7-10.

Discussion
The proposed software will help in minimizing the visualization of medical images, saving time and effort for surgeons and radiologist, with relevantly fast run time.It requires few minutes to render a data-set of 300 images.This method has a potential to be adapted in several minimally invasive surgeries where the surgeon is required to view internal structures mapped with the patient body in real-time.Results indicate that using the proposed method can help in the rendering and 3D visualization of CT volumes in very short time that lead to exact recognition of different large size objects.However, it is still challenging to observe small size objects such as blood vessels and veins.Further development is required to improve the accuracy towards a better visualization of objects with size less than 10 mm.

Related work
In 2010, a group of researchers from university of München, Germany, started a project that maps the CT scans obtained with patient body.Their AR system of optical tracking and video see-through head HMD for visualization was developed to keep track of the objects in the scene.This process is carried out by two separate optical tracking systems.Four infrared ARTtrack2 cameras have been mounted to the room's ceiling to obtain an outside-in optical tracking system, while an infrared (IR) camera mounted directly on the HMD is used as an inside-out optical tracking system.They have used video image as context layer, while is rendering focus layer with volume rendering.Occlusion handling is shown for instruments and hands" [14].Images of their work are shown in figure 11.Similarly, in 2018, a recent research carried out by group of researchers and scientists [15], they have developed a VR imaging technique that displays and interacts with optical coherence tomography (OCT) data.Their application was installed on a highend notebook (Windows 10 home, 64bit, NVIDIA GeForce GTX 980 8192MB GDDR5, 32 RAM, CPU Intel Core i7-6700K CPU @ 4.00 GHZ, 4 cores).As in most VR applications, they used in their development phase headsets that connects to a powerful personal computer (PC).As a result, the used hardware delivers very high frame rates, when rendering high quality OCT data.They have reported that their application reaches a normal of 180 frames per seconds (fps) while rendering high quality volumetric data.They have used HTC Vive (VR headset) to render the data in a virtual reality environment [15].Pictures of their work are shown in figures 12 and 13.Their development approach is to render original point-cloud data rather than polygons or meshes, which enhance the detail level and preserves complexity rather than reducing it.
The relation between their work and our work maybe be summarized as followed: is both works tend to image medical data in virtual environments.However, there is no point of comparison between both research works, since both projects use different types of hardware.Our work was tested on a smartphone and the research of [15] was tested on a high-end PC.
Nevertheless, our work has more potential in future medical applications, with more interaction with the real world since we have implemented the volume rendering technology in MR rather than VR.Conclusion and Future Work

Conclusion
In this study, we discussed the developed system and software that will be used as a new method in visualization of medical images.The software can deliver better visualizations to surgeons and radiologists, helping to create a better environment for surgeries.

Future work
The research presented in this study also provide a strong basis for future work in awareness and in volume rendering technologies.One area of future work is in uniting the knowledge gained about mixed reality with knowledge about medicine.Another extent is in applying the results studied here to the many real-world situations in which reconstruction of 3D medical data is an important problem.

Fig. 2 .
Fig. 2.Compositing of pixels' color/opacity along the ray[11,12], where c is the color of pixel, and ∝ (alpha) refers to the opacity.

Fig. 4 .
Fig. 4. (left) A 3D object rendered with ray-marching by using the first dataset, full opacity, no clipping, front facing, (center) rotated 90 degrees on the Y-axis and (right) rotated 90 degrees on X-axis.

Fig. 11 .
Fig. 11."(a) Illustration of the occlusion problem.(b, c, d) Render pipeline for correct occlusion handling, (b) video texture, (c) hit texture for the skin, (d) final composition of (b) and (c).(e) like (d) with in-body MPR.(f) Focus and Context rendering with shaded volume rendering for the focus layer (bone), virtual mirror and instrument."[14]

Fig. 12 .
Fig. 12. "Stereoscopic illustration of the VR environment displaying volume OCT data of a peripheral retinal tear" [15]