Virtual Slicer: Visualizer for Tomographic Medical Images Corresponding Handheld Device to Patient

This paper introduces an interface that helps understand the correspondence between the patient and medical images. Surgeons determine the extent of resection by using tomographic images such as MRI (Magnetic Resonance Imaging) data. However, understanding the relationship between the patient and tomographic images is difficult. This study aims to visualize the correspondence more intuitively. In this paper, we propose an interactive visualizer for medical images based on the relative position and orientation of the handheld device and the patient. We conducted an experiment to verify the performances of the proposed method and several other methods. In the experiment, the proposed method with a line laser showed the minimum error.


INTRODUCTION
Deciding the resection area is one of the important factors of a surgery. For example in cancer surgery, increasing the resection area is one way to remove all the cancer cells. However, if too many healthy cells are resected, the burden on the patient is increased. Therefore surgeons are required to grasp the 3-D extent of the lesion to determine the appropriate resection area. Fig.2 shows an illustration of an ideal resection.
The surgeons use tomographic medical images such as MRI (Magnetic Resonance Imaging), CT (Computed Tomography) and Ultrasound Imaging data to decide the resection area. These modalities can be classified into two types. One type is MRI and CT. They have a high sensitivity towards tumors and are able to draw detailed images of tissues. They are generally taken by clinical specialist at the preoperative examination date. Another modality is Ultrasound. Ultrasound images have less sensibility towards tumors than MRI and CT. However it can be taken by Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Laval Virtual VRIC ' 14 3 shows a surgery scene in an operation room. During surgery, doctors estimate the cancer area by using different images taken by different modalities. Understanding the relationship between the patient and the tomographic images is difficult. It requires the surgeons to have the ability to identify tumors and organs from the medical images and be able to estimate the 3-D position from the 2-D images. Under present circumstances, the solution relies solely on the skill of experienced doctors. In response to this situation, there is a lot of research and development of medical support application regarding visualization of tissues and lesions.
In this paper, we propose an interactive visualizer which displays medical images based on the relative position and orientation of handheld device and the patient. Furthermore, we evaluated this proposed method by user experiment.

Related Works
There are two major approaches in presenting medical images. One approach is reconstructing a 3-D model from the medical images. Tamaki et al [1] have developed a system that reconstructs the 3-D model of breast cancer by using an ultrasound probe. The 3-D model is then superimposed on the video image. Since the ultrasound can capture medical images in real-time, the system is very robust against the deformation of the breasts. However, the system requires large equipment in the operating room. Another drawback of this system is that it cannot be used for ductal carcinoma in situ (DCIS) or intraductal cartinoma that can not be detected by the ultrasonic diagnostic equipment.
Thomas Wendler et al [2] and Asli Okur et al [3] have also developed a similar approach. Both of these studies superimpose on the image and present the image on a display that is often situated away from the operative field. The surgeon is required to look alternately between the display and the surgical field. A number of studies have been made using this method. [4][5] [6] Using these methods can cause the interruption of surgical flow. This hand-eye coordination problem has been discussed by Breedveld et al [7]. Kondo et al [8] proposed a system projecting a 3-D organ model to a known phantom shape. These systems use Spatial Augmented Reality (SAR) with a projector. Since the recontructed 3-D model is directly projected to the phantom, it is easier to understand where the model is located in the real environment. This study was only conducted with phantom models, not with actual patients.
Volonte et al [9] have developed an intraoperative system that aides surgery conducted with a robot arm. The system projects 3-D model which is reconstructed from CT images to the patient. Tissues that has little movement such as the lower costal margin, liac crest, and navel are used as corresponding points to register the CT images with the patient.
Nicolau et al [10] has developed a system which helps the percutaneous puncture of liver cancer surgery by projecting CT images to the patient body.
In these systems, the 3-D model is projected to the surgical field directly. However, there are some projection misalignment caused by the accuracy of the projector. Furthermore, the 3-D location recognition becomes difficult because the 3-D model that is projected to the real object can be perceived as being on the skin surface. H. Liao et al have developed an augmented reality system that superimposes the 3-D image using a half mirror. [11] Stereoscopic vision can be achieved by binocular disparity using a Integral Videography technique. Provided that the system requires the special use of medical equipment such as OpenMRI.
The second major approach is presenting the tomographic medical image such as MRI and CT images without reconstructing the 3-D model. This approach avoids some system errors that happens during the reconstruction or registration of the 3-D model. Since the doctors use the tomographic images on a daily basis, this approach can be used without difficulty because the doctors are already familiar with the analysis of the tomogaraphic images. Furthermore, detailed information contained in the tomographic image is not lost in this approach. Eric Soehngen et al [12] input the medical images to the handheld device and place it near the operative field. Maki Sugimoto et al has developed Osirix[13], a visualization system that works on the Mac OS and iOS. They also placed it near the surgical field.
These systems have the possibility to solve the hand-eye coordination problem. However, they do not interact with the actual environment. L. Schwarz et al [14] have developed a system that recognizes the gestures of the doctors by using the depth camera called Kinect. Images that correspond to the gestures are displayed. Ma Meng et al. have developed a system that displays the tomographic image that matches the position of the user's hand that is detected by the Kinect [15]. This allows the users to manipulate the display hands-free．

Interactive Visualizer for Tomographic Medical Images 3.1 Mental Rotation
Mental Rotation is the ability to rotate mental representations of 2-D and 3-D objects. When the surgeon manipulates the ultrasound probe, which is situated away from the display like many inspection equipment; the surgeon needs to rotate the displayed image mentally to correspond with the actual patient. In our proposed method, the rotation of the image corresponds to the movement of the display, so that the surgeon can understand the images more intuitively.

Proposed Method
Our goal is to develop a visualizer for medical images that helps understand the relationship between patient and images. Fig.1 shows the fundamental concept. Also, Fig.4 shows the design concept of our system. The system displays the tomographic image which corresponds to the position where the surgeons specifies in the actual environment. A line laser that is attached to the handheld device is used to specify where the surgeon wants to see. The display shows the image of the position where the laser is lit.
The handheld device is operated by the surgeon. The handheld device and patient are tracked. Tomographic image is varied by the section. Our system displays the image which corresponds to the position and orientation of the handheld device and the patient.  Fig.6 shows the procedure a surgeon takes to use our system. At first, the surgeon attaches the marker to the patient as a reference marker. Tomographic images such as MRI and CT images are taken. At this point, the posture of the patient should be the same as the posture during surgery. For example, it is common to take tomographic images of the patient in the prone position for the breast cancer surgery. We have additionally taken images with supine position. We used a support instrument to fix the posture uniquely. The surgeon registers these images to the handheld device. Then the surgeon registers the positions of the markers in the images manually. Before the surgery, the 3-D position tracking device is attached around the operating table. The handheld device is then manipulated by the surgeon. Fig.7 shows the flow of the process of our system. At first, tomographic images are inputted. Then voxel data is made from these images. Next, the system acquires the position and the orientation data of the reference marker and the handheld device. Then coordinate transformation is made to the image coordinates from the display coordinates. If there are some image data that exists in the drawing frame, the image is displayed.

Calibration
In this system, the registration between the images and the patient is calculated by using the transformation matrix of the world coordinates from the image coordinates.
The calculation requires more than four corresponding points. In this paper, nine markers were attached. Since markers that are placed on the body surface is on the same plane, the transformation matrix is not unique. By calculating the cross vector of one marker to the other, we were able to acquire the normal vector of the markers. By doing so we obtained the normal direction to obtain a new corresponding point.

Position of Reference Marker
The placement of the markers is determined by body structure.
First, the surgeon attaches three markers in locations with minimal deformation. One marker is placed on manubrium as the top marker, another one is placed on xiphisternum as the bottom marker and the last marker is placed in the center of the top and bottom markers. (red markers in the Fig.8) Next, the surgeon places another marker above the nipple, that is perpendicular to the sternum from the center marker. Using the marker above the nipple and center marker of the sternum, another marker is placed that makes an isosceles triangle. (green markers in the Fig.8) Finally, the surgeon attaches four markers make a square with sides of 4.5 cm. (blue markers in the Fig.8)

Line Laser
We attach a line laser to the handheld device to make it easy to understand the displaying position of a tomographic image. The laser is mounted so that the laser light becomes parallel to the handheld device visually. Fig.9 shows a line laser module attached with a handheld device. The angle of the laser mount can be adjusted to direct the light under the device. A transformation matrix of the display coordinates from the device coordinates →ℎ is obtained by using the relationship between optical marker which is attached to the handheld device and the display of the device. The value of screen resolution was taken in consideration when this was calculated. In this paper, the screen resolution of the handheld device (iPad2) is 264dpi. A transformation matrix of the image coordinates from reference marker coordinates → is obtained by using relative positions between the optical marker, display, and pixel spacing of the display. Finally, a transformation matrix of the image coordinates from display coordinates → is obtained by using these matrices.

Voxel Data
At first, the number of pixels of the columns and rows, the pixel spacing, and the slice thickness are read from DICOM data. The luminance values of the coordinates in between the tomographic images can be determined by calculated using ratio distance of the pixel values of the images in between. Then the luminance values of tomographic images are registered to a voxel coordinate by using these values. Fig.11 shows an example of completion of luminance values.

View Mode
Our system has two modes of visualization. The first mode is the overview mode. This mode displays an overlooking view of the images. The second one is the actual size mode. In this mode, the actual size of the tomographic images is displayed.

Implementation 4.1 Tracking Tool
In this implementation, we used Natural Point Optitrack V100:R2 for 3-D position tracking. The spec of IR camera resolution is 640 x 480 pixels. We attached 10 cameras to the ceiling of an experimental environment. A frame rate is 100fps and latency is 10ms. It is possible to make a rigid body from more than three optical markers.

Handheld Device
We used iPad2 MC981J/A as a handheld device. The spec of IPS display resolution was 1024 x 768 pixels and the pixel density was 264 ppi. The CPU used was Apple A5 (1 GHz). The version of operating system was iOS 6.0.1 (10A523).
An attached line laser was LE650-5-3-F-S (22x100) 90. The laser module can adjust the focus. The laser class was 1. The wavelength was 650nm. The optical poser was 5 mW and the fan angle was 90 degrees. The divergence was 0.6 mrad.

Libraries
We used Virtual Reality Peripheral Network (VRPN) 07.30 library to connect with the tracking tool. This library provides position and orientation information of rigid bodies to the handheld device.
DICOM Toolkit (DCMTK) library 3.6.0 was used to read DICOM image files. DICOM is short for The Digital Imaging and Communications in Medicine. It is a standard for distributing and viewing any kind of medical images.

Experiment 5.1 Overview and Procedure
We used a MRI data of a patient without tumor tissue. A sphere was placed virtually to portray the tissue the subjects had to find.
Subjects had to point to the location indicated by using four different types of displaying methods: DICOM Viewer, Osirix, Virtual Slicer (Proposed Method) and Virtual Slicer with Line laser (Proposed Method).
Subjects were asked to answer the center position of the sphere with the probe (Fig.13). We evaluated the error distance between the corresponding position and center of the sphere.

Experimental Conditions
• The number of subjects was 16 people (15 men and 1 woman).

•
The number of attempts was 16 times.

•
A sphere with a diameter of 20mm was placed

Pointing Accuracy
To evaluate the pointing accuracy in this experimental environment, we had a prior experiment. A marker is placed at the position of the 0 mm scale of the ruler. And a subject pointed at 10 mm intervals at the tip of the probe. The number of attempts was 10 times at each distance. The average error was -0.20mm and the average standard deviation was 1.68 mm.

Phantom
For this experiment, a phantom of the patient was made by a 3-D printer: Stratasys FORTUS. We captured a 3-D model of a patient by using a 3-D scanner after taking the MRI. By combining the MRI and 3-D scanning data sets, the 3-D geometry of the patient was reconstructed. During the taking of MRI and 3-D scanning, the reference markers mentioned in Section 3.5 were attached to the patient. In the printing process of the phantom, the markers were replaced by recto-reflective markers for the registration process.

DICOM Viewer
This is a conventional viewer that is used in medical field. It shows the tomographic image, the value of slice thickness and the image size. It was displayed on the handheld device which is situated away from the phantom.

Osirix
This viewer displays voluntary tomographic images by using Osirix (open source software) [16]. It can change the cutting plane dynamically. It was displayed on a PC which is positioned away from the phantom.

Virtual Slicer (Proposed Method)
This viewer uses proposed methods without a line laser pointer.

Virtual Slicer with Line Laser (Proposed Method)
This viewer uses proposed methods with a line laser pointer.

Tomography Images
We made the tomographic images for the experiment. One sphere with a diameter of 20 mm is placed. We made four types of data by changing the position of the sphere. This experimental design refers to the verification experiment of position recognition made by Hongen Liao, et al [10]. Significant differences between these four methods were found by the analysis of variance (ANOVA) at the 1% level of significance.

Discussion
The distance error of our proposed method (Virtual Slicer with Line Laser) was the smallest of the four methods. It is possible to consider the proposed method reduced the error of position indication compared with DICOM Viewer and Osirix.
The dispersion of the four methods showed an unequal variance. The dispersions of DICOM Viewer and Osirix were increased compared with the proposed methods.
In this experiment, the subjects were students of science and engineering. Therefore most of the subjects did not have expertise in the medical field. The subjects had lecture on image interpretation and the position of nipple and markers in the images, which can be used for the landmark. Osirix displays three different tomographic images from different viewpoints on a window at the same time. It is assumed that there was confusion due to seeing three different images simultaneously. It could lead to erroneous position recognition and the larger average error distance. We did not find significant difference between the two configurations of the proposed method. However, the average error of the proposed method without the laser was 4 mm larger than the one with laser. The proposed method displays an image directly under the device. Due to thickness of the device itself (20 mm), there might be some obscure in understanding the actual location of the tomographic images.

Limitations & Future Works
In the breast cancer surgery procedure of the breast oncology department, the surgeons try to cut within a 5 mm margin from the edge of tumor. To improve the accuracy of recognition is still one of our future works. Associate with tomographic images, surgeons perform a surgery by using various clues such as stiffness of the tissue and pre-injected dyes. Those clues contribute to make better position recognition.
In the experiment, the subjects were students of science and engineering. Since most of the subjects did not have expertise in the medical field, they were not familiar with tomographic images. We plan to continue our user experiment with medical students and residents.
Our system does not take into account the deformation of the model that can be caused by respiration or pose changes of patients. A combination of a tracking method that takes deformation into consideration with our proposed method will be able to be applied for such deformation of soft bodies.

Conclusion
This paper proposed Virtual Slicer: an interactive visualizer for tomographic medical images based on the relationship of handheld device and reference markers on a patient. We conducted an experiment to verify our proposed method. As the result of the experiment, the proposed method with the line laser showed the minimum error value.