THE OPTICAL AND VESTA PROJECTS: OPHTHALMIC VISUALIZATION AND VIRTUAL SURGERY

In this paper we describe two projects. The first is our novel and superior method of calculating the shape of the cornea of the eye from measuring instuments, called videokeratographs, that typically shine a pattern (usually rings) of light onto the cornea and then capture the reflection pattern on a CCD array using a built-in video camera. Accurate and detailed information about the shape of the cornea is critical for the diagnosis and correction of vision problems. The second project is a virtual environment simulation of the surgical procedure of minimallyinvasive gallbladder removal, which is known as laparoscopic cholecystectomy. Our simulation is designed to enable us to easily experiment with different approaches to addressing the major issues of surgical simulation, and it represents our attempt to leverage a great deal of surgical experience in creating a simulation that focuses on the most important, most error-prone steps in the procedure. Currently we are expanding the simulation code to serve as a general-purpose surgical simulation authoring tool, experimenting with new tissue modeling techniques, and constructing a new haptic interface device. 1. The OPTICAL Project: OPtics and Topography Involving the Corneal and Lens The cornea is a clear fibrous tissue that covers the front of the eye. The cornea extends outward slightly from the vaguely spherical shape of the eye. This is the entry point of light into the eye and also the main refractive surface. There is a thin layer of tears which covers the front surface of the cornea; this prevents the cornea from drying and protects it from contaminants in the air. 1 This paper originated as a conference presentation in China, and was originally approved by the guest editor, Prof. Zhigeng Pan, to be included in the special IJVR issue devoted to VR in China (IJVR4-4). However, IPI Press did not include the paper in that issue because it was submitted in postscript (.ps) format, which we were not equipped to handle, as well as lacking an abstract, references, a biography, and author contact information. However, Prof. Pan prevailed upon us to include the paper in this later issue anyway, so we extracted the text from the file, and Editor-in-Chief Richard Blade wrote the abstract, as well as got biographical and contact information from the web. Though the content of the paper is clearly interesting and worthwhile, references and some images would have made it much more useful to our readers, and we apologize for the lack of those. 2 See website: http://www.cs.berkeley.edu/optical/ Vol.5, No. 1 (2001) THE INTERNATIONAL JOURNAL OF VIRTUAL REALITY 2 Accurate and detailed information about the shape of the cornea is critical for the diagnosis and correction of vision problems. The cornea is the main refractive element in the human visual system, performing 3/4 of the refraction, or bending, of light in the eye. Because of this primary role in the human visual system, the quality of vision is determined by the shape of the cornea, and even minute subtle shape variations, at the micron level, can have significant effects on visual performance. Eye care practitioners need to know the shape of a patient’s cornea to fit contact lenses, to diagnose keratoconus, an eye condition where the cornea has an irregular shape with a local protrusion, or "cone", which has dramatic effects on vision, and to plan and evaluate the results of various types of surgery that improve vision by altering the cornea, referred to as corneal refractive surgery. Recent laser surgical techniques are one of the driving forces behind the enthusiasm for the development of techniques to precisely measure the shape of patients’ corneas and report those results with an intuitive and natural display. This information is needed to screen out corneal conditions that are contraindicated for this surgery, to do surgical planning, and to evaluate the results of such surgery. When performing surgery such as radial keratotomy and excimer laser ablation, it is essential that corneal surgeons fully understand the topography of the patient’s cornea. Improved scientific visualization techniques can greatly aid in surgical planning and evaluation. Although this underscores the importance of having such accurate shape information, ironically the commercially available instruments to measure corneal shape lack the sophisticated mathematics and computer algorithms to achieve this goal, and often display misleading and erroneous results. Recently, instruments to measure corneal topography have become commercially available. These devices, called videokeratographs, typically shine a pattern (usually rings) of light onto the cornea and then capture the reflection pattern on a CCD array using a built-in video camera. Given such an image of distorted rings, it is necessary to develop a reconstruction algorithm to recover the corneal shape. Algorithms for the reconstruction of corneal shape as well as for scientific visualization techniques to depict corneal shape suffer from the approach of thinking in terms of one-dimensional quantities along radial directions measured relative to an assumed center. These directions are called meridians and are analogous to lines of longitude on the globe. The current corneal maps displayed on all commercially available instruments are really one-dimensional, defining and ignoring shape information along any other direction. We have developed a novel algorithm the reconstruction of corneal shape. The construction begins by guessing a possible surface shape. Then the algorithm measures the difference between the guessed surface and the real cornea. First we identify features in the image and their corresponding points in the source pattern. If we assume that the lens system can be modeled by a pinhole or nodal point, then we can simulate the process that formed the image by using backward ray tracing. A ray from an image feature is traced through the nodal point to the surface. If the guessed surface has the correct local shape, the ray will intersect the source pattern at the corresponding feature. More commonly, the surface is incorrect, so the ray misses the feature. The aim is to change the surface so that the ray intersects the correct location. However, we must change the shape of the surface globally, otherwise rays from other features will still miss their corresponding features. Vol.5, No. 1 (2001) THE INTERNATIONAL JOURNAL OF VIRTUAL REALITY 3 The appropriate global change is computed using constrained optimization. From the traced rays we formulate an error function that measures the difference between the guessed surface and the true cornea. The surface that minimizes this error function has a similar shape to the true cornea. In order to make the problem more easily solved, we constrain the surface to interpolate one or more points. These constraints and the error function define a stand and constrained minimization problem. We solve this problem iteratively, by taking an initial guess, and stepping toward the solution. We have carefully formulated the error function and chosen a surface representation so that each step can be performed efficiently. Each iteration requires tracing rays, computing a set of normals, and fitting a new surface (the surface changes shape to indicate the fitting is taking place) to the normals. One of the challenges of corneal topography is to convey the shape of the cornea to clinicians. The difficulty lies in the scale of the deviations that need to be depicted. Elevation changes at the micron level are significant, yet the cornea is on the order of a centimeter in diameter; this can be four orders of magnitude different! Hence, a straightforward display of the corneal surface will yield a smooth surface appearance for all corneas and will not reveal the subtle shape that we wish to depict. Consequently, corneal shape is usually illustrated as a color-coded display known as a "corneal map". This display is similar to a topographic map except that instead of having zones of equal height, the "isobars" are zones of equal values of some parameter displayed in the same color. In current corneal topography instruments, the curvature at a given point on the cornea is computed only along a meridian. This approach has shortcomings for non-rotationally symmetric corneas. This occurs for common astigmatic or toric corneas, as well as for more irregular shapes such as is the case in post-surgical or keratoconic corneas, where the displayed corneal maps can be especially misleading. The word "keratoconic" literally means "conical cornea", so named because a keratoconic cornea has a tiny protrusion whose shape is somewhat reminiscent of a cone. The usual parameters that are displayed is called axial curvature and instantaneous curvature. However, they can produce misleading results. For example, changing the direction of the patient’s gaze can significantly alter the both the values as well as the shape of the corneal map. We have developed two alternative parameters, "Gaussian power with cylinder", that overcome these problems. Gaussian power is related to the geometric mean of the maximum and minimum curvatures at each data point and cylinder is related to their difference. For different gaze directions, the shape and values of the patterns of the corneal map remain invariant with our Gaussian power map. 2. The VESTA Project: Virtual Surgery for Surgical Training and Augmentation As with any complex mechanical task, surgical procedures require surgeons to practice intensively in order to master the necessary techniques. However, traditional training approaches, such as practice on rubber models, cadavers, or animals, are not conducive to easy repetition of a procedure and they typically do not accurately reflect the anatomical conditions to be expected during surgery on a live patient. These methods also provide a student with only limited exposure 


The OPTICAL Project: OPtics and Topography
Involving the Corneal and Lens 2 The cornea is a clear fibrous tissue that covers the front of the eye.The cornea extends outward slightly from the vaguely spherical shape of the eye.This is the entry point of light into the eye and also the main refractive surface.There is a thin layer of tears which covers the front surface of the cornea; this prevents the cornea from drying and protects it from contaminants in the air. 1 This paper originated as a conference presentation in China, and was originally approved by the guest editor, Prof. Zhigeng Pan, to be included in the special IJVR issue devoted to VR in China (IJVR4-4).However, IPI Press did not include the paper in that issue because it was submitted in postscript (.ps) format, which we were not equipped to handle, as well as lacking an abstract, references, a biography, and author contact information.However, Prof. Pan prevailed upon us to include the paper in this later issue anyway, so we extracted the text from the file, and Editor-in-Chief Richard Blade wrote the abstract, as well as got biographical and contact information from the web.Though the content of the paper is clearly interesting and worthwhile, references and some images would have made it much more useful to our readers, and we apologize for the lack of those. 2 See website: http://www.cs.berkeley.edu/optical/Accurate and detailed information about the shape of the cornea is critical for the diagnosis and correction of vision problems.The cornea is the main refractive element in the human visual system, performing 3/4 of the refraction, or bending, of light in the eye.Because of this primary role in the human visual system, the quality of vision is determined by the shape of the cornea, and even minute subtle shape variations, at the micron level, can have significant effects on visual performance.
Eye care practitioners need to know the shape of a patient's cornea to fit contact lenses, to diagnose keratoconus, an eye condition where the cornea has an irregular shape with a local protrusion, or "cone", which has dramatic effects on vision, and to plan and evaluate the results of various types of surgery that improve vision by altering the cornea, referred to as corneal refractive surgery.
Recent laser surgical techniques are one of the driving forces behind the enthusiasm for the development of techniques to precisely measure the shape of patients' corneas and report those results with an intuitive and natural display.This information is needed to screen out corneal conditions that are contraindicated for this surgery, to do surgical planning, and to evaluate the results of such surgery.When performing surgery such as radial keratotomy and excimer laser ablation, it is essential that corneal surgeons fully understand the topography of the patient's cornea.Improved scientific visualization techniques can greatly aid in surgical planning and evaluation.
Although this underscores the importance of having such accurate shape information, ironically the commercially available instruments to measure corneal shape lack the sophisticated mathematics and computer algorithms to achieve this goal, and often display misleading and erroneous results.
Recently, instruments to measure corneal topography have become commercially available.These devices, called videokeratographs, typically shine a pattern (usually rings) of light onto the cornea and then capture the reflection pattern on a CCD array using a built-in video camera.Given such an image of distorted rings, it is necessary to develop a reconstruction algorithm to recover the corneal shape.
Algorithms for the reconstruction of corneal shape as well as for scientific visualization techniques to depict corneal shape suffer from the approach of thinking in terms of one-dimensional quantities along radial directions measured relative to an assumed center.These directions are called meridians and are analogous to lines of longitude on the globe.The current corneal maps displayed on all commercially available instruments are really one-dimensional, defining and ignoring shape information along any other direction.
We have developed a novel algorithm the reconstruction of corneal shape.The construction begins by guessing a possible surface shape.Then the algorithm measures the difference between the guessed surface and the real cornea.
First we identify features in the image and their corresponding points in the source pattern.If we assume that the lens system can be modeled by a pinhole or nodal point, then we can simulate the process that formed the image by using backward ray tracing.A ray from an image feature is traced through the nodal point to the surface.If the guessed surface has the correct local shape, the ray will intersect the source pattern at the corresponding feature.
More commonly, the surface is incorrect, so the ray misses the feature.The aim is to change the surface so that the ray intersects the correct location.However, we must change the shape of the surface globally, otherwise rays from other features will still miss their corresponding features.
The appropriate global change is computed using constrained optimization.From the traced rays we formulate an error function that measures the difference between the guessed surface and the true cornea.The surface that minimizes this error function has a similar shape to the true cornea.In order to make the problem more easily solved, we constrain the surface to interpolate one or more points.These constraints and the error function define a stand and constrained minimization problem.We solve this problem iteratively, by taking an initial guess, and stepping toward the solution.We have carefully formulated the error function and chosen a surface representation so that each step can be performed efficiently.Each iteration requires tracing rays, computing a set of normals, and fitting a new surface (the surface changes shape to indicate the fitting is taking place) to the normals.
One of the challenges of corneal topography is to convey the shape of the cornea to clinicians.The difficulty lies in the scale of the deviations that need to be depicted.Elevation changes at the micron level are significant, yet the cornea is on the order of a centimeter in diameter; this can be four orders of magnitude different!Hence, a straightforward display of the corneal surface will yield a smooth surface appearance for all corneas and will not reveal the subtle shape that we wish to depict.Consequently, corneal shape is usually illustrated as a color-coded display known as a "corneal map".This display is similar to a topographic map except that instead of having zones of equal height, the "isobars" are zones of equal values of some parameter displayed in the same color.
In current corneal topography instruments, the curvature at a given point on the cornea is computed only along a meridian.This approach has shortcomings for non-rotationally symmetric corneas.This occurs for common astigmatic or toric corneas, as well as for more irregular shapes such as is the case in post-surgical or keratoconic corneas, where the displayed corneal maps can be especially misleading.The word "keratoconic" literally means "conical cornea", so named because a keratoconic cornea has a tiny protrusion whose shape is somewhat reminiscent of a cone.
The usual parameters that are displayed is called axial curvature and instantaneous curvature.However, they can produce misleading results.For example, changing the direction of the patient's gaze can significantly alter the both the values as well as the shape of the corneal map.
We have developed two alternative parameters, "Gaussian power with cylinder", that overcome these problems.Gaussian power is related to the geometric mean of the maximum and minimum curvatures at each data point and cylinder is related to their difference.For different gaze directions, the shape and values of the patterns of the corneal map remain invariant with our Gaussian power map.

The VESTA Project: Virtual Surgery for Surgical Training and Augmentation 3
As with any complex mechanical task, surgical procedures require surgeons to practice intensively in order to master the necessary techniques.However, traditional training approaches, such as practice on rubber models, cadavers, or animals, are not conducive to easy repetition of a procedure and they typically do not accurately reflect the anatomical conditions to be expected during surgery on a live patient.These methods also provide a student with only limited exposure to variations in anatomy and pathology.In contrast, computer-based surgical simulations can model a wide variety of different anatomical configurations, admit the possibility of quickly and easily repeating a procedure as many times as the student might wish, and incorporate a level of modeling that is as accurate as is necessary to train the task at hand.
In general, surgical simulation development involves the three issues of: geometric modeling of anatomical structures as three-dimensional surface meshes or volumetric objects, physical modeling of the mechanical behavior of these structures, and design of an interface to perform all the tasks arising in the procedure being simulated.Graphical organ models can be generated from a variety of medical imaging modalities, such as CT, MRI, or ultrasound, using techniques like the marching cubes algorithm.The designer has a very wide variety of options for modeling the physical behavior of objects in the environment.Usually, most objects in a surgical simulation are highly deformable.Such objects can be modeled as mass-spring-damper meshes, with finite element methods, or as dynamic parametric surfaces, as well as by many other approaches.The primary issues in choosing a physical modeling technique are computational complexity and modeling accuracy.As is so often the case, one must frequently sacrifice speed for accuracy and vice versa.A simulation's interface must require the student to perform the same movements that are necessary in the real procedure to accomplish key tasks, and it should provide adequate response in the form of haptic or force feedback.
Consequently, we have developed a virtual environment simulation of the surgical procedure of minimally-invasive gallbladder removal, which is k nown as laparoscopic cholecystectomy.The simulation is designed to enable us to easily experiment with different approaches to addressing the three issues mentioned above, and it represents our attempt to leverage a great deal of surgical experience in creating a simulation that focuses on the most important, most error-prone steps in the procedure.The system consists of the simulation code running on a dual-processor SGI Octane and an interface device constructed from actual laparoscopic instruments and modified Sensable Technologies PHANToM forcefeedback devices.The simulation provides the user with the view from a simulated laparoscope (a camera) inserted in the abdomen of the virtual patient.The image is display ed on a monitor as in the actual procedure, and the student stands in front of the monitor and interacts with the virtual environment by manipulating three laparoscopic instruments interfaced to the computer.One instrument controls the camera, and the other two control tools that can be changed during the simulation corresponding to the situation.We model the gallbladder, biliary ducts, and occluding fat as deformable objects using massspring-damper meshes.Although this is not a quantitatively accurate technique, the use of such meshes allows us to integrate our dynamic meshes in real time.We also model the liver as a rigid body, because it is not necessary to interact with this organ during the procedure.The system can be configured to provide instructions for each step in the procedure or to simply indicate when the operation has been performed successfully or has failed.In addition, we can incorporate real patient data into the simulation by generating models from a patient's MRI scan.
Our system provides a modular testbed for a variety of experiments in simulation and physical modeling as well as serving as a proof-of-concept for our task-based design.Currently, we are expanding the simulation code to serve as a general-purpose surgical simulation authoring tool, experimenting with new tissue modeling techniques, and constructing a new haptic interface device.

Funding Acknowledgement
This work was supported in part by the National Science Foundation under grant number ASC-9720252, "Visualization and Simulation in Scientific Computing for the Corn ea", and under grant number CDA-9726362, "Virtual Environments for Telesurgery and Surgical Training: Efficient Computation, Visualization, and Interaction".