This paper presents a novel surgery navigation system based on a three-dimensional (3D) imaging technique, integral videography (IV). In our system, the 3D structure of the object of interest is reconstructed using surface rendering and corresponding pixel distribution methods. We developed a high-speed algorithm that renders high-quality IV images from the surface model in real time and allows interactions like rotating and scaling to be done smoothly. Using the patient-image registration method, IV images can be displayed with the correct size and relative position with respect to the surgical instruments. Experiments were carried out with various anatomical models, and the results show that our system could be useful in many clinical situations such as orthopedic surgery and neurosurgery.