View Synthesis for Realistic Virtual Walk Through Based on Omni-directional Images

—Virtual walk through can be widely applied in many industries such as virtual environment construction, history heritage conservation and scenic site exhibition etc. This paper proposes a more convenient and efficient approach to create realistic virtual walk through from catadioptric omni-directional images via view synthesis technique. Our innovation mainly lies in three aspects: omni-directional image preprocessing, image rectification and novel view interpolation. Acquisition and unwarping of omni-directional image are discussed firstly. Then, for specialty of cylindrical panoramic imaging, epiline-sampling method is adopted for rectification, which samples reference images along epilines as much as possible grounding on epipolar geometry. In this way, it can detract the rectified images from image deformation and resolution degeneration, which usually take place due to perspective transformation when using some other algorithms. As to novel view generation, a corresponding interpolation algorithm is developed. Pixels on novel view are formulated according to the cylindrical panoramic imaging model. Experiments carried out on both synthetic and real scene are given at the end of this paper, with a demonstration of the method’s application in realistic virtual walk through.


INTRODUCTION
With the advantage of wide field of view (360°horizontally), Technologies based on catadioptric omni-directional images develop rapidly, and has been vastly applied in 3D reconstruction, virtual reality, robot navigation, augmented reality and so on. Therefore, as to convenient and efficient virtual walk through application, a more nature idea is to utilize omni-directional images for the wider FOV, compared with other methods ground on normal perspective images.
Generally, to create realistic virtual walk through from omni-directional image pair via view synthesis technique, there are three key steps: (1) reference images acquisition and preprocessing; (2) image rectification; and (3) novel view synthesizing. Our earlier work [1] has proposed a catadioptric omni-directional imaging system named PROIS (Parabololid Reflective Omni-directional Imaging System) which is the image capturing system used in this paper as well. Therefore here, we mainly focus on the latter two topics.
Manuscript received on 20 November 2008 E-Mail: chenwang@nudt.edu.cn Different approaches for rectifying omni-directional image pair has been exploited [2][3][4][5]. In [2], the authors represented omni-directional image coordinates as a point z in complex plane, and then specified the rectification by 1 coth z  . This method is shown to be conformal, locally distortionless, and unique up to scale and transformation. [4] encoded the nonlinearity of the projection model of omni-directional image system in a catadioptric fundamental matrix with bilinear form, and estimated it with a quotient of two Lie groups to implement the rectification. [3] and [5] similarly introduced the geometric characterization of panoramic omni-directional camera and applied warping equations to images, so that the epipolar lines of image pair can become parallel scanlines.
However, with the consideration that catadioptric omni-directional image is concentric deformed while our goal is to generate virtual walk through for human being, it is sensible to keep those captured images in an omni-directional style and meanwhile more natural to human vision. Thereby in this paper, we adopt an approach different from all methods above. We first unwarp the catadioptric omni-directional images to cylindrical panoramic images, which are not only omni-directional but also more natural to derive perspective views. Then, adopt an appropriate method to rectify cylindrical panoramic image pair. And lastly generate novel view from rectified images according to pixel correspondence.
Lots of work can be found in the literature on cylindrical panoramic images rectification. References [6][7][8] studied the epipolar geometry of cylindrical image and proposed relevant mathematical formulas. [7][8][9][10] discussed image rectification and corresponding matching problems of cylindrical image respectively in issues of plenoptic modeling, range estimation and 3D reconstruction. But all of them did not address how to do that in detail. [11] segmented panoramic cylindrical image into several parts along axes direction, projected each segment on to plane tangent to the cylinder, and then rectified the projected images. However, when working with similar approaches exploited in [11], which are intendedly developed for rectifying perspective images, the homographic and perspective project transformation (HPPT for abbr.) is necessarily applied to image pair and therefore inevitable results in resolution degeneration and image distortion [12][13][14], and eventually affects the correctness of corresponding matching. Although [13] presented a method, which transforming a line of planar image into a vertical line of a cylindrical image to minimize the side-effect of HPPT, it can not apply to cylindrical image rectification directly due to the particularity of cylindrical imaging model.
To cope with this problem, in this paper, epipolar geometry and re-sampling effect of cylindrical panoramic image are evaluated firstly. Then with two criterions named retentive-pixel-rate and effective-pixel-rate, which are used for measuring the re-sampling effect, we adopt epiline-sampling method for image rectification, which samples reference images along epilines as much as possible grounding on epipolar geometry. This approach can achieve better performance with respect to the measurement criterions when compared with other methods using HPPT. Lastly, for novel view generation, a view interpolation algorithm based on rectified images is proposed.
Rest of the paper is organized as following. Section 2 briefly describes the PROIS and corresponding unwarping algorithm. Section 3 discusses the epipolar geometry of cylindrical panoramic image pair. In section 4 and 5, image rectification and novel view interpolation algorithms are proposed respectively. And experiments carried out both on synthetic and real scene are given in section 6.

II.
CATADIOPTRIC OMNI-DIRECTIONAL IMAGE ACQUISITION AND UNWARPING Fig.1 illustrates the PROIS in [1] and the corresponding procedure of unwarping catadioptric omni-directional image into cylindrical panoramic image [15][16]. Fig.2 depicts the unwarping principle. In Fig.2 (a) there is an catadioptric image with inner radius r and outer radius R . The region between these boundary circles is our interested area.
As illustrated in Fig.3, let 1 V and 2 V denote two viewpoints with origins 1 v and 2 v . 1 C and 2 C are two cylindrical panoramic images, and p indicates a scene point with coordinates For generality, we assume coordinates 1 V is coincide with world coordinates. The plane spanned by 1 v , 2 v and p is so-called epipolar plane, and its intersection lines EPL1 L and EPL2 L with 1 C and 2 C are named epilines.   (2) and (3) present the epipolar geometry of cylindrical panoramic image pairs.

Coordinates transforming
Since cylindrical panorama image is in fact stored in plane format, as shown in section 2, and for what we actually have is the pixel's image coordinates, a transformation from image coordinates to cylindrical camera coordinates is necessary. For point q in cylindrical panorama with corresponding image pixel ( , ) uv , the transforming is: Substitute  and v in (4) with those in (5), the coordinates transformation can be done.

EPILINE-SAMPLING
Re-sampling effect is inevitable when undergoing image rectification. It usually leads to resolution degeneration and image distortion. But different approaches result in distortions at different degrees. The less the re-sampling effect is, the better the rectification algorithm is. In view of the specialty of the cylindrical panoramic imaging model, we found that rectification via re-sampling reference images along epilines is superior to other approaches using HPPT in terms of retentive-pixel-rate and effective-pixel-rate, which are criterions been propose to measures re-sampling effect [2], [17]- [19]. For completeness, here we give their definitions as following.
Retentive-pixel-rate: the ratio between the amount of pixels re-sampled from reference image and the total pixel number of the reference image.
Effective-pixel-rate: the ratio between the amount of pixels re-sampled from reference image and the total pixel number of the rectified image. Therefore, with these criterions, to reduce the re-sample effect and to improve the veracity of correspondence matching, we exploit the epiline-sampling approach rather than applying HPPT to reference images, i.e. under the constraint of epipolar geometry, figure out the epilines of the cylindrical images and sample pixels along them.
As Fig. 4 shown, assume the radius and height of 1 C , 2    / ee m f y x  ) be  . In order to sample pixel information on 1 A , 2 A as much as possible, the range of  should be as large as possible, that is to say, epipolar plane should intersect 1 A and 2 A maximally. Calculating the largest range of  should take vertex's coordinates of 1 A , 2 A into consideration. x y z denotes vertexes above PH , otherwise, denotes vertexes below PH ) , and the normal of the plane containing ( , , ) [0, max(β )] above PH , the intersection line between epipolar plane and 1 C , 2 C can sample 1 A , 2 A as much as possible.
Following is the algorithm of calculating epilines on 1 A and 2 A . Let x y z , 1 e , 2 e be max z , and Z coordinates of intersection point between the same line and plane spanned by ( , , ) k k k n n n x y z , 1 e , 2 e be min z , then according to (3), for any VI. EXPERIMENTS Experiments are carried out on both synthetic and real omni-directional images captured with PROIS, and we compare our results with those of [11] which rectifies images based on HPPT.
We firstly rectify parts of cylindrical images with algorithm of HPPT and proposed method. Corresponding results are shown in Fig.6 and Fig.7. Then, stereo matching is done via approach of [16] and view synthesis results are given in Fig.8 and Fig.9, where the five figures are respective novel views at different positions between the reference cylinders centers, and a presents the ratio of novel viewpoint' distance from the right cylinder center to the total distance of two cylinder centers.  As experiment results illustrated, image rectification with HPPT results in worse image distortions (see Fig.6(b) and Fig.6(b), where images are deformed seriously), while the proposed approach gives better results, since it keeps the epiline length and the dimension of rectified image just the same as the reference images, which will certainly lead to less resolution degeneration and image distortion, as shown in Fig.6(c) and Fig.  7(c). A more detail comparison between the two approaches on retentive-pixel-rate (RPR for abbr.) and effective-pixel-rate (EPR for abbr.) is given in Table 1.
We also exploit the proposed technique in recent application for scenic site exhibition. Fig.10 shows some snapshots of this project, in which users can ramble in the city randomly by map navigation and can look around at each spot.

VII. CONCLUSIONS
For more conveniently and efficiently creating virtual walk through in real scene, this paper proposed an approach base on catadioptric omni-directional images and view synthesis technique, and mainly discussed image rectification and view interpolation for cylindrical panoramic images. Our approach utilizes epiline-sampling method for cylindrical image, which grounds on the epipolar geometry and samples reference image along epiline as much as possible. Experiment shows the proposed approach can reduce image distortion and resolution degeneration obviously and maintain the scene information well. Compared with algorithm based on homographic and perspective project transformation, our method is much better. Meanwhile, view interpolation from rectified image pairs has good performance as well. However, when try to rectify cylindrical panoramic images at the head direction of epipolars, there will be somewhat trouble. Because regions adjacent to epipolars can not be sampled enough easily, since all epilines intersect at epipolars. One possible solution is to capture omni-directional images at different viewpoints more than two, and from every two among those, rectify parts of them efficiently except the area near epipolars, then synthesize rectified region together to form a whole novel cylindrical panoramic image.