Fast 3 D Shape Recovery from Shadows Projected on Arbitrary Curved Surfaces

The shadow based 3D surface reconstruction methods usually assume that shadows are projected on planar surfaces. However, shadows are often projected on curved surfaces in the real scene. Recently, the shadow graph has been proposed for representing shadow information efficiently, and for recovering 3D shapes from shadows projected on curved surfaces. Unfortunately, the method requires a large computational cost and is weak to the image intensity noises. In this paper, we introduce 1D shadow graphs which can represent shadow information quite efficiently, and can be used for recovering 3D shapes with much smaller computational costs than before. We also extend our method, so that we can recover 3D shape quite accurately by using shading information as well as shadow information. The proposed method is tested by using the real and synthetic images.


INTRODUCTION
The shape from shading has been studied extensively for recovering accurate 3D shapes, and the propagation based methods [1], the energy minimization methods [2] and the local methods [3] have been proposed. Although the shape from shading is useful for recovering 3D shapes from single view images, it requires strong constraints such as smoothness of surfaces, constant albedo, and so on. It is also weak to image noises, since it uses differential properties of surface and intensity.
If the light source moves in the 3D space, shadows of objects caused by the light are also useful for recovering 3D shapes. Such methods are called shape from shadow [4,5]. Recently, the shadow graph [5] has been proposed for representing shadow information in images. It can represent the relationship between attached shadows and cast shadows in a directed graph with costs, and is used for recovering 3D shapes from shortest path finding. Unfortunately, shortest path finding in the shadow graph requires a lot of computational costs, and we cannot recover 3D shapes in a reasonable amount of time.
To cope with this problem, we in this paper introduce 1D shadow graphs which are extracted from constant circular motions of a light source. We show that the 1D shadow graphs Manuscript received on 16 March 2009 E-Mail: junsato@nitech.ac.jp can be used efficiently for recovering 3D shapes from shadow information, and the computational costs can be reduced drastically. We also show that shading information can be used for extracting accurate shadow areas, and hence for recovering accurate 3D shapes from shadows.

II. SHADOW GRAPH
We first explain the shadow graph proposed by Yu et al [5]. The shadow graph is a directed graph with costs which represent the relative height of points computed from the shadow and the orientation of light source. Fig.1 (b) is a shadow graph of Fig.1  (a). As shown in these figures, a node of the graph corresponds to a point on the surface, and the cost between two nodes represents the relative height between two points on the surface. Let us consider a 3D surface as shown in Fig.1 (c), and let h(x) be the height of the surface at point x. The light source is at infinity and the orientation of the light source is  . Suppose [x a , x c ] is a shadow area as shown in Fig.1 (c). x a is the light source side boundary of the shadow area, and is called an attached shadow boundary. x c is the anti-light source side boundary of the shadow area, and is called a cast shadow boundary. In the shadow graph, the following two types of shadow constraints are extracted from the attached and the cast shadow boundaries, and these constraints are represented by the cost of edges of the graph.

Anti-Shadow Constraints
We next consider constraints for a surface point which is not in the shadow area ] , [ where, C between node l x and node x.

3D Shape Recovery from Shadow Graph
Suppose a 3D surface is illuminated by a moving light source at infinity, and is viewed from a fixed camera with orthogonal projection. In the shadow graph extracted from different orientations of light source, each node is a pixel of the camera image, and each cost of edge between a pair of nodes represents a relative height between a pair of pixels. Thus, the upper bound of relative height between an arbitrary pair of image pixels can be computed by finding the shortest path between a pair of nodes in the shadow graph. Thus, the 3D shape can be recovered by finding the shortest paths for all the pairs of nodes in the graph.
There are many methods for finding the shortest path in a graph. However, most of them do not work properly when there are negative costs in a graph as the shadow graph. Thus, in this research we use Warshall-Floyd algorithm [6] which can cope with negative costs.
If we find all the shortest paths between a fixed point and other points in the graph, then we can recover relative height between the fixed point and other points, and hence we can recover the 3D shape relative to the fixed point. Unfortunately, this method requires a lot of computational cost, and the cost for for N nodes, and we have 2 N nodes from an N N  image. This is unrealistic in practice.

III. FAST 3D SHAPE RECOVERY FROM 1D SHADOW GRAPHS
In this section, we propose a method for recovering the 3D shape from a combination of 1D shadow graphs. In this method, we consider a single 2D image as a set of 1D image data in two independent directions, such as row and column directions, and generate 1D shadow graphs for these 1D image data. These 1D shadow graphs are combined together for recovering the 3D shape of the surface. Since the number of nodes used in a single path finding in a 1D shadow graph is very small, the computational cost for recovering 3D shapes is much smaller than the original method. The computational cost for In the following part of this section, we explain our method assuming these two independent directions are row and column directions. However, the method can be applied for any two independent directions of light source motions.

3D Slice Reconstruction from 1D Shadow Graph
Suppose the light source moves in row and column directions in the 3D space as shown in Fig.2. Under these light source motions, the extracted shadow graphs are connected only in the row direction or in the column direction, and thus the extracted graphs are independent in rows and columns of an image. By finding the shortest path in each 1D graph, we can recover a set of row slices and column slices of the 3D surface with low computational cost. Since 1D graphs do not have connection with other rows and columns, the row and column slices of the surface recovered from the 1D graph have an ambiguity of vertical translation. Thus, we next fix the vertical position of each slice and recover the whole shape of the surface.

3D Shape Recovery from 1D Shadow Graph
Let us consider a pixel   j i, in the N N  image. From the 3D slice reconstruction described in the previous section, we have two slices which go through the pixel   j i, , that is i th row slice and j th column slice as shown in Fig.3. Since each slice reconstructed from 1D Shadow Graph is independent of the other slices, each slice has ambiguity of vertical translation. Although each slice has ambiguity of vertical translation, the height ij h of pixel   j i, in the i th row slice must be identical to the height ij h of pixel   j i, in the j th column slice, if there is no error. Thus, we consider the following cost function for all the points in the image.
where, i t is a vertical translation of i th row slice, and j t is that of j th column slice. By computing (3), we can recover the 3D shape of the surface, which minimizes the error in vertical position of row and column slices. However, the row and column slices recovered from 1D shadow graphs sometimes include large errors caused by image noises and missing edges in the shadow graph. Thus, we in this paper apply a robust regression algorithm, RANSAC [7], for combining row and column slices accurately excluding outliers in the slice data.
Suppose we have N column slices and N row slices from an N N  image. These row slices and column slices intersect at 2 N intersections in the combined data. These slice intersections sometimes include outliers, where the height of row slice and/or column slice has a large error. These slice intersections should be excluded from the minimization of (3).
For excluding these outliers, we first combine row slices and column slices one by one, so that a row slice and a column slice have identical height, ij ij h h   , at the knot   j i, of these two slices, as shown in Fig.4. Then, 1 2  N knots are enough for fixing relative height of N row slices and N column slices. Thus, we chose a set of 1 2  N knots randomly, and count the number of intersections where the relative height between row slice and column slice is less than a threshold. We iterate the procedure several times, and find a set of knots which provides us the maximum number of inlier intersections. Then,

IV. EXTRACTING SHADOWS FROM SHADING INFORMATION
In general shadow areas are extracted from images by using a threshold on image intensity. However, a single threshold is not enough for separating shadow areas from others properly. Furthermore, inter-reflections among surfaces cause distortions in surface brightness, and shadow areas are often classified into non-shadow areas because of the inter-reflections. Thus, we in this section use shading information, i.e. intensity history [8] of image points, for extracting shadow areas properly. The intensity history of an image point is a set of image intensities obtained by changing the light source position. Suppose the light at infinity moves with a constant angular velocity. Then, the image intensity at a fixed image point changes sinusoidally. However, because of the inter-reflections, the intensity history is often distorted as shown in Fig.5 (a). Thus, we fit a quadratic curve to the intensity history by using RANSAC as shown in Fig.5 (b), and classify an image point into shadow area if the intensity of the point is outlier to the quadratic curve. By using the shadow areas extracted from the intensity history, we can recover 3D shapes accurately and reliably from 1D shadow graphs proposed in section 3.

V. REFINING SHDOW GRAPHS FROM SHADING INFORMATION
The intensity history can also be used for refining shadow graphs. The shadow graph sometimes has incorrect edges because of the errors in shadow boundaries. In particular, attached shadow boundaries are sometimes unstable, since they are surface points where light lines are tangent to the surface. Suppose we have a 3D surface whose row or column slice is as shown in Fig. 6 (a). If we have a correct attached shadow boundary, we can reconstruct the slice correctly. However, if the attached shadow boundary has some error, the reconstructed shape of the slice is distorted as shown in Fig. 6 (b). This is because a wrong edge is made from the wrong attached shadow boundary in the shadow graph.
Fortunately, such wrong edges can be removed by using the intensity history. By using the intensity history, we can roughly guess the surface orientation at each point on the surface. This is achieved by simply finding a light source orientation which provides us maximum intensity in the intensity history curve. Such orientation of the light source coincides with the orientation of the surface at each point in the slice. Although there exists one dimensional ambiguity in the measured surface orientation, the ambiguity is in the direction perpendicular to the slice and does not cause any problem in the following discussions. In the slice, the surface orientation at an attached shadow boundary is perpendicular to the light orientation. Thus, if the surface orientation of an attached shadow boundary computed from the light orientation is far from the surface orientation computed from the intensity history, it is considered as a wrong attached shadow boundary. We remove edges in the shadow graph, whose nodes correspond to such wrong attached shadow boundaries. By removing these wrong edges in the 1D shadow graphs, we can refine the 3D shapes. VI. EXPERIMENTS

Real Image Experiments
In this section, we show the results from 3D surface reconstruction from real images. Fig.7 (a) shows a face model used in this experiment. The camera is fixed in front of this object, and the light is moved around the object. The motion of the light source is circular with the radius of 1m. It moves from 0 degrees to 180 degrees with the interval of 10 degrees in two directions, one in the row direction and the other in the column direction of the camera image. Hence, we obtained 38 images in total. These image are shown in Fig.7 (b) and (c). The size of each image is 120 120  . As shown in Fig.7, shadows are on the complex curved surface of the face and we have very complex shadows. The proposed method is applied to these images and the 3D shape of the face is recovered. The recovered 3D shape is shown in Fig.8. Fig.8 (a) and (b) are the result viewed from two different viewpoints. As shown in Fig.8, the complex shape of the face is recovered properly, and we find the proposed method is very efficient for recovering arbitrary 3D surfaces from shadows. The results from some other 3D objects are shown in Fig.9.

Computational Cost Evaluation
We next evaluate the computational cost of the proposed method. The computational time of the proposed method was measured changing the image size from 30 30  to 120 120  , and was compared with that of the 2D shadow graph method. Fig.10 shows the computational time of these two methods. As shown in this figure, the computational time of the proposed method is much faster than that of the 2D shadow graph method.

Accuracy Evaluation
We next evaluate the accuracy of the proposed method by using synthetic image data.
In this experiment, we used a sinusoidal surface shown in Fig.11 (a). The camera is fixed in front of the object assuming orthogonal projection. The light source is at the infinity and moves around the object from 0 degrees to 180 degrees in two directions, one in the direction of rows and the other in the direction of columns of camera image. The image size is 120 120  . The accuracy of the proposed method is evaluated by the difference D between the true height ij h and the height ij h recovered from the proposed method as follows: min max 2 1 (4) where, max h and min h are the maximum height and the minimum height of the true shape, and N is the size of the image.

1) Light Source Interval and Accuracy of Reconstruction:
We first evaluated the relationship between the interval of light source motions and the accuracy of reconstruction. We evaluated in four different intervals of light source motions, that is  5 ,  10 ,  15 and  20 , in which the number of images is 74, 38, 26 and 20 respectively. We added intensity noises with the standard deviation of 2 to all the image pixels. There is no inter-reflection in this experiment.
The 3D shapes recovered from the proposed method in these four sampling intervals are shown in Fig.11, and the relationship between the sampling interval and the error D in (4) is shown in Fig.12. As shown in these figures, the accuracy of the proposed method degrades as the number of light source motions goes down. However, we can obtain reasonable shapes from the proposed method, even though the number of light source motions is small.  2) Inter-Reflections and Accuracy of Reconstruction: We next evaluated the tolerance of the proposed method to the intensity distortions caused by inter-reflections. We generated synthetic images which include inter-reflections up to 3rd order, and used them for recovering 3D shapes from the proposed method. The 3D object is the same as before. The interval of light source motions is  10 , and the intensity noise with the standard deviation of 2 is added to all the image pixels.
The 3D shape reconstructed from the proposed method is shown in Fig.13 (a). As shown in this figure, the 3D shape is recovered quite accurately, even if we have inter-reflections in images. For comparison we recovered the same object without using intensity history for extracting shadow areas. The result is shown in Fig.13 (b). As shown in Fig.13 (a) and (b), the intensity history is very useful for recovering accurate 3D shape from shadows. The error D of recovered shape is 4.93% in (a) and 16.43% in (b) respectively. Fig.14 (a) shows an example image with strong inter-reflections, and (b) and (c) show shadow areas extracted by using a single threshold and the intensity history respectively. As shown in these figures, the proposed method works quite well under inter-reflections.
(a) using intensity history (b) using a single threshold

VII. CONCLUSION
In this paper, we proposed a method for recovering 3D shapes from shadow information projected onto arbitrary curved surfaces. We showed that 1D shadow graphs obtained from two independent orbits of light source motions can be used efficiently for recovering accurate 3D shapes from shadows with small computational costs. We also showed that intensity history is very useful for extracting accurate shadow areas and for recovering accurate 3D shapes from shadow information. The proposed method is robust to intensity noises and is very practical, since light source motions are often circular motions, such as sun motions. The future work includes the extension of the proposed method, so that it can recover 3D shapes from unknown light source motions.