Image-based modeling methods for generating 3D models from an image sequence have been widely studied. Most of these methods, however, require huge redundant spatio-temporal images to estimate scene depth. This is not an effective use of capturing higher resolution texture. On the other hand, a route panorama, which is a continuous panoramic image along a path, is an efficient way of consolidating information from multiple viewpoints into a single image. A route panorama captured by a line camera also has the advantage of capturing higher resolution easily. In this paper, we propose a method for estimating the depth of an image from a route panorama using color drifts. The proposed method detects color drift by deformable window matching of the color channels. It also uses a hierarchical belief propagation to estimate the depth stably and decrease the computation cost thereof.