Simulation sickness in virtual reality applications is usually due to the non-realtime display of virtual environment. Realtime display of arbitrarily complex scene is a hard problem in traditional geometry-based computer graphics. Image-based modeling and rendering (IBMR) provides an alternative approach whose rendering time complexity is independent of scene complexity. However, due to the lack of geometrical information, capabilities that are obvious to geometry-based virtual reality become difficult problems in image-based virtual reality. In this paper we discuss how two fundamental capabilities, navigation and illumination control, can be achieved in image-based virtual reality applications. To navigate an image-based scene, we need to know where pixels should be moved to and how to solve their visibility. While correspondences or optical flows can answer the former question, the visibility is more difficult to answer, as depth information may not be available. Deriving from epipolar geometry, we propose a triangle-based visibility-ordering algorithm, which can correctly resolve the occlusion without depth information. To control the illumination, we propose a new image representation that not just allows navigation but also re-rendering under various illuminations. By treating each image pixel as an ordinary surface element, we measure the apparent BRDF of each pixel from reference images. By manipulating these apparent pixel BRDFs, we are able to re-render (change the illumination of the scene in an image) without any geometry information. Even shadows can be correctly re-rendered.