Virtual assembly is a promising application of virtual reality in design and manufacturing that has drawn much attention from industry and research institutes. Physically based modeling has been an important research topic in computer graphics and virtual reality. In this paper, physically based modeling issues in virtual assembly are investigated. The specific requirements and characteristics of physically based modeling in virtual assembly versus those in traditional computer graphics are analyzed and studied. The mass properties of the assembly models are extracted from the Computer Aided Design (CAD) system while the design models are transferred from the CAD system to the virtual assembly environment. The assembly models are categorized using human strength survey data. The interaction between the parts, the environment objects, and the human are analyzed. In the fully immersed virtual environment, it is discovered that the gravity acceleration needs to be scaled down to achieve maximum realistic feeling. Finally, the benefits and limitations of physically based modeling in virtual environments are discussed.