Motivated by the seismic damage observed to reinforced concrete (RC) frame structures during the Wenchuan earthquake, the effect of infill walls on the seismic performance of a RC frame is studied in this paper. Infill walls, especially those made of masonry, offer some amount of stiffness and strength. Therefore, the effect of infill walls should be considered during the design of RC frames. In this study, an analysis of the recorded ground motion in the Wenehuan earthquake is performed. Then, a numerical model is developed to simulate the infill walls. Finally, nonlinear dynamic analysis is carried out on a RC frame with and without infill walls, respectively, by using CANNY software. Through a comparative analysis, the following conclusions can be drawn. The failure mode of the frame with infill walls is in accordance with the seismic damage failure pattern, which is strong beam and weak column mode. This indicates that the infill walls change the failure pattern of the frame, and it is necessary to consider them in the seismic design of the RC frame. The numerical model presented in this paper can effectively simulate the effect of infill walls on the RC frame.