This study proposes a motion cue based pedestrian detection method with two-frame-filtering(Tff) for video surveillance.The novel motion cue is exploited by the gray value variation between two frames.Then Tff processing filters the gradient magnitude image by the variation map.Summations of the Tff gradient magnitudes in cells are applied to train a pre-detector to exclude most of the background regions.Histogram of Tff oriented gradient(HTffOG) feature is proposed for pedestrian detection.Experimental results show that this method is effective and suitable for real-time surveillance applications.