Modal identification involves estimating the modal parameters, such as modal frequencies, damping ratios, and mode shapes, of a structural system from measured data. Under the condition that noisy impulse response signals associated with multiple input and output locations have been measured, the primary objective of this study is to apply the local or global noise removal technique for improving the modal identification based on the polyreference time domain (PTD) method. While the traditional PTD method improves modal parameter estimation by over-specifying the computational model order to absorb noise, this paper proposes an approach using the actual system order as the computational model order and rejecting much noise prior to performing modal parameter estimation algorithms. Two noise removal approaches are investigated: a "local" approach which removes noise from one signal at a time, and a "global" approach which removes the noise of multiple measured signals simultaneously. The numerical investigation in this article is based on experimental measurements from two test setups: a cantilever beam with 3 inputs and 10 outputs, and a hanged plate with 4 inputs and 32 outputs. This paper demonstrates that the proposed noise-rejection method outperforms the traditional noise-absorption PTD method in several crucial aspects.