Laser tracking system (LTS) is an advanced device for large size 3D coordinates measuring with the advantages of broad range, high speed and high accuracy. However, its measuring accuracy is highly dominated by the geometric errors of the tracking mirror mechanism. Proper calibration of LTS is essential prior to the use of it for metrology. A kinematics model that describes not only the motion but also the geometric variations of LTS is developed. Through error analysis of the proposed model, it is claimed that gimbals axis misalignments and tracking mirror center off-set are the key contributors to measuring errors of LTS. A self-calibration method is presented of calibrating LTS with planar constraints. Various calibration strategies utilizing single-plane and multiple-plane constraints are proposed for different situations. For each calibration strategy, issues about the error parameter estimation of LTS are exploded to find out in which conditions these parameters can be uniquely estimated. Moreover, these conditions reveal the applicability of the planar constraints to LTS self-calibration. Intensive studies have been made to check validity of the theoretical results. The results show that the measuring accuracy of LTS has increased by 5 times since this technique for calibration is used.