The inherent nonlinearities of the rudder servo system(RSS) and the unknown external disturbances bring great challenges to the practical application of fault detection technology. Modeling of whole rudder system is a challenging and difficult task. Quite often, models are too inaccurate, especially in transient stages. In model based fault detection, these inaccuracies might cause wrong actions. An effective approach, which combines nonlinear unknown input observer(NUIO) with an adaptive threshold, is proposed. NUIO can estimate the states of RSS asymptotically without any knowledge of external disturbance. An adaptive threshold is used for decision making which helps to reduce the influence of model uncertainty. Actuator and sensor faults that occur in RSS are considered both by simulation and experimental tests. The observer performance, robustness and fault detection capability are verified. Simulation and experimental results show that the proposed fault detection scheme is efficient and can be used for on-line fault detection.