The measurement of the rotation error of an aerostatic ultra-precision spindle is critically important to evaluate and hence ensure the precision of machine tools. The Donaldson reversal method, which was taken as the efficient method for error separation theoretically, has been widely used to separate shape errors of standard artifact. However, the accuracy analysis of the Donaldson reversal method has not been fully studied and understood. In this study, a nanometer system for measuring the radial rotation error of aerostatic ultra-precision spindle was constructed based on the Donaldson reversal method. The comparative experiments were carried out to investigate the effects of the motor drive, and an angle correction algorithm was proposed to alleviate the effect of angle deviation. The method of harmonic analysis was applied to investigate the effect of artifact eccentricity, and the relationship between the axial motion and measuring error was also studied. The measuring accuracy can be improved by reducing the cogging torque of motor, the angle deviation, artifact eccentricity and spindle axial motion. Experimental results showed that the measurement uncertainty of both the spindle rotation error and artifact form error can be controlled in nanometer level. Besides, the separated value of the artifact form error was very close to the nominal roundness, which verifies the accuracy of the measurement system and the validity of the error separation method.