The mapping relationship between the charge-coupled device (CCD) of a camera and the digital micro-mirror device (DMD) of a projector, together with the algorithm to estimate the parameters of the projector, determines the accuracy of projector calibration. In this paper, a new method is presented to achieve sub-pixel-level mapping between the camera and projector images. Instead of directly mapping the circle centers from the CCD to the DMD, which is pixel-precision-based, a set of pixels on the circle edge are extracted on the CCD and mapped onto the DMD. The locations of the circle centers are extracted on the DMD by the least-squares fitting technique, thus achieving sub-pixel precision. Compared with existing approaches based on the direct mapping of the circle centers on the camera image to the projector image, the proposed method is characterized by higher accuracy in mapping the points and thus the projector calibration. The experimental results are presented to show the effectiveness of the proposed method in the improvement of the accuracy of the projector calibration.