Fabric defect detection plays an important role in automated inspection and quality control in textile manufacturing. As the fabric images have complex and diverse textures and defects, traditional detection methods show a poor adaptability and low detection accuracy. Robust principal component analysis (RPCA) model that can be used to separate the image into object and background have proven applicable in fabric defect detection. However, how to represent texture feature of the fabric image more effectively is still problematic in this kind of method. In addition, the use of the traditional RPCA may result in low accuracy and more noises in sparse part. In this article, a novel fabric defect detection method based on multilevel deep features fusion and non-convex total variation regularized RPCA (NTV-RPCA) is proposed. Firstly, the image representation ability is well enhanced through multilevel deep features extracted by a convolutional neural network. Then, the non-convex total variation regularized RPCA is proposed in which total variation constraint significantly reduces the noises in sparse part and non-convex solution is more approximate to the authentic one. Next, multilevel saliency maps generated by the sparse matrixes are fused via RPCA to produce a more reliable detection result. Finally, the defect region is located by segmenting the fused saliency map via a threshold segmentation algorithm. Qualitative and quantitative experiments conducted on two public fabric image databases demonstrate that the proposed method improves the adaptability and detection accuracy comparing to the state-of-the-arts.