In contrast to defect-free fabric images with macro-homogeneous textures and regular patterns, the fabric images with the defect are characterized by the defect regions that are salient and sparse among the redundant background. Therefore, as an effective tool for separating an image into a redundant part (the background) and sparse part (the defect), the low-rank decomposition model provides an ideal solution for patterned fabric defect detection. In this paper, a novel patterned method for fabric defect detection is proposed based on a novel texture descriptor and the low-rank decomposition model. First, an efficient second-order orientation-aware descriptor, denoted as GHOG, is designed by combining Gabor and histogram of oriented gradient (HOG). In addition, a spatial pooling strategy based on human vision mechanism is utilized to further improve the discrimination ability of the proposed descriptor. The proposed texture descriptor can make the defect-free image blocks lay in a low-rank subspace, while the defective image blocks have deviated from this subspace. Then, a constructed low-rank decomposition model divides the feature matrix generated from all the image blocks into a low-rank part, which represents the defect-free background, and a sparse part, which represents sparse defects. In addition, a non-convex log det as a smooth surrogate function is utilized to improve the efficiency of the constructed low-rank model. Finally, the defects are localized by segmenting the saliency map generated by the sparse matrix. The qualitative results and quantitative evaluation results demonstrate that the proposed method improves the detection accuracy and self-adaptivity comparing with the state-of-the-art methods.