This article proposes a fast and accurate network for surface defect detection, termed SDDNet. SDDNet mainly addresses two challenging issues - large texture variation and small size of defects - by introducing two modules: feature retaining block (FRB) and skip densely connected module (SDCM). FRB fuses multiple pyramidal feature maps with different resolutions and is plugged on the top of pooling layers, aiming to preserve the texture information, which may be lost because of downsampling. SDCM is designed to propagate the fine-grained details from low- to high-level feature maps for better prediction of defects, especially small defects. Extensive experiments conducted on the publicly available data sets NEU-DET (88.8% mAP), DAGM (99.1% mAP), and Magnetic-Tile (93.4% mAP) have demonstrated the effectiveness of the proposed SDDNet and its feasibility for real-time industrial applications.