With the advances in radar technology, there is an increasing interest in radar-based applications for automated surveillance, tracking, and recognition. Radars capture the Doppler-induced frequency shifts, which can be utilized to recognize the small motions by vibrating or rotating parts of the target. This paper introduces a fast and compact deep Gabor neural network for human motion classification using backscattered signals from a continuous-wave Doppler radar. The joint time-frequency representation produced by the S-method is employed to depict the micro-Doppler signature of a walking person. We introduce a new Gabor layer as a generic feature extractor for designing compact neural architectures. Each Gabor layer consists of several steerable Gabor filters that can be trained to extract the salient micro-Doppler features and improve computation efficiency. For human motion classification from a distance, the Deep Gabor Network (DGN) is trained in an end-to-end manner to process the local patches in the time-frequency representation of the radar signal. A Bayesian optimization technique is employed to select the optimal network hyperparameters. The experimental results on a real radar dataset show that the proposed method achieves competitive classification rates compared to several existing approaches while having a significantly smaller model size and shorter prediction time.