© 2019 IEEE. This paper introduces a hierarchical learning paradigm based on a predesigned directional filter bank front-end analogous to the energy model for complex cells. The filter bank front-end is designed to extract common primitive features such as orientations and edges. Each energy response is subjected to a shunting inhibition operator to enhance contrast and reduce the effects of illumination variations. This is followed by a divisive-normalization, which bounds the responses of the feature maps. The normalized responses are then propagated through a two-layer convolutional neural network (CNN) back-end for classification. The efficiency of the proposed approach is demonstrated using the CIFAR-10 dataset, and its performance is compared against that of the DTCWT ScaterNet front-end.