In order to realize the fertility detection and classification of hatching eggs, a method based on deep learning is proposed in this paper. The 5-days hatching eggs are divided into fertile eggs, dead eggs and infertile eggs. Firstly, we combine the transfer learning strategy with convolutional neural network (CNN). Then, we use a network of two branches. In the first branch, the dataset is pre-trained with the model trained by AlexNet network on large-scale ImageNet dataset. In the second branch, the dataset is directly trained on a multi-layer network which contains six convolutional layers and four pooling layers. The features of these two branches are combined as input to the following fully connected layer. Finally, a new model is trained on a small-scale dataset by this network and the final accuracy of our method is 99.5%. The experimental results show that the proposed method successfully solves the multi-classification problem in small-scale dataset of hatching eggs and obtains high accuracy. Also, our model has better generalization ability and can be adapted to eggs of diversity.