Glaucoma is one of the leading causes of blindness in the world and there is no cure for it yet. But it is very meaningful to detect it early as earlier detection makes it possible to stop further loss of visions. Although deep learning models have proved their advantages in natural image analysis, they usually rely on large datasets to learn to extract hidden features, thus limiting its application in medical areas where data is hard to get. Consequently, it is meaningful and challenging to design a deep learning model for disease diagnosis with relatively fewer data. In this paper, we study how to use deep learning model to combine domain knowledge with retinal fundus images for automatic glaucoma diagnosis. The domain knowledge includes measures important for glaucoma diagnosis and important region of the image which contains much information. To make full use of this domain knowledge and extract hidden features from image simultaneously, we design a multi branch neural network (MB-NN) model with methods to automatically extract important areas of images and obtain domain knowledge features. We evaluate the effectiveness of the proposed model on real datasets and achieve an accuracy of 0.9151, sensitivity of 0.9233, and specificity of 0.9090, which is better than the state-of-the-art models.