Cataract is a chronic eye disease that causes irreversible vision loss. Automatic cataract detection can help people prevent visual impairment and decrease the possibility of blindness. To date, many studies utilize deep learning methods to grade cataract severity on fundus images. However, they mainly focus on the classification performance and ignore the model interpretability, which may lead to a semantic gap between networks and users. In this paper, we present a deep learning network to improve the model interpretability, which consists three main modules: deep feature extraction, visual saliency module and semantic description module. Visual and semantic interpretation jointly employed to provide cataract-grade oriented interpretation for the overall model. Experimental results on real clinical data set show that our method improves the interpretability for cataract grading while ensuring the high classification performance.