Cataract is the most prevailing reason for blindness across the globe, which occupies about 4.2% population of the world. Even with the developments in visual sciences, fundus image-based diagnosis is deemed as a gold standard for cataract detection and grading. Though the increase in the workload of ophthalmologists and complexity of fundus images, the results may be subject to intelligence. Therefore, the development of an automatic method for cataract detection is necessary to prevent visual impairment and save medical resources. This paper aims to provide a novel hybrid convolutional and recurrent neural network (CRNN) for fundus image-based cataract classification. The proposed CRNN fuses the advantages of convolution neural network and recurrent neural network to preserve long- and short-term spatial correlation between the patches. Coupled with transfer learning, we adopt AlexNet, GoogLeNet, ResNet and VGGNet to extract multilevel feature representation and to analyse how well these models perform cataract classification. The results demonstrate that the proposed method outperforms state-of-the-art methods with an average accuracy of 0.9739 for four-class cataract classification and provides a compelling reason to be applied for other retinal diseases.