Retinal diseases such as age-related macular degeneration and diabetic macular edema will lead to irreversible blindness without timely diagnosis and treatment. Optical coherence tomography (OCT) has been widely utilized to detect retinal diseases because of its non-contact and non-invasive imaging peculiarities. Due to the lack of ophthalmic medical resources, automatic analyzing and diagnosing retinal OCT images is necessary with computer-aided diagnosis algorithms. In this study, we propose a lightweight retinal OCT image classification model integrating convolutional neural network (CNN) and Transformer to classify various retinal diseases with few parameters of the model. Local lesion features extracted by CNN can be encoded with the whole OCT image through the Transformer, which improves the classification ability. A convolutional block attention module is also integrated into our model to enhance the representational power. Compared with several classical models, our model achieves the best accuracy of 0.9800 and recall of 0.9799 with the least number of parameters and prediction time for an image on the OCT-C8 dataset. Moreover, on the OCT2017 dataset, our model outperforms the four state-of-the-art models except almost equal to another, achieving an average accuracy, precision, recall, specificity and F1-score of 0.9985, 0.9970, 0.9970, 0.9990, and 0.9970. Simultaneously, the number of parameters of our model has been reduced to just 1.28 M, and the average prediction time for an image is only 2.5 ms.