Clinical lesions progress continuously but previous grading strategies are not fine-grained enough to model the continuously changing features of lesions. For lack of temporal sequential medical data to provide lesion progression information, we propose to use the severity ranking of disease lesions as spatial ranking label to represent temporal progression. Absolute ranking labels and relative ranking labels are calculated from severity ranking of datasets. A two-branch framework with spatial-temporal feature encoder is designed which using ranking labels to exploit the ranking relation between query and reference images. Furthermore, ranking loss is designed to enforce that sample features should be distributed in the feature space based on ranking scores. Our model achieves five-grade accuracy of 0.9204 on myopic maculopathy dataset. Compared with discrete grading, great improvement for automatic diagnosis is achieved. Experiments on B-mode fatty liver ultrasound dataset and glaucoma dataset also show generality of our algorithm.