Diabetic retinopathy (DR), the leading cause of blindness for working-age adults, is generally intervened by early screening to reduce vision loss. A series of automated deep-learning-based algorithms for DR screening have been proposed and achieved high sensitivity and specificity ( > 90%). However, these deep learning models do not perform well in clinical applications due to the limitations of the existing publicly available fundus image datasets. In order to evaluate these methods in clinical situations, we collected 13,673 fundus images from 9598 patients. These images were divided into six classes by seven graders according to image quality and DR level. Moreover, 757 images with DR were selected to annotate four types of DR-related lesions. Finally, we evaluated state-of-the-art deep learning algorithms on collected images, including image classification, semantic segmentation and object detection. Although we obtain an accuracy of 0.8284 for DR classification, these algorithms perform poorly on lesion segmentation and detection, indicating that lesion segmentation and detection are quite challenging. In summary, we are providing a new dataset named DDR for assessing deep learning models and further exploring the clinical applications, particularly for lesion recognition. (C) 2019 Elsevier Inc. All rights reserved.