Adaptive optics scanning light ophthalmoscope (AO-SLO) can directly image the cone photoreceptor mosaic in the living human retina, which offers a potentially great tool to detect cone-related ocular pathologies by quantifying the changes in the cone mosaic. However, manual quantification is very time-consuming and automation is highly desirable. In this paper, we developed a fully automatic method based on multi-task learning to identify and quantify cone photoreceptors. By including cone edges in the labels as the third dimension of the classification, our method provided more accurate and reliable results than the two previously reported methods. We trained and validated our network in an open data set consisting of over 200,000 cones, and achieved a 99.20% true positive rate, 0.71% false positive rate, and 99.24% Dice's coefficient on the test set consisting of 44,634 cones. All are better than the reported methods. In addition, the reproducibility of all three methods was also tested and compared, and the result showed the performance of our method was generally closer to the gold standard. Bland-Altman plots show that our method was more stable and accurate than the other two methods. Then ablation experiment was further done, and the result shows that multi-task learning is essential to achieving accurate quantifications. Finally, our method was also extended to segment the cones to extract the size information. Overall, the method proposed here demonstrated great performance in terms of accuracy and reliability, which can be used to efficiently quantify the subtle changes associated with the progression of many diseases affecting cones.(c) 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement