Geodesic voting method is known as a powerful tool for extracting curvilinear structures, which is able to find a tree structure from a single point. However, this method may fail to generate accurate results in complex scenarios such as complex network-like structures, due to the limitation of single source point. In order to solve this problem, we propose an adaptive curvature-penalized geodesic voting method where multiple source points with geometric voting constraint can be used for constructing the voting score map. In addition, we exploit the introduced adaptive geodesic voting method for the task of retinal vessel tracking, in conjunction with a deep learning-based junction points detection procedure. Experimental results on both synthetic images and retinal images prove the efficiency of the introduced adaptive geodesic voting method.