Purpose To determine the impact of visual impairment (VI) on health-related quality of life (HRQoL) and to compare the health burden of VI in different areas in mainland China. Methods A cohort of 6830 people from rural villages and a cohort of 3251 people from an urban city were included to receive comprehensive ophthalmologic examinations and complete the European Quality of Life-5 Dimensions 3 Levels (EQ-5D-3L) questionnaire. For urban and rural populations, a unified VI grouping standard was adopted: the eyes were classified into normal group, mild-moderate group, and severe group according to WHO standards, and then divided into 6 groups considering both eyes. We estimated the effects of VI on the EQ-5D index score using linear regression models and the association between VI and self-reported EQ-5D health problems using logistic regression models. Associations were assessed by the Spearman correlation coefficient. Results The prevalence of VI and the index scores of EQ-5D-3L for each subgroup of VI were higher for the rural cohort. In these two cohorts, the severity of VI in rural population (Spearman r = 0.205; p < 0.0001) and urban population (Spearman r = 0.164; p < 0.0001) is correlated with the EQ-5D index score. In the rural cohort, the difference in index scores with bilateral severe VI compared to those without VI, after adjusting for covariates, was - 0.053 for the rural cohort and - 0.084 for the urban cohort, respectively. In the rural cohort, the odds ratio for bilateral severe VI was 4.39 for mobility, 6.33 for self-care, and 5.88 for usual activities. The incidence of anxiety or depression and pain or discomfort in the urban cohort was greater; the OR for bilateral severe VI in the urban cohort was 4.75. Conclusions VI has a negative impact on HRQoL in the rural and urban areas of China, especially in urban population. It is also more likely to cause anxiety or depression among the urban cohort, which deserves special attention.