Serological tests play an essential role in monitoring and combating the COVID-19 pandemic. Recombinant spike protein (S protein), especially the S1 protein, is one of the major reagents used for serological tests. However, the high cost of S protein production and possible cross-reactivity with other human coronaviruses pose unavoidable challenges. By taking advantage of a peptide microarray with full spike protein coverage, we analyzed 2,434 sera from 858 COVID-19 patients, 63 asymptomatic patients and 610 controls collected from multiple clinical centers. Based on the results, we identified several S protein-derived 12-mer peptides that have high diagnostic performance. In particular, for monitoring the IgG response, one peptide (aa 1148-1159 or S2-78) exhibited a sensitivity (95.5%, 95% CI 93.7-96.9%) and specificity (96.7%, 95% CI 94.8-98.0%) comparable to those of the S1 protein for the detection of both symptomatic and asymptomatic COVID-19 cases. Furthermore, the diagnostic performance of the S2-78 (aa 1148-1159) IgG was successfully validated by ELISA in an independent sample cohort. A panel of four peptides, S1-93 (aa 553-564), S1-97 (aa 577-588), S1-101 (aa 601-612) and S1-105 (aa 625-636), that likely will avoid potential cross-reactivity with sera from patients infected by other coronaviruses was constructed. The peptides identified in this study may be applied independently or in combination with the S1 protein for accurate, affordable, and accessible COVID-19 diagnosis.