Background: Perceptual and electrophysiological studies have found reduced speech discrimination in quiet and noisy environment, delayed neural timing, decreased neural synchrony, and decreased temporal processing ability in elderlies, even those with normal hearing. However, recent studies have also demonstrated that language experience and auditory training enhance the temporal dynamics of sound encoding in the auditory brainstem response (ABR). The purpose of this study was to explore the pitch processing ability at the brainstem level in an aging population that has a tonal language background. Method: Mandarin speaking younger (n = 12) and older (n = 12) adults were recruited for this study. All participants had normal audiometric test results and normal suprathreshold click-evoked ABR. To record frequency following responses (FFRs) elicited by Mandarin lexical tones, two Mandarin Chinese syllables with different fundamental frequency pitch contours (Flat Tone and Falling Tone) were presented at 70 dB SPL. Fundamental frequencies (f0) of both the stimulus and the responses were extracted and compared to individual brainstem responses. Two indices were used to examine different aspects of pitch processing ability at the brainstem level: Pitch Strength and Pitch Correlation. Results: Lexical tone elicited FFR were overall weaker in the older adult group compared to their younger adult counterpart. Measured by Pitch Strength and Pitch Correlation, statistically significant group differences were only found when the tone with a falling f0 (Falling Tone) were used as the stimulus. Conclusion: Results of this study demonstrated that in a tonal language speaking population, pitch processing ability at the brainstem level of older adults are not as strong and robust as their younger counterparts. Findings of this study are consistent with previous reports on brainstem responses of older adults whose native language is English. On the other hand, lexical tone elicited FFRs have been shown to correlate with the length of language exposure. Older adults' degraded responses in our study may also be due to that, the Mandarin speaking older adults' long term exposure somewhat counteracted the negative impact on aging and helped maintain, or at least reduced, the degradation rate in their temporal processing capacity at the brainstem level.