机构:[1]Beihang Univ, Sch Biol Sci & Med Engn, Beijing 100083, Peoples R China[2]Nagoya Univ, Grad Sch Informat, Nagoya 4640823, Japan[3]Capital Med Univ, Beijing TongRen Hosp, Dept Otolaryngol Head & Neck Surg, Beijing 100730, Peoples R China临床科室耳鼻咽喉-头颈外科首都医科大学附属北京同仁医院首都医科大学附属同仁医院[4]Nagoya Univ, Informat Technol Ctr, Nagoya 4640823, Japan
The loss of speech function due to conditions such as laryngectomy and vocal cord paralysis significantly impacts the quality of life for patients. Achieving effective communication for these patients is a goal pursued by researchers. This study primarily explores a method for reconstructing Mandarin speech based on voice-related neck and facial surface electromyography (sEMG). Neck and facial sEMG signals and speech waveform were synchronously collected during normal speech production. A speech reconstruction model for Mandarin speech, based on multi-scale feature extraction from EMG and a generative adversarial network (GAN), was developed. Both subjective and objective evaluations were conducted to assess the speech reconstruction performance of the model. The evaluation results indicate that the model effectively reconstructs speech from neck and facial sEMG signals. The reconstructed speech closely matches the original in terms of spectrogram and fundamental frequency, with mel-cepstrum distortion of 8.45 dB, log F0 RMSE of 0.40, F0 correlation coefficient of 0.71 and F0 voiced/unvoiced estimation accuracy of 0.80. The character error rate of the reconstructed speech is 0.32, while the tone error rate is 0.26. The subjective listening test results show that the naturalness of the reconstructed speech is acceptable, with a mean opinion score greater than 3. This study demonstrates the potential of deep learning techniques in effectively reconstructing Mandarin speech from sEMG.
基金:
National Natural Science Foundation of China, China [12372299, 32071315]; JST CREST, Japan [JPMJCR19A3]
语种:
外文
WOS:
第一作者:
第一作者机构:[1]Beihang Univ, Sch Biol Sci & Med Engn, Beijing 100083, Peoples R China
通讯作者:
推荐引用方式(GB/T 7714):
Li Fengji,Shen Fei,Ma Ding,et al.Mandarin speech reconstruction from surface electromyography based on generative adversarial networks[J].MEDICINE IN NOVEL TECHNOLOGY AND DEVICES.2025,26:doi:10.1016/j.medntd.2025.100359.
APA:
Li, Fengji,Shen, Fei,Ma, Ding,Zhou, Jie,Wang, Li...&Niu, Haijun.(2025).Mandarin speech reconstruction from surface electromyography based on generative adversarial networks.MEDICINE IN NOVEL TECHNOLOGY AND DEVICES,26,
MLA:
Li, Fengji,et al."Mandarin speech reconstruction from surface electromyography based on generative adversarial networks".MEDICINE IN NOVEL TECHNOLOGY AND DEVICES 26.(2025)