Cross-Lingual Singing Voice Synthesis Using Multilingual Acoustic Modeling and Deep Learning
Keywords:
cross-lingual singing voice synthesis; multilingual acoustic modeling; score-conditioned attention; language-adversarial training; neural vocoding; singing corpusAbstract
Cross-lingual singing voice synthesis aims to generate a sung performance in a target language from a musical score and lyrics while preserving melody, rhythm, singer identity, and expressive detail. Most neural singing systems are trained within a single language and entangle language-specific phonotactics with acoustic modeling, so pronunciation transfer can cause phoneme substitution errors, duration drift, pitch instability, and timbre degradation. This paper develops a methodological framework based on multilingual acoustic modeling and deep learning. A score encoder represents pitch, duration, and expression; a language-conditioned phoneme encoder maps lyrics into a shared articulatory space; a transformer backbone predicts acoustic frames; and an adversarial language classifier encourages language-invariant hidden representations while preserving singer-dependent timbre. Cross-lingual alignment is handled by monotonic attention constrained by score timing, and duration and pitch predictors provide explicit prosodic conditioning. Objective functions combine spectral reconstruction, duration, pitch, alignment, and language-invariance terms. The analysis explains why shared acoustic representations can reduce negative transfer across languages but also why they risk collapsing phonemic contrasts that are meaningful in only one language. It discusses boundary conditions involving unseen phonemes, tone languages, low-resource languages, singer leakage, and vocoder artifacts. The contribution is a coherent design rationale and analytical account rather than a claim of empirical superiority. The framework offers a technically grounded basis for building multilingual singing systems, organizing future corpora, and evaluating cross-lingual transfer with metrics that separate pronunciation accuracy from musical and timbral fidelity.
References
1. Nishimura, M., Hashimoto, K., Oura, K., Nankaku, Y., & Tokuda, K. (2016). Singing voice synthesis based on deep neural networks. In Proceedings of Interspeech (pp. 2478–2482). ISCA.
2. Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., ... & Saurous, R. A. (2017). Tacotron: Towards end-to-end speech synthesis. In Proceedings of Interspeech (pp. 4006–4010). ISCA.
3. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T. Y. (2019). FastSpeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems (Vol. 32). Curran Associates.
4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30, pp. 5998–6008). Curran Associates.
5. Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in Neural Information Processing Systems (Vol. 33, pp. 17022–17033). Curran Associates.
6. Lu, P., Wu, Y., & Lian, J. (2020). XiaoiceSing: A high-quality and integrated singing voice synthesis system. In Proceedings of Interspeech (pp. 1306–1310). ISCA.
7. Ren, Y., Tan, X., Qin, T., Luan, J., Zhao, S., Zhao, Z., & Liu, T. Y. (2020). DeepSinger: Singing voice synthesis with data mined from the web. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 1979–1989). ACM.
8. Liu, J., Li, C., Ren, Y., Chen, F., & Zhao, Z. (2022). DiffSinger: Singing voice synthesis via shallow diffusion mechanism. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 11020–11028.
9. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., ... & Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8440–8451). ACL.
10. Zhang, Y., Pan, C., Guo, W., Li, R., Zhu, Z., Wang, J., ... & Zhao, Z. (2024). Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37, 1117-1140.
11. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations. ICLR.
12. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (pp. 1724–1734). ACL.
13. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (pp. 4171–4186). ACL.
14. Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., & Collobert, R. (2020). MLS: A large-scale multilingual dataset for speech research. In Proceedings of Interspeech (pp. 2757–2761). ISCA.
15. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., ... & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.
16. Huang, R., Chen, F., Ren, Y., Liu, J., Cui, C., & Zhao, Z. (2021). Multi-Singer: Fast multi-singer singing voice vocoder with a large-scale corpus. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 3945–3954). ACM.
17. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems (Vol. 27, pp. 2672–2680). Curran Associates.
18. Tan, X., Qin, T., Soong, F., & Liu, T. Y. (2021). A survey on neural speech synthesis. arXiv preprint arXiv:2106.15561.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Intelligent Engineering Systems

This work is licensed under a Creative Commons Attribution 4.0 International License.