Zero-Shot Multilingual Singing Voice Conversion Using Disentangled Speaker and Style Representations

Authors

  • Alister L. Neelin Department of Computer Science, Binghamton University, Binghamton, NY, USA. Author
  • Salvador Sacristan Department of Computer Science, University of Houston, Houston, TX, USA. Author
  • Annemarie Kostolzin School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, OR, USA. Author
  • Qushi L. Bhattacharyya Department of Electrical Engineering and Computer Science, University of Kansas, Lawrence, KS, USA. Author

Keywords:

zero-shot singing voice conversion; speaker disentanglement; style representation; multilingual speech; self-supervised learning; neural audio synthesis

Abstract

Zero-shot singing voice conversion aims to transform a source singing recording into the voice of an unseen target singer while preserving melodic, linguistic, and expressive content. The problem becomes substantially harder in multilingual settings, where source lyrics, target reference material, and singing conventions may differ across languages. A central obstacle is that speaker identity and singing style are statistically entangled: a short target reference may contain vibrato, dynamics, register, pronunciation habits, and recording conditions that are not intrinsic to the singer. This article develops a methodological framework for zero-shot multilingual singing voice conversion that factorizes a singing signal into language-agnostic content, speaker identity, singing style, and melody. The framework combines self-supervised content encoding, a reference-conditioned speaker encoder, a conditional style encoder, adversarial and correlation-based disentanglement penalties, and explicit melody preservation. Training objectives are formulated for reconstruction, content consistency, melody accuracy, and speaker-style independence, and an inference rule is given for controlling style transfer through latent interpolation. The analysis addresses cross-lingual generalization, style-speaker confounding, pitch-range mismatch, error propagation through neural vocoding, and evaluation logic. The contribution is a technically explicit design and analytical assessment that connects disentangled representation learning to the practical requirements of zero-shot multilingual singing voice conversion.

References

1. Chou, J.-C., Yeh, C.-C., & Lee, H.-Y. (2019). One-shot voice conversion by separating speaker and content representations with instance normalization. In Proceedings of Interspeech 2019 (pp. 664–668).

2. Kameoka, H., Kaneko, T., Tanaka, K., & Hojo, N. (2018). StarGAN-VC: Non-parallel many-to-many voice conversion using star generative adversarial networks. In 2018 IEEE Spoken Language Technology Workshop (SLT) (pp. 266–273). IEEE.

3. Qian, K., Zhang, Y., Chang, S., Yang, X., & Hasegawa-Johnson, M. (2019). AutoVC: Zero-shot voice style transfer with only autoencoder loss. In Proceedings of the 36th International Conference on Machine Learning (pp. 5210–5219). PMLR.

4. Kaneko, T., & Kameoka, H. (2018). CycleGAN-VC: Non-parallel voice conversion using cycle-consistent adversarial networks. In 2018 26th European Signal Processing Conference (EUSIPCO) (pp. 2100–2104). IEEE.

5. Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021). HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3451–3460.

6. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (Vol. 33, pp. 12449–12460).

7. Défossez, A., Copet, J., Synnaeve, G., & Adi, Y. (2022). High fidelity neural audio compression. arXiv preprint arXiv:2210.13438.

8. Skerry-Ryan, R. J., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R., Clark, R., & Saurous, R. A. (2018). Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron. In Proceedings of the 35th International Conference on Machine Learning (pp. 4693–4702). PMLR.

9. Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R. J., Battenberg, E., Shor, J., Xiao, Y., Ren, F., Jia, Y., & Saurous, R. A. (2018). Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In Proceedings of the 35th International Conference on Machine Learning (pp. 5180–5189). PMLR.

10. Choi, H., Lee, J., Kim, W., Lee, J., Heo, H., & Lee, K. (2021). Neural analysis and synthesis: Reconstructing speech from self-supervised representations. In Advances in Neural Information Processing Systems (Vol. 34, pp. 16251–16265).

11. Zhang, Y., Jiang, Z., Li, R., Pan, C., He, J., Huang, R., ... & Zhao, Z. (2024, November). Tcsinger: Zero-shot singing voice synthesis with style transfer and multi-level style control. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 1960-1975).

12. Kobayashi, K., Toda, T., Neubig, G., Sakti, S., & Nakamura, S. (2014). Statistical singing voice conversion with direct waveform modification based on the spectrum differential. In Proceedings of Interspeech 2014 (pp. 2514–2518).

13. Sisman, B., Zhang, M., Dong, M., & Li, H. (2020). On the study of generative adversarial networks for singing voice conversion. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7234–7238). IEEE.

14. Zhao, Y., Huang, W.-C., Tian, X., Yamagishi, J., Das, R. K., Kinnunen, T., Ling, Z., & Toda, T. (2020). Voice conversion challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion. In Proceedings of the Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020 (pp. 80–98).

15. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30, pp. 5998–6008).

16. Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. In International Conference on Learning Representations.

17. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems (Vol. 27, pp. 2672–2680).

18. van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., van den Driessche, G., Lockhart, E., Sifre, L., & Hassabis, D. (2018). Parallel WaveNet: Fast high-fidelity speech synthesis. In Proceedings of the 35th International Conference on Machine Learning (pp. 3918–3926). PMLR.

19. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., & Lerchner, A. (2017). beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.

20. Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. arXiv preprint arXiv:1701.07875.

Downloads

Published

2026-10-22