Fine-Grained Prosody Transfer for Expressive AI-Generated Audiobooks and Digital Narration

Authors

  • Mariola Trost School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, OR, USA. Author
  • Tripti Pandit Department of Computer Science and Engineering, University at Buffalo, Buffalo, NY, USA. Author
  • Nela Dudko Department of Computer Science, University of North Texas, Denton, TX, USA. Author

Keywords:

prosody transfer, expressive speech synthesis, audiobook narration, style disentanglement, uncertainty modeling, neural text-to-speech

Abstract

AI-generated audiobooks and digital narration require more than intelligible speech. They require local prosodic decisions that convey emphasis, suspense, pacing, character voice, and discourse structure across long passages. Current neural text-to-speech systems can control global style, speaker identity, and approximate speaking rate, but they often blur the distinction between a reference narrator’s expressive timing and the target text’s lexical content. This manuscript develops a methodological framework for fine-grained prosody transfer in long-form narration. The framework represents prosody as a structured sequence of log F0, F0 change, phone duration, energy, and boundary cues; aligns reference prosody to target text through hierarchical cross-attention; and suppresses speaker-identity leakage with adversarial invariance while preserving linguistically licensed prominence and phrasing. An uncertainty-weighted objective prevents ambiguous or weakly aligned prosodic targets from dominating training, and constrained interpolation provides interpretable control over transfer strength. Analytical assessment addresses intelligibility protection, identity–prosody entanglement, long-range prosodic drift, and evaluation validity. The central argument is that fine-grained prosody transfer should be treated as constrained structured prediction rather than global style embedding. This framing clarifies model design, reduces predictable failure modes, and supports production workflows in which narration quality depends on local expressive decisions rather than acoustic fidelity alone.

References

1. Taylor, P. (2009). Text-to-speech synthesis. Cambridge University Press.

2. Skerry-Ryan, R. J., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R. J., Clark, R., & Saurous, R. A. (2018). Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron. In Proceedings of the 35th International Conference on Machine Learning (pp. 4693–4702). PMLR.

3. Wang, Y., Stanton, D., Zhang, Y., Ryan, R. J., Battenberg, E., Shor, J., Xiao, Y., Jia, Y., Ren, F., & Saurous, R. A. (2018). Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In Proceedings of the 35th International Conference on Machine Learning (pp. 5180–5189). PMLR.

4. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T. Y. (2019). FastSpeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems (Vol. 32, pp. 3171–3180).

5. Kim, J., Kong, J., & Son, J. (2021). Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proceedings of the 38th International Conference on Machine Learning (pp. 5530–5540). PMLR.

6. Zhang, Y., Huang, R., Li, R., He, J., Xia, Y., Chen, F., ... & Zhao, Z. (2024, March). Stylesinger: Style transfer for out-of-domain singing voice synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 17, pp. 19597-19605).

7. Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerry-Ryan, R. J., Saurous, R. A., Agiomyrgiannakis, Y., & Wu, Y. (2018). Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4779–4783). IEEE.

8. Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in Neural Information Processing Systems (Vol. 33, pp. 17022–17033).

9. Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., André, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., & Truong, K. P. (2016). The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2), 190–202.

10. Cutler, A., Dahan, D., & van Donselaar, W. (1997). Prosody in the comprehension of spoken language: A literature review. Language and Speech, 40(2), 141–201.

11. Hirschberg, J. (2002). Communication and prosody: Functional aspects of prosody. Speech Communication, 36(1–2), 31–43.

12. Wagner, P., & Watson, D. G. (2010). Experimental and theoretical advances in prosody: A review. Language and Cognitive Processes, 25(7–9), 905–945.

13. Kameoka, H., Kaneko, T., Tanaka, K., & Hojo, N. (2018). StarGAN-VC: Non-parallel many-to-many voice conversion using star generative adversarial networks. In 2018 IEEE Spoken Language Technology Workshop (SLT) (pp. 266–273). IEEE.

14. Zhou, K., Sisman, B., Liu, R., & Li, H. (2022). Emotional voice conversion: Theory, databases and ES-1. Speech Communication, 137, 1–18.

15. Oord, A. van den, Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.

16. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30, pp. 5998–6008).

17. Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., Wang, X., Leng, Y., Yi, Y., He, L., Soong, F., Qin, T., Zhao, S., & Liu, T. Y. (2022). NaturalSpeech: End-to-end text to speech synthesis with human-level quality. arXiv preprint arXiv:2205.04421.

18. Casanova, E., Weber, J., Shulby, C. D., Junior, A. C., Gölge, E., & Ponti, M. A. (2022). YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In Proceedings of the 39th International Conference on Machine Learning (pp. 2709–2720). PMLR.

Downloads

Published

2026-10-16