AI-Assisted Vocal Performance Assessment Through Style Recognition and Fine-Grained Acoustic Feature Analysis
Keywords:
vocal performance assessment; style recognition; fine-grained acoustic features; uncertainty calibration; interpretable machine learning; singing voice analysisAbstract
Vocal performance assessment is central to music pedagogy, auditioning, and production, yet automated systems often reduce evaluation to pitch and rhythmic accuracy while neglecting style, timbre, and expressive technique. This reduction creates a mismatch between machine scores and expert judgment, especially when legitimate stylistic choices are interpreted as errors. The manuscript develops a methodological framework for AI-assisted vocal performance assessment that couples style recognition with fine-grained acoustic feature analysis. The framework extracts frame-level descriptors of pitch, loudness, spectral shape, harmonicity, phonation, and onset behavior; aggregates them into performance-level summaries; infers a probabilistic style posterior; and compares observed descriptors with style-conditioned reference expectations. These deviations are mapped to interpretable rubric dimensions through monotone, parameterized functions, and the resulting dimension scores are fused with an uncertainty-aware decision rule. A central insight is that style should not be treated as a post hoc label but as a conditioning variable that changes the interpretation of acoustic deviations. A second insight is that reliability reporting must accompany scoring because style ambiguity and descriptor instability can otherwise produce overconfident judgments. The framework specifies calibration, regularization, and boundary conditions without assuming the availability of a particular corpus or benchmark. Its significance lies in providing an auditable route from acoustic evidence to feedback, which can support transparent assessment, targeted vocal coaching, and more defensible human-machine collaboration in performance evaluation.
References
1. Sundberg, J. (1987). The science of the singing voice. Northern Illinois University Press.
2. Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., André, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., & Truong, K. P. (2016). The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2), 190–202.
3. McFee, B., Raffel, C., Liang, D., Ellis, D. P. W., McVicar, M., Battenberg, E., & Nieto, O. (2015). librosa: Audio and music signal analysis in Python. In Proceedings of the 14th Python in Science Conference (pp. 18–25). SciPy.
4. Salamon, J., & Gómez, E. (2012). Melody extraction from polyphonic music signals using pitch contour characteristics. IEEE Transactions on Audio, Speech, and Language Processing, 20(6), 1759–1770.
5. Caclin, A., McAdams, S., Smith, B. K., & Winsberg, S. (2005). Acoustic correlates of timbre space dimensions: A confirmatory study using synthetic tones. Journal of the Acoustical Society of America, 118(1), 471–482.
6. Zhang, Y., Jiang, Z., Li, R., Pan, C., He, J., Huang, R., ... & Zhao, Z. (2024, November). Tcsinger: Zero-shot singing voice synthesis with style transfer and multi-level style control. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 1960-1975).
7. Kim, J. W., Salamon, J., Li, P., & Bello, J. P. (2018). CREPE: A convolutional representation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 161–165). IEEE.
8. Titze, I. R. (1994). Principles of voice production. Prentice Hall.
9. Thompson, S., & Williamon, A. (2003). Evaluating evaluation: Musical performance assessment as a research tool. Music Perception, 21(1), 21–41.
10. Juslin, P. N. (2000). Cue utilization in communication of emotion in music performance: Relating performance to perception. Journal of Experimental Psychology: Human Perception and Performance, 26(6), 1797–1813.
11. Pearce, M. T., & Wiggins, G. A. (2012). Auditory expectation: The information dynamics of music perception and cognition. Topics in Cognitive Science, 4(4), 625–652.
12. Blaauw, M., & Bonada, J. (2017). A neural parametric singing synthesizer modeling timbre and expression from natural songs. Applied Sciences, 7(12), 1313.
13. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30 (pp. 6402–6413). Curran Associates.
14. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 1321–1330). PMLR.
15. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (pp. 1050–1059). PMLR.
16. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).
17. Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427–437.
18. Eyben, F., Weninger, F., Gross, F., & Schuller, B. (2013). Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In Proceedings of the 21st ACM International Conference on Multimedia (pp. 835–838). ACM.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Intelligent Engineering Systems

This work is licensed under a Creative Commons Attribution 4.0 International License.