Transfer Learning for Robust Neural Speech Decoding Across Subjects Using Uncertainty-Aware Representations and Adaptive Calibration
Keywords:
neural speech decoding; transfer learning; uncertainty quantification; adaptive calibration; cross-subject generalization; selective predictionAbstract
Neural speech decoding has advanced rapidly, yet decoders trained on one participant often degrade when transferred to another because electrode geometry, neural dynamics, articulation strategies, and recording stability shift the input distribution and the mapping to speech units. This article develops a transfer-learning framework for cross-subject neural speech decoding that treats uncertainty-aware representation learning and adaptive calibration as coupled design requirements rather than post hoc corrections. The framework specifies a subject-conditioned latent decoder, decomposes predictive uncertainty into aleatoric and epistemic components, aligns second-order latent statistics across subjects with uncertainty-weighted penalties, and calibrates subject-specific output probabilities through constrained affine temperature and bias terms. A risk-controlled decision rule then converts calibrated probabilities and residual uncertainty into selective decoding decisions under an explicit error budget. The central claim is analytical: robust transfer in neural speech decoding requires preserving label-relevant subject variation while suppressing nuisance variation, and calibration must be learned as a transferable component because confidence, rather than accuracy alone, determines whether decoded speech can be acted upon. The framework also identifies failure modes, including negative transfer from over-alignment, unreliable uncertainty estimates under small target samples, and calibration guarantees that weaken when cross-subject exchangeability fails. The resulting design provides a technically grounded basis for cross-subject decoder evaluation and for integrating uncertainty into neural speech interfaces.
References
1. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. Clinical Neurophysiology, 113(6), 767-791.
2. Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019). Speech synthesis from neural decoding of spoken sentences. Nature, 568(7753), 493-498.
3. Moses, D. A., Metzger, S. L., Liu, J. R., Anumanchipalli, G. K., Makin, J. G., Sun, P. F., Chartier, J., Dougherty, M. E., Liu, P. M., Abrams, G. M., Tu-Chan, A., Ganguly, K., & Chang, E. F. (2021). Neuroprosthesis for decoding speech in a paralyzed person with anarthria. New England Journal of Medicine, 385(3), 217-227.
4. Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M., & Shenoy, K. V. (2021). High-performance brain-to-text communication via handwriting. Nature, 593(7858), 249-254.
5. Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., ... & Chang, E. F. (2023). A high-performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976), 1037-1046.
6. Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345-1359.
7. Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, M., ... & Lempitsky, V. (2016). Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59), 1-35.
8. Sun, B., Feng, J., & Saenko, K. (2016). Return of frustratingly easy domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), 2058-2065.
9. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 5574-5584.
10. Huang, J., Tostado-Marcos, P., Narasimha, S. M., Patel, A. N., Arneodo, E. M., Gentner, T. Q., ... & Gilja, V. (2024, July). Guiding brain-to-vocalization decoder design using structured generalization error. In 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (pp. 1-4). IEEE.
11. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 6402-6413.
12. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 70, 1321-1330.
13. Angelopoulos, A. N., & Bates, S. (2021). A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511.
14. Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015). Learning transferable features with deep adaptation networks. Proceedings of the 32nd International Conference on Machine Learning, 37, 97-105.
15. Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017). Adversarial discriminative domain adaptation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7167-7176.
16. Saito, K., Watanabe, K., Ushiku, Y., & Harada, T. (2018). Maximum classifier discrepancy for unsupervised domain adaptation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3723-3732.
17. Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114.
18. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the 33rd International Conference on Machine Learning, 48, 1050-1058.
19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998-6008.
20. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Intelligent Engineering Systems

This work is licensed under a Creative Commons Attribution 4.0 International License.