Hardware-Aware Acceleration of Secure Deep Learning Inference
Keywords:
secure inference; hardware acceleration; trusted execution environments; homomorphic encryption; privacy-preserving machine learning; confidential computing; systems architecture; governance; sustainabilityAbstract
Secure deep learning inference has emerged as a critical systems problem because machine learning services increasingly process sensitive biomedical, financial, industrial, and personal data whose exposure can create legal, ethical, and economic harm. The central challenge is not simply to design faster neural accelerators or stronger cryptographic protocols, but to align hardware architecture, cryptographic machinery, deployment infrastructure, and governance structures across the entire inference lifecycle. This paper examines hardware-aware acceleration of secure deep learning inference from a system-level perspective. It analyzes how trusted execution environments, homomorphic encryption, secure multiparty computation, differential privacy, and hybrid protocols interact with accelerators such as graphics processing units, field-programmable gate arrays, application-specific integrated circuits, and confidential computing platforms. Particular attention is given to structural trade-offs involving latency, memory bandwidth, energy consumption, cost, tenancy, robustness, fairness, and sustainability. The discussion also considers supply-chain governance, attestation, auditing, policy implications, and the uneven deployment realities of cloud, edge, and embedded environments. A central argument is that secure inference performance cannot be understood as a narrow optimization problem. Instead, it is a socio-technical design problem in which hardware capabilities, security assumptions, operational incentives, and regulatory expectations jointly determine what forms of private inference are technically feasible and socially legitimate. The paper concludes with a forward-looking research agenda for verifiable, efficient, equitable, and sustainable secure inference infrastructures.
References
1. Acar, A., Aksu, H., Uluagac, A. S., & Conti, M. (2018). A survey on homomorphic encryption schemes: Theory and implementation. ACM Computing Surveys, 51(4), 1-35. https://doi.org/10.1145/3214303
2. Juvekar, C., Vaikuntanathan, V., & Chandrakasan, A. (2018). GAZELLE: A low latency framework for secure neural network inference. In Proceedings of the 27th USENIX Security Symposium (pp. 1651-1669). USENIX Association.
3. Mishra, P., Lehmkuhl, R., Srinivasan, A., Zheng, W., & Popa, R. A. (2020). Delphi: A cryptographic inference service for neural networks. In Proceedings of the 29th USENIX Security Symposium (pp. 2505-2522). USENIX Association.
4. Liu, J., Juuti, M., Lu, Y., & Asokan, N. (2017). Oblivious neural network predictions via MiniONN transformations. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (pp. 619-631). ACM. https://doi.org/10.1145/3133956.3134056
5. Mohassel, P., & Zhang, Y. (2017). SecureML: A system for scalable privacy-preserving machine learning. In 2017 IEEE Symposium on Security and Privacy (pp. 19-38). IEEE. https://doi.org/10.1109/SP.2017.12
6. Riazi, M. S., Weinert, C., Tkachenko, O., Songhori, E. M., Schneider, T., & Koushanfar, F. (2018). Chameleon: A hybrid secure computation framework for machine learning applications. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security (pp. 707-721). ACM. https://doi.org/10.1145/3196494.3196522
7. Costan, V., & Devadas, S. (2016). Intel SGX explained. IACR Cryptology ePrint Archive, 2016, 86. https://eprint.iacr.org/2016/086
8. Sabt, M., Achemlal, M., & Bouabdallah, A. (2015). Trusted execution environment: What it is, and what it is not. In 2015 IEEE Trustcom/BigDataSE/ISPA (Vol. 1, pp. 57-64). IEEE. https://doi.org/10.1109/Trustcom.2015.357
9. Tramèr, F., & Boneh, D. (2019). Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. In International Conference on Learning Representations. OpenReview.
10. Bost, R., Popa, R. A., Tu, S., & Goldwasser, S. (2015). Machine learning classification over encrypted data. In Proceedings of the 2015 Network and Distributed System Security Symposium. Internet Society. https://doi.org/10.14722/ndss.2015.23241
11. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., ... & Zhao, S. (2021). Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2), 1-210. https://doi.org/10.1561/2200000083
12. Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., ... & Seth, K. (2017). Practical secure aggregation for privacy-preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (pp. 1175-1191). ACM. https://doi.org/10.1145/3133956.3133982
13. Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4), 211-407. https://doi.org/10.1561/0400000042
14. Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and machine learning: Limitations and opportunities. fairmlbook.org.
15. Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., ... & Barnes, P. (2020). Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 33-44). ACM. https://doi.org/10.1145/3351095.3372873
16. Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017). Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (pp. 3-18). IEEE. https://doi.org/10.1109/SP.2017.41
17. Sze, V., Chen, Y. H., Yang, T. J., & Emer, J. S. (2017). Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12), 2295-2329. https://doi.org/10.1109/JPROC.2017.2761740
18. Peng, H., Huang, S., Zhou, T., Luo, Y., Wang, C., Wang, Z., ... & Ding, C. (2023, October). Autorep: Automatic relu replacement for fast private network inference. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 5155-5165). IEEE.
19. Han, S., Pool, J., Tran, J., & Dally, W. J. (2015). Learning both weights and connections for efficient neural networks. Advances in Neural Information Processing Systems, 28, 1135-1143.
20. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., ... & Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2704-2713). IEEE.
21. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171-4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
22. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Intelligent Engineering Systems

This work is licensed under a Creative Commons Attribution 4.0 International License.