Multi-Task Learning for Simultaneous Anatomical Segmentation and Lesion Localization in Three-Dimensional Medical Imaging

Authors

  • Enzio Lattuada Department of Computer Science and Engineering, University of Nevada, Reno, Reno, NV, USA. Author

Keywords:

multi-task learning; three-dimensional medical imaging; anatomical segmentation; lesion localization; uncertainty weighting; volumetric deep learning

Abstract

Three-dimensional medical imaging supports diagnosis, treatment planning, and disease monitoring, yet anatomical segmentation and lesion localization are often treated as separate computational tasks. This separation creates redundant feature extraction, inconsistent spatial reasoning, and evaluation procedures that obscure whether one task benefits or harms the other. This manuscript develops a technically grounded multi-task learning framework for simultaneous anatomical segmentation and lesion localization in volumetric images. The central contribution is a coupled formulation in which a shared three-dimensional encoder supports a dense anatomical segmentation decoder and a sparse lesion heatmap decoder, while a segmentation-conditioned inference rule reweights lesion evidence by anatomical plausibility without imposing a hard mask. The framework specifies volumetric inputs and outputs, task-specific losses for class-imbalanced segmentation and point-supervised lesion localization, uncertainty-based loss balancing, and an explicit analysis of task conflict through gradient direction. It also defines a separate but coordinated evaluation logic that distinguishes anatomical overlap, boundary accuracy, lesion detection sensitivity, false positives per volume, and localization distance. The analysis identifies a key asymmetry: segmentation errors can attenuate lesion evidence near anatomical boundaries, whereas lesion sparsity and label ambiguity can weaken shared representations if localization gradients are allowed to dominate. These mechanisms imply that coupling is most defensible when it is soft, uncertainty-aware, and evaluated task by task. The formulation provides a reproducible methodological basis for joint volumetric analysis and clarifies the conditions under which shared learning is technically preferable to independent pipelines.

References

1. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 (pp. 234–241). Springer. https://doi.org/10.1007/978-3-319-24574-4_28

2. Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (2016). 3D U-Net: Learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016 (pp. 424–432). Springer. https://doi.org/10.1007/978-3-319-46723-8_49

3. Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2), 203–211. https://doi.org/10.1038/s41592-020-01008-z

4. Caruana, R. (1997). Multitask learning. Machine Learning, 28(1), 41–75. https://doi.org/10.1023/A:1007379606734

5. Kendall, A., Gal, Y., & Cipolla, R. (2018). Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7482–7491). IEEE. https://doi.org/10.1109/CVPR.2018.00781

6. Misra, I., Shrivastava, A., Gupta, A., & Hebert, M. (2016). Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3994–4003). IEEE. https://doi.org/10.1109/CVPR.2016.433

7. Sener, O., & Koltun, V. (2018). Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems (Vol. 31, pp. 527–538). Curran Associates.

8. Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60–88. https://doi.org/10.1016/j.media.2017.07.005

9. Kohl, S. A. A., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K. H., Eslami, S. M. A., Rezende, D. J., & Ronneberger, O. (2018). A probabilistic U-Net for segmentation of ambiguous images. In Advances in Neural Information Processing Systems (Vol. 31, pp. 6965–6975). Curran Associates.

10. Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S., & Cardoso, M. J. (2017). Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (pp. 240–248). Springer. https://doi.org/10.1007/978-3-319-67558-9_28

11. Milletari, F., Navab, N., & Ahmadi, S.-A. (2016). V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In Proceedings of the Fourth International Conference on 3D Vision (pp. 565–571). IEEE. https://doi.org/10.1109/3DV.2016.79

12. Taha, A. A., & Hanbury, A. (2015). Metrics for evaluating 3D medical image segmentation: Analysis, selection, and tool. BMC Medical Imaging, 15, 29. https://doi.org/10.1186/s12880-015-0068-x

13. Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E., Chartrand, G., Lohofer, F., Holch, J. W., Sommer, W., Hofmann, F., Hostettler, A., Lev-Cohain, N., Drozdzal, M., Amitai, M. M., Sosna, J., ... & Menze, B. (2023). The liver tumor segmentation benchmark (LiTS). Medical Image Analysis, 84, 102680. https://doi.org/10.1016/j.media.2022.102680

14. Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., Lanczi, L., Gerstner, E., Weber, M.-A., Arbel, T., Avants, B. B., Ayache, N., Buendia, P., Collins, D. L., Cordier, N., ... & Van Leemput, K. (2015). The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10), 1993–2024. https://doi.org/10.1109/TMI.2014.2377694

15. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (PMLR Vol. 48, pp. 1050–1059). PMLR.

16. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (Vol. 30, pp. 6402–6413). Curran Associates.

17. Liu, S., Jin, J., Li, W., & Yan, X. (2026). Segmentation-Guided Semi-Supervised Landmark Detection on Intraoral 3D Scans. Preprints. https://doi.org/10.20944/preprints202609.1722.v1

Downloads

Published

2026-11-14