Digital Twin-Based Urban Perception Modeling Through Real-Time Social Sensing and Multimodal Artificial Intelligence

Authors

  • Sathvik Gaba Department of Computer Science and Engineering, University of Nevada, Reno, Reno, NV, USA. Author

Keywords:

digital twins; urban perception; social sensing; multimodal learning; uncertainty quantification; spatiotemporal modeling

Abstract

Urban perception, the affective and evaluative character of streetscapes and public places, is central to planning, mobility, and human-centered urban analytics, yet it remains difficult to model because social media signals are sparse and biased, visual evidence is incomplete, and perceptions change over time. This manuscript develops a digital twin-based methodological framework for real-time urban perception modeling that integrates geotagged social sensing with multimodal artificial intelligence. The digital twin is formulated as a probabilistic spatiotemporal state layer that maintains estimates of perceived safety, comfort, liveliness, and related dimensions, together with their uncertainty, over street segments and places. Social posts are treated as biased observations of this latent state rather than direct measurements. Visual, textual, geospatial, and temporal encoders are fused through reliability-weighted attention, while a graph regularizer encodes spatial autocorrelation and street-network dependence. A calibration-aware learning objective separates model uncertainty from social sampling bias, and a sequential assimilation rule updates the twin as new streams arrive. The framework yields explicit decision rules for identifying perception deficits under uncertainty and provides diagnostic criteria for modality failure, temporal lag, and feedback contamination. The main contribution is not a single predictive architecture but an integrated representation, inference, and update logic that makes urban perception estimable under realistic data constraints. The analysis clarifies how multimodal fusion, spatial regularization, and uncertainty calibration interact, and it identifies implementation trade-offs for municipal digital twins, participatory sensing platforms, and human-centered urban analytics.

References

1. Batty, M. (2018). Digital twins. Environment and Planning B: Urban Analytics and City Science, 45(5), 817–820.

2. Kitchin, R. (2014). The real-time city? Big data and smart urbanism. GeoJournal, 79(1), 1–14.

3. Goodchild, M. F. (2007). Citizens as sensors: The world of volunteered geography. GeoJournal, 69(4), 211–221.

4. Liu, Y., Liu, X., Gao, S., Gong, L., Kang, C., Zhi, Y., Chi, G., & Shi, L. (2015). Social sensing: A new approach to understanding our socioeconomic environments. Annals of the Association of American Geographers, 105(3), 512–530.

5. Aman, J., Matisziw, T. C., Kim, J. B., & Luo, D. (2022, April). Sensing the city: Leveraging geotagged social media posts and street view imagery to model urban streetscapes using deep neural networks. In Proceedings of the International Conference on Computer-Aided Architectural Design Research in Asia (pp. 595-604). Association for Computer-Aided Architectural Design Research in Asia.

6. Salesses, P., Schechtner, K., & Hidalgo, C. A. (2013). The collaborative image of the city: Mapping the inequality of urban perception. PLOS ONE, 8(7), e68400.

7. Dubey, A., Naik, N., Parikh, D., Raskar, R., & Hidalgo, C. A. (2016). Deep learning the city: Quantifying urban perception at a global scale. In Computer Vision – ECCV 2016 (pp. 196–212). Springer.

8. Naik, N., Philipoom, J., Raskar, R., & Hidalgo, C. (2014). Streetscore: Predicting the perceived safety of one million streetscapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 779–785). IEEE.

9. Ordóñez, V., & Berg, T. L. (2014). Learning high-level judgments of urban perception. In Computer Vision – ECCV 2014 (pp. 494–510). Springer.

10. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30, pp. 5998–6008). Curran Associates.

11. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (Vol. 139, pp. 8748–8763). PMLR.

12. Baltrušaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443.

13. Tao, F., Zhang, H., Liu, A., & Nee, A. Y. C. (2019). Digital twin in industry: State-of-the-art. IEEE Transactions on Industrial Informatics, 15(4), 2405–2415.

14. Li, D., Yu, W., & Shao, Z. (2021). Smart city based on digital twins. Computational Urban Science, 1(1), Article 4.

15. Ruths, D., & Pfeffer, J. (2014). Social media for large studies of behavior. Science, 346(6213), 1063–1064.

16. Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019). Social data: Biases, methodological pitfalls, and ethical boundaries. Frontiers in Big Data, 2, Article 13.

17. Senaratne, H., Mobasheri, A., Ali, A. L., Capineri, C., & Haklay, M. (2017). A review of volunteered geographic information quality assessment methods. International Journal of Geographical Information Science, 31(1), 139–167.

18. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 1321–1330). PMLR.

19. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (Vol. 48, pp. 1050–1059). PMLR.

20. Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.

Downloads

Published

2026-10-16