AdjustAR: AI-Driven In-Situ Adjustment of Site-Specific Augmented Reality Content

Nels Numan; Jessica Van Brummelen; Ziwen Lu; Anthony Steed

doi:10.1145/3746058.3758362

AdjustAR

AI-Driven In-Situ Adjustment of Site-Specific Augmented Reality Content

ACM UIST 2025 Poster

Nels Numan¹ Jessica Van Brummelen² Ziwen Lu¹ Anthony Steed¹

¹University College London ²Niantic Spatial, Inc.

This figure illustrates our system, CoCreatAR, being used by a user ex-situ on a desktop computer and another user in-situ on a mobile phone. It shows various meshes and visualizations augmented on the real-world location, including RGB-D point clouds, coarse meshes, and a virtual character representing the in-situ user on the ex-situ user's screen. For example, some coarse meshes are shown as rough planes outside of the original location mesh.

Abstract

Site-specific outdoor AR experiences are typically authored using static 3D models, but are deployed in physical environments that change over time. As a result, virtual content may become misaligned with its intended real-world referents, degrading user experience and compromising contextual interpretation. We present AdjustAR, a system that supports in-situ correction of AR content in dynamic environments using multimodal large language models (MLLMs). Given a composite image comprising the originally authored view and the current live user view from the same perspective, an MLLM detects contextual misalignments and proposes revised 2D placements for affected AR elements. These corrections are backprojected into 3D space to update the scene at runtime. By leveraging MLLMs for visual-semantic reasoning, this approach enables automated runtime corrections to maintain alignment with the authored intent as real-world target environments evolve.

Examples

Overview of early examples showcasing AdjustAR. From top left to bottom right: two birds already properly aligned with the fountain edge (AdjustAR does not perform adjustments); an arrow pointing to a first aid kit that has been moved (AdjustAR moves the arrow to the new location); a virtual sign pointing at a missing physical sign (AdjustAR displays a rendering of the environment at authoring time); arrows pointing at books, with some correctly and others incorrectly placed (AdjustAR moves the incorrectly placed ones).

Resources

Paper

BibTeX

If you find this work useful for your research, please cite:

@inproceedings{numanAdjustAR2025,
  title = {AdjustAR: {AI}-{Driven} {In}-{Situ} {Adjustment} of {Site}-{Specific} {Augmented} {Reality} {Content}},
  shorttitle = {AdjustAR},
  booktitle = {Adjunct Proceedings of the 38th {{Annual ACM Symposium}} on {{User Interface Software}} and {{Technology}}},
  author = {Numan, Nels and Van Brummelen, Jessica and Lu, Ziwen and Steed, Anthony},
  year = {2025},
  month = {sept},
  series = {{{UIST}} '25},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  doi = {10.1145/3746058.3758362},
}

Acknowledgements

This work was partially supported by the European Union's Horizon 2020 Research and Innovation program as part of project RISE under grant agreement No. 739578. We thank Simon Julier, Gabriel Brostow, and Niladri Dutt for valuable research discussions, and Dat Chu for assistance with testing logistics. The raccoon model used in this work was sourced from Google Poly.