[2024]
An audio AR system for context-inspired storytelling
Video link: https://youtu.be/p4Ckd2ibXwI
We present Stories Everywhere (StEve), a mobile Augmented Reality (AR) system designed to generate immersive, surreal audio narratives contextualized by the user’s surroundings.
The system captures environmental data—including images of the surroundings, weather, and time—which is processed through a pipeline combining a Visual Question Answering, large language and text-to-speech model. This enables dynamic storytelling that integrates real-world elements into narratives delivered to the user whilst walking.
We also introduce a preliminary exploration of whether our immersive AI-generated stories can maintain environmental awareness while being engaging. A prototype iOS application was developed and evaluated through small-scale situational tests and a case study.
Results indicate that the system reliably integrates environmental cues into captivating audio stories, with participants reporting enjoyment and successful recognition of contextual elements. However, latency, connectivity, and model performance remain challenges for real-time immersion.
This work demonstrates the feasibility of context-aware AR storytelling as a tool for enhancing attentiveness while maintaining entertainment value, and it outlines technical, experiential, and methodological directions for future research and refinement.