Enhancing Art Engagement with Character-Driven Augmented Reality – Phase 2

Industry Partner:

Die Post

Description:

Swiss Post has a long-standing commitment to art and has been collecting contemporary art since 1924. Its collection now comprises more than 400 works of art of particular relevance to Switzerland and the Swiss population. However, despite the collection’s significance, making it accessible to the Swiss population remains challenging. The places where art can be displayed are limited and often inaccessible to the public.

To address this situation, the Swiss Post and the Game Technology Center (GTC) at ETH Zürich have engaged in a research collaboration to explore the use of augmented reality (AR) in combination with virtual characters to create novel and playful interactive experiences with the Swiss Post’s art collection. In 2024, the collaboration entered its second phase, building on the foundational work completed in the initial phase. This year’s efforts focused on refining and expanding the existing mobile application to create a more engaging and scalable user experience.

Refinement of Virtual Characters

A focus of Phase 2 was improving the quality and appeal of the virtual characters within the app. Building on the prototype developed in Phase 1, significant attention was paid to enhancing the realism and expressiveness of the characters and making them more engaging by reworking their appearance and improving the dialog system and voice synthesis.

Introduction of “Artwork of the Day”

Phase 2 introduced the “Artwork of the Day” mechanic. Drawing inspiration from daily reward systems in games and advent calendars, this feature replaces the previous ArtCards mechanic. The AR experience is now triggered directly on any surface, such as a table, without an ArtCard. Each day, the game characters present a new artwork and challenge the player with a quiz about it, making the experience more dynamic and engaging. The daily artwork feature presents users with new content every day, encouraging them to return regularly. This shift also reduced content production effort while enhancing user retention and interaction through shorter, more frequent sessions.

Exploration of AI-Driven Content Generation

To further reduce the burden on the Swiss Post’s curatorial team, Phase 2 also focused on exploring automated content creation using LLMs. An automated content pipeline was developed that includes data acquisition, data retrieval, question generation, translation into four languages, and dialog generation to automatically synthesize quiz questions and dialogs based on structured or unstructured information about the artworks and artists. This approach provides a scalable solution for content production and has shown promise in reducing the manual effort required to curate content for new artworks.

Evaluation

In December 2024, we will conduct an evaluation user study with both Post and ETH employees to collect feedback on the app and assess its functionality and user experience.

The project’s progress this year reflects the joint commitment of Swiss Post and the GTC to make art more accessible to the Swiss public through innovative digital experiences.