Body Projection Mapping: Enhancing Connection at Hybrid Festivals

Body Projection Mapping: Enhancing Connection at Hybrid Festivals

Body Projection Mapping: Enhancing Connection at Hybrid Festivals

Date

Date

Date

03.2022 - 06.2022

03.2022 - 06.2022

03.2022 - 06.2022

Role

Role

Role

UX Research, Concept Development, Interaction Design

UX Research, Concept Development, Interaction Design

UX Research, Concept Development, Interaction Design

Stakeholder

Stakeholder

Stakeholder

Effenaar

Effenaar

Effenaar

Introduction

What makes a festival unforgettable? Sure, the music matters. But often it’s the unexpected moments — laughing with strangers, dancing in sync, discovering something together. When Effenaar asked us to rethink what a hybrid festival could be, we didn’t want to mimic the physical world online. We wanted to ask: what can digital do that physical can’t?

This project took place during the COVID-19 pandemic — a time when live events were limited, but the need for human connection felt stronger than ever. As a passionate live music lover, I saw this challenge as a chance to explore how technology could extend festival magic in new directions. Could we build a virtual experience that wasn’t just a livestream, but a space for genuine connection?

Phase 1: Reframing the Problem

We started by mapping the essence of festivals through the "flower method," identifying three anchors: festival, virtual, and interactive. We believed that hybrid festival is still a festival, therefore it should contain the feeling of (physical) festival. But the virtual part takes part in a hybrid festival, it is something that differ from physical festival. Lastly, we all agreed that interaction is one of most important aspect when it comes to a festival, after all it is not just music that brings people excitement, it is about people. 

In order to understand what makes festival exciting, we started another brainstorm section by listing what the expectations are during physical festival and how we could translate them into a virtual festival, as well as an user scenarios about people going to a physical festival. However, in this process, we realized that a physical festival and a virtual festival cannot be the same. Instead of thinking about how we could make people feel the same excitement, we should focus on how we could create an exciting virtual experiment. 

We have conducted 11 User interviews and quickly revealed that many people find virtual festivals isolating. They described livestreams as cold and lonely, with little human warmth. But when we asked why they wouldn't attend a physical festival, we discovered a new perspective:

  • "My friends can’t make it, and I don’t want to go alone."

  • "I get anxious in crowds."

  • "I had to miss it due to personal or physical limitations."

These responses helped us realize that a hybrid festival could be more than a replacement — it could be an opportunity. We weren’t designing for less-than; we were designing for different.

Phase 2: Client Alignment & Concept Directions

We held a joint interview with Effenaar and other student teams. Their feedback gave us space to experiment: they weren’t interested in gimmicky VR, but wanted bold, meaningful experiences. There was no set budget — just the desire for ideas with impact.

We proposed three early concepts:

  • A festival map with avatars and live chat, where physical and virtual attendees could interact.

  • Meet & greet booths where artists could hold live moderated calls with online fans.

  • A livestream interface with customizable chats and emoji reactions for each stage.

To sharpen our design thinking, we joined a workshop called Design for Humanity, which helped us anchor our concepts in the basic human need for connection. The outcome? A reframed design goal:

"Create presence, not just access."

We created story-based user journeys to better understand what that presence could look like. One example:

Janis, a festivalgoer who had excitedly bought her tickets months in advance, only to break both legs in a snowboarding accident just weeks before the event. Attending physically was impossible — but through our concept, she could still visit virtually, watch her favorite artists, connect with friends, and be part of the moment.

Phase 3: Cultural Research & Interaction Models

In order to find out what kind of interaction we want to recreate in virtual festival and whether it is technical feasible, I started my research on that.

My research focused on emergent digital interaction patterns — from global virtual concerts to playful community-driven events in China. Platforms like Bilibili and the viral success of cloud nightclubs like Xiugou inspired new ways of thinking about online presence.

This Xiugou nightclub, as shown below, opened at bilibili exploded at the beginning of 2022. In just two months after the live broadcast, the influencer "Xugou Nightclub Boss" has gained 600,000 followers, becoming the largest "cloud nightclub" in China. The gameplay is novel and simple. During the process, netizens can operate their own puppy images by sending bullet comment commands, and simply interact with the music to achieve "cloud dance". And the technology is relatively simply, by using unity.

Another function that I looked into is Bullet chat (danmu) on Bilibili. While chaotic, these systems fostered a shared experience that felt more like being together than watching alone.

During the process we have also discussed what kind of interaction that would be involved. We figured that there will be:

  • Online ↔ Online Interaction

  • Online ↔ Physical Interaction

  • Online ↔ Artist Interaction

Every group members would come up with ideas based on these aspects (by their choices) and all the research results. I have chosen online - artist part. So now we started our design phase.

Phase 4: Refining Concept

Based on the ideation phase, we know that we want to create something that will bring people together (physical visitors, online visitors and the artists). 

We have conducted interviews with artists, one of biggest frustration from the artists is the lack of feedback from audience during livestream. Unlike physical concerts, the artists can see how people react or give orders to audience.

With these aspects together, and the knowledge I learned from virtual club that is now very popular in China (technology), I came up with an idea that is a virtual platform that is able to let artists receive feedbacks from online visitors. Other than that, since Hit the City would be a hybrid festival, there will be physical visitors as well. We don't want people who join the festival online feel excluded when they feel the needs to express themselves. 

The second pitch happened in Effenaar together with some of Effenaar staffs. We represented our ideas and based on the feedbacks we decided to focus on body projection mapping.​

For body projection mapping, it was not an idea that first appeal to one of the Effenaar staffs. He is in charge with the equipments and he thoughts it would be a tiring work since people might have to move the projectors constantly even though he though it is a fancy idea. However, we explained that the current technology is able to detect the body figures therefore the images that are projected on human body is trackable. It is a funny realisation that sometimes the clients are not aware of the newest technology and as a creative technologist we need to inform them. It is a lesson for both parties.

Concept

"An interactive experience that allows interaction between the artist and the festival visitors."

Via Body Projection Mapping, we now call it Aura, the users can choose a sticker or a pattern to project on artists' body. These stickers are preselected by the artists or Effenaar. They can design their own patterns/stickers based on their albums, logos, or even about the hot topics they might want to raise. This gives a chance to reward online visitors and engage the interaction between the visitors and artists.

Implementation

​A thesis about real-time body tracking and projection mapping in the interactive arts explains the process like the following figure. However, it is relatively hard for us to learn from it as they made their own algorithm. We consulted Geert Jan. He informs us that we can use processing. There is already a library of body mask in processing, we can use it directly. So here are what we need:​

  • Kinect

  • Projector

  • Object Detection software

  • Processing

Iteration

​Version 1

Processing has examples for Kinect v2 for body tracking and body mask, as Geert Jan showed us. For the first version, we implemented that and now you can see there is a body mask that is your body shape that is detected by Kinect and it projects on your body!

Version 2

Now it is the time to put an image on user's body. 

Version 3

For the final version, we have removed/hidden the body mask so that there is only image that is visitable. 

Input system

The idea of body projection mapping is quite simple to understand, but how do online visitors interact with it? An input system is needed. I made a Figma interactive prototype to inform it.

The interactions that are involved, you can check the document for more detailed explanations: 

  • A drag and drop functionality that allows users to select the stickers to the desired location.

  • Body outline when dragging

  • Instruction for first time users

  • Waiting list

  • Limited spots

  • Timer

  • Pre-selected stickers

Online users selected designs via a drag-and-drop interface, placing them onto a live silhouette of the artist. The visual would then be mapped and projected in real time, visible both online and in the venue.

Final Phase: Testing, Reflection & Delivery

Results

During the observations, we have found out three common issues that happened during the users interacted with the input systems.

  1. Most of users would skip the instruction. We found out that they usually skipped the instruction and then started messing around the input system, and learning by doing it. Therefore, it caused some of the confusion. We asked for the reasons and they indicated that the text is relatively long and they did not have the patient.

  2. The users were not aware of the meaning of BPM. One of the reasons is because BPM usually stands for bear per minute. It is also because they skipped the instruction.

  3. No users noticed the waiting list and they did not know where to find out whether there are available spots for them to put the stickers. 1 user thought that the waiting list is the sticker itself where you can drag and drop directly.

  4. The button color isn’t very noticeable. The users didn’t know it’s clickable.

There are also some small details we noticed such as:

  1. We tested the input system with a laptop. we noticed that sometimes the distance between the stickers and available spots are far and the Touch Bar isn’t big enough to do it.

  2. Most users were way to focus on the stickers and did not notice the body outline (where to show the available spots). It is due to the distance as well, also because they skipped the instruction.

Based on the feedbacks, we have improved our prototype:

  1. Renamed the feature to Aura

  2. Redesigned the onboarding with shorter, clearer steps

  3. Improved button contrast

Onboarding before

Onboarding after

Button before vs after

Reflection

This was my first time designing across physical and digital experiences in such a layered way. I learned how to:

  • Translate human emotion into interactive systems

  • Rapidly test and adapt based on feedback

  • Combine research from different cultures and contexts

  • Advocate for inclusion in hybrid design

  • Most importantly, it taught me how UX can bridge not just devices — but people.

Key Takeaways

  • Led UX research and cultural analysis to define concept directions

  • Designed interaction flows and prototypes for cross-medium experiences

  • Built technical prototypes using Kinect, Processing, Arduino, and Figma

  • Conducted user testing and iterated interfaces based on findings

  • Contributed to a broader vision for inclusive, expressive hybrid festivals

More projects