Interview Series【8】Experimenting Arts in AR and VR to Connect Virtuality with Reality

Interview Series【8】Experimenting Arts in AR and VR to Connect Virtuality with Reality


“depicts three arts projects in visual arts, theatre and multi-disciplinary arts that are experimenting with Augmented Reality (AR) or Virtual Reality (VR) applications in order to go beyond the existing instruments and link virtuality to reality.”


Chinese text: Venus Lau

Mobile phones are our closest everyday carrier of technology. We watch the screens as we connect with others, upload life stories, pay bills, buy and sell, find the ways around, and even record health conditions. This applies to arts too. Under the pandemic, many creations have turned to online platforms. This provides an excellent opportunity to explore the feasibility of integrating technology and arts. This interview series depicts three arts projects in visual arts, theatre and multi-disciplinary arts that are experimenting with Augmented Reality (AR) or Virtual Reality (VR) applications in order to go beyond the existing instruments and link virtuality to reality.

Feel the Resilience through AR Anytime and Anywhere

AR technology has become a familiar term to many people since the popular mobile game Pokémon GO, in which players catch Pokémons in AR, caused a fad several years ago. “Letting Go” was the first part of the arts project “Letting Go Carry On”. Nine budding young artists were commissioned to create virtual works using AR technology. When an audience opens the work on the website with a mobile phone or tablet, it will turn on the camera function. The space where the audience is situated will be displayed and virtual images will also appear on the screen, mixing the virtual and the real. Creative Director Kingsley Ng said, “this part emphasises introspection, allowing people to have their own time to feel the resilience.”

“Letting Go” is the first part of “Letting Go Carry On”. (From left) Creative Director Kingsley Ng, participating artist Aza Kwok, and Co-curator Eugenia Law. (Photo by Wan Sui-lun)

Goodbye Heaviness by Aza Kwok. (Photo provided by the interviewee)

Artists find ways to express resilience with AR technology. This is one the artworks – Precious. (Photo provided by the interviewee)

Reality Composer, an AR creation tool by Apple, was applied in this project. Artists had to learn by themselves how to use the tool to create augmented reality experiences. Kingsley Ng recalled that the creative team spent a lot of time testing the app continuously. Reality Composer gave users an intuitive interface with multiple functions such as fixed-point shooting, triggering actions when moving objects close to the camera, or playing sound.

Virtuality is not necessarily a marvellous and exciting spectacle. The creators presented works with the theme “resilience” capitalising on the characteristics of various media and the resulting works were succinct and implicit. Aza Kwok’s work Goodbye Heaviness was a scene with twelve virtual stones forming a circle. When the user tapped the screen, the stones would slowly rise to the sky. Aza explained that she lived near Wong Shek Pier where she found many stones of special shapes. She then scanned the stones in 3D figures to produce an animation.

Her work carried her feelings during the pandemic and also showed a form of resilient mind with AR. “I felt trapped in the past year. In addition I had some emotions and always fancied about escaping from the present situation. My father told me that Wong Shek Pier had always been a spot on the ‘getaway’ route. I felt there was a great power. Even the stones there were magical,” she said. “So I made a stone circle which obviously should sink but instead flew upwards against gravity.”

Besides in the pier, the charm of resilience can be everywhere, which is the chemical reaction combining AR and arts. Users can watch the works in different places, take photos to share, and so on. Kingsley explained that site-specific creation was based upon the characteristics of the site or its cultural-historical development. This experiment was user-dominant, who created his or her own site-specific works anytime, anywhere and at his or her own pace. Other works of the project such as Absentmindfulness – the Earth and Exhalation in which plants grew in any environment; Sundown invited everybody to pause and enjoy the sunset.

“Letting Go” allows people to use their digital devices to play with the AR artworks, adding colours to their lives anywhere and anytime. (Photos provided by interviewee)

Public Space Turns into Stage at the Fingertips

Arts in the aiR was another project dealing with the relationship between AR and space. Five virtual performances including Chinese music, Western music, Indian-Pakistan folk songs, dance and drama were set up in installations in public housing estates. Curator Tung Tsz-ching was originally an Erhu performer. The pandemic had forced her to put off her performance, so she started to look into developing arts in a virtual direction.

Growing up in a public housing estate, she wanted to bring performances to public spaces in the housing estates and blend the beauty of the two. “Many buildings in public housing estates are beautiful, especially the older estates,” she said. “These estates also take humanistic care into consideration and provide a lot of public space so that people could gather and interact. Now these spaces have become the ‘stages’ for AR performances.”

With the support from Hong Kong Housing Society, Tung Tsz-ching had launched the Arts in the aiR in five estates under the Housing Society, namely Kwun Tong Garden Estate, Yue Kwong Chuen, Ming Wah Dai Ha, Lok Man Sun Chuen and Chun Seen Mei Chuen, and later would gradually extend to other housing estates. The project invited a group of artists to design respective performances based on the characteristics of the housing estates in order to enhance the connection and locality of the two. For instance, the theatre performance was inspired by the Kwun Tong Garden Building, which told the history of Kwun Tong as a salt farm in the Northern Song Dynasty.

Curator of Arts in the aiR Tung Tsz-ching is an Erhu player. She hopes to bring different performing arts to public housing estates with this project. (Photo by Cafe Tong)

Arts in the aiR also took a bold step forward to put portrait videos in AR mode. Tung Tsz-ching explained that this approach was quite new in Hong Kong. “Most common app in AR usually display animations, such as figures or special effects,” she said. “Now that we filmed physical performances done by people and we had to remove the background. This was very difficult for details like motions and hair.”

The project set up “AR Hotspots” in different housing estates, and users could enjoy the performances by scanning the hotspots with relevant mobile app.

“AR Hotspots” are placed in different public housing estates. Users can scan the hotpots with mobile application “ArtsInTheAiR” to view the performances at the spot. (Photo by Cafe Tong)

Users can choose five different performances in the mobile application, namely Chinese and Western music, Indian-Pakistan folk songs, dance and drama. Each performance is about four minutes. (Photo by Cafe Tong)

“Some uncles and aunties found out that performers would ‘pop up’ on the screen when they take pictures or videos of the physical environment with mobile phones at the spot. They were amazed and would call their children, friends and neighbours to join the game.” Tung Tsz-ching said, adding that the creative team had set up street counters in these housing estates to introduce the project and the response was heartening.

She continued, “Many foreign domestic helpers enjoyed the performances of Indian-Pakistan folk songs very much but most of them were Indonesians. I wondered why they loved Indian folk songs. I found out that it was because Bollywood movies were very popular in their social circles and they liked to listen to those songs, similarly as many Asians liked watching Hong Kong movies. Through these works, people had more opportunities to connect with each other.”

Experiencing Optional Fragmented Reality through VR Theatre

Could we really connect or even get into other people’ world through VR? The work A Mind Apart made a further attempt to apply VR technology to make a virtual reality theatre experience. The audience can “enter” the world of a couple and witness the disintegration of their relationship.

The so-called “theatre” combined multiple medium and took place in a hotel room instead of on the theatre stage. After putting on the VR headset, the audience would see the couple’s home and watch the video segment by segment.

Jantzen Tse was responsible for the filming, editing and soundscape design for the project. “We took reference from works such as Black Mirror that let viewers choose from different plots. There were in total 12 storylines so what you might see could be completely different from mine,” he said. Curator, director and co-scriptwriter Chan Ka Wai, Kiwi explained that the story described the inner anxiety of the male and female protagonists, and the fragmented storyline made it impossible for us to see the whole reality but reflected on how judgments and misunderstandings caused harm.

Creative team members of A Mind Apart: Jantzen Tse (left) and Kiwi Chan. (Photo by Cafe Tong)

A Mind Apart is staged in a hotel room. Audience view the performance with a VR headset. (Photo provided by the interviewee)

A Mind Apart was staged in Mixed Reality. In other words, the VR headset would make an audience see things in the actual hotel room in this moment and see the couple from the video in the other moments. The creative team intentionally set up in a hotel room with a layout similar to that of the apartment in the video in order to create a space where virtuality and reality overlapped. There was a bed in the video and the audience could actually lie down on it in the hotel room as if they were there in the scene. The hotel room was also placed with objects that appeared in the video including a selfie tripod, a telescope, an Ultraman figure and a mobile phone, etc. After watching a video segment, a message would prompt out on the screen instructing the audience to go for the next designated object or pick between any of the two objects. Then they had to locate the object in the hotel room and scan the sign on it to start the next video segment.

A Mind Apart allows the audience to “enter into” the relationship of a couple. They can decide how the story goes. (Photo provided by the interviewee)

The team shoots 360-degree videos, hoping to discover unusual viewing experiences. (Photo provided by the interviewee)

“We hoped the audience could walk in a space, instead of just sitting there turning their heads from time to time,” Kiwi said. “So we put an object on a higher level and the other under the bed so that the audience had to start or watch a video segment from a different point of view.” The team found that with headsets on, the audience might lose the sense of distance in the real space and had to spend extra time to accommodate. A virtual experience in turn raised the individual’s attention to the bodily sensation, which certainly was a bonus to this experiment.

Applying the technology posed a major challenge to the project. After the technical team had designed the app, they did not expect that not every mobile phone was compatible with the app. Hence they had to keep testing the app with different models of mobile phones. Kiwi said that although many technical problems happened in the live performance, the overall response was overwhelming. A lot of audiences stayed behind the performance to discuss the story and their feelings with the creative team.

Even though the audience assume that they are in the scene, they are still aware of the feeling of being absent and the sense of a bystander. How can immersive experience balance curiosity and sympathy? How do we get back to reasoning after sympathy? These are some major issues in the creation with virtual technology. After all, putting aside the VR headsets and mobile phones, we still share experience and exchange knowledge with real people.

Related Projects :