Virtual spaces for music, part 2: What’s next?

Next summer Helsinki gets a chance to experience the planet’s biggest virtual pop band, Gorillaz, who perform at Flow Festival.

Hopefully the August night will be dark enough to appreciate the group’s visuals, which include hologram-like images projected onto a polyester film for 3D effect. Such simulated holograms have also been used by former Flow headliners such as Janelle Monáe, MIA and Björk, with the latter also releasing the first virtual reality (VR) album in 2019. ABBA, meanwhile, has commissioned lifelike replicas of their younger selves to appear onstage in London beginning in May 2022.

Musical performances are being shaken up by technology such as VR, mixed reality (MR) and augmented reality (AR), which mingles digital information with the physical environment in real time. These are clumped together under the name extended reality (XR), which in turn is the cornerstone of the metaverse.

While still a relatively vague concept to many, the “metaverse” was first described in a 1992 sci-fi novel as a seamless blend of physical and virtual reality.

It could be the next major phase of the internet, knitting together physical and digital life. Using VR headsets or glasses or smartphone apps, its social technologies allow avatars to interact, for instance, in a concert experience.

Rocket growth

The sector is poised for head-spinning growth. The global market for AR, VR and mixed reality (MR) is forecast to explode nearly tenfold in just the next three years, from this year’s 30 billion US dollars to nearly 300 billion dollars by 2024.

So could tech-driven Finland and its music industry take a spotlight in this lucrative market, cross-fertilising tech and musical creativity?

That’s not far-fetched when you consider, for instance, that the iconic Oculus Rift VR headset was partly developed in Oulu. The startup that created it was bought by Facebook for over two billion dollars in 2014. The device and its successors are key to Facebook’s plans to delve into the metaverse and immersive social experiences – even inspiring its parent company to rename itself Meta.

Steven M. LaValle, Professor of Information Technology and Electrical Engineering at the University of Oulu, was head scientist of Oculus VR until it was acquired by Facebook. The university calls him “a famed roboticist and a pioneer in VR perception technology” and artificial intelligence (AI).

LaValle was recently awarded a European Research Council Advanced Grant for a perception engineering project aimed at expanding the VR concept across various disciplines. He is excited about various developments that could be exploited by the Finnish music industry.

“Things that I’ve found promising in recent years include being able to attend musical performances captured using panoramic cameras and a microphone array that results in fully spatialised audio – and video – on the receiving end,” he says. “Another is fully immersive music videos which allow artists tell a story along with their music.”

Looking into the future, LaValle suggests “purely synthetic, immersive virtual worlds that could accompany music with artistically crafted patterns which move to the music. These could be generated automatically, or developed by skilled artists,” he adds.

Also on his personal wish list: “an experience where a musically incompetent person like me could pretend to play instruments and jam with a real band! This could be an advanced version of an auto-tune app or Guitar Hero.” The latter is a series of games in which the player uses a guitar-shaped controller to simulate playing lead, bass and rhythm guitar on songs.

Air guitars and Instagram filters

Virtually shredding is a natural desire in Oulu, home of the Air Guitar World Championships – a rudimentary form of virtual musical performance in itself. The 25-year-old competition was only held virtually last summer.

In November, the festival unveiled the Air Guitar Party at Home experience, billed as “a unique, virtual party concept and a feel-good experience for groups of all sizes”. It was developed with support from Business Finland, which backs various other XR ventures such as Varjo, which makes human-eye resolution headsets. 

The air guitar event began as an offshoot of the Oulu Music Video Festival, which has likewise embraced cutting-edge tech. Held virtually in 2020, its hosts and winners ported filters that turned their faces into fantasy characters. They were designed by Maisa Immonen, a self-taught Helsinki-based 3D and XR artist who designs popular Instagram filters and record covers for alt-pop artists such as New Ro.

Winning the top prize was a startling video by the Helsinki band Pariisin Kevät, which is designed to be watched on a phone, collapsing a day’s worth of phone use including pop-up messages and ever-shifting apps – including the song itself playing on Spotify, and various subplots and characters – ending with a battery at 0% charge. “Onks se rakkautta” also won the Emma (or ‘Finnish Grammy’) Award for Best Music Video.

XR needs a better soundtrack

Fun stuff – but what are some other areas ripe for artistic tech innovations?

“Finland is excellently positioned to benefit from the development of XR technologies,” says Jani Jokitalo, who oversees MR projects at Business Finland.

“Finland has an extensive ecosystem of XR companies, which cover all the key stages of the value chain, from hardware development to software development, platform development, industry-specific applications and content development,” he says, pointing to the auditory dimension as a fertile area.

“Music and volumetric (3D) sound are important when creating a truly immersive experience. However in many XR applications, sound has not yet been fully utilised,” he says.

Besides Varjo, another promising tech start-up is Holonic Systems, co-founded in 2017 in Helsinki’s Kallio neighbourhood by CEO Ove Holmqvist.

Its Holon is an auditory AR app that generates interactive synth music from a user’s movements, position, environmental data and other signals.

“Many use our editor app, Holon.ist, as a modulation source to add expression to performances,” Holmqvist explains. “It’s easy to create notes and other musical events from movements or any other signals. We’ve had users control kinetic sculptures with their Apple Watch and others performing with Movesense sensors and iBeacons connected to objects in an abandoned mine.”

Others have used Holon.ist for circus acts and VJ:ing [the creation or manipulation of imagery in realtime].

“Our tech converts movement patterns back to sensory ones, which we perceive as music. The resulting functional feedback or biofeedback can be a driving factor in performances, leading to dissociative trance states and mindful, meditative states,” says Holmqvist.

While reminiscent of generative music apps such as Brian Eno's Bloom, Holon is fundamentally different, according to Holmqvist.

“Real-time synthesis makes for more than a listening experience because the user effectively performs the music, not just remixes it,” he says. “Our data sonification represents a bottom-up approach to music making. We aim to enable social musical communication through gestural and spatial computing.”

Holmqvist has high hopes for interactive music: “We already see many artists using game development tools for music-making, so this convergence is interesting to us.” 

In-game music: No limits

Live music events within games have been led by Epic Games’ Fortnite, which moved decisively into this arena during the pandemic, launching in-game concerts on combat-free Party Royale Island.

Electronic music heavyweights such as Diplo and Deadmau5 broadcast from a purpose-built LA studio, while Ariana Grande’s Rift Tour dispensed with the traditional stage set-up, with her avatar swooping through space in sparkling bubbles. Epic recently unveiled a new series featuring musicians from around the world, allowing players to interactively experience their music in-game.

Finnish games company Supercell, which created Clash of Clans and Brawl Stars, has not so far followed this lead but recently released a video single by fictional group the Bad Randoms, as a spinoff from Brawl Stars. The pop-punk tune has racked up more than 12 million views on YouTube since August – but it remains to be seen whether Finnish gaming firms will take bolder steps into music like rivals such as Epic and Riot Games.

Helsinki indie rockers Poets of the Fall, meanwhile, kick-started their international fan base through games. Their career took off when their debut single was used in Max Payne 2, produced by Finnish firm Remedy Entertainment. More songs have been featured in other Remedy games, some performed in-game by the band.

The intersection of music and gaming will be discussed at a Music Finland webinar on December 15, led by Kush Patel of the London firm Music Ally. Besides sync and licensing in games, in-game marketing and promotion, he’s an expert on livestreaming, immersive experiences and the metaverse, as well as artists as gamers.

As he sees it, gaming is still an uncharted frontier for musicians.

“The metaverses associated with some of the biggest game titles like Roblox or Fortnite offer artists a unique opportunity to express themselves creatively in a way that’s simply not possible in the real world. Imagination can run free, which leads to truly unique, memorable experiences for fans,” says Patel.

“One of my favourite recent collabs between gaming and music was a concert by Canadian artist Ruth B in Minecraft. It stood out to me in particular because it also had a strong user generated content (UGC) and educational aspect. It involved a wider campaign to encourage young girls to use their Science, Technology, Engineering, and Maths (STEM) skills to build the concert world in Minecraft. Fans were invited to create, not just consume. This UGC aspect is really interesting and a great way of deepening engagement with the campaign.”

Opera becomes immersive

Annastina Haapasaari leads the Opera Beyond project at the Finnish National Opera and Ballet (FNOB), which explores the possibilities of new tech for performing arts and develops the organisation’s digital strategy.

She also co-hosts the Radio Helsinki talk show Taide >< teknologia (Art or Technology). It discusses issues such as how new tech can support human performances and help convey an artist’s message – or become a gimmick that distracts listeners from the actual music.

The most ambitious production in the framework of Opera Beyond project so far was the immersive Laila installation which premiered at the FNOB in autumn 2020. The spatial installation utilised an array of technologies, from motion tracking to projection mapping and AI, to create a multisensory, interactive universe. (Read more here.)

The FNOB offers another immersive premiere in November 2022 – this time a main stage production, “which has its own characteristics and challenges,” she says. “The production combines circus and opera, and the immersive elements will give it a whole new dimension. For the first time ever, we will also have an entirely digital visiting artist performing in a show.” 

Besides musical innovations at the Opera House, Haapasaari is excited about developments in immersive or spatial sound, both with collective, speaker system-based, and individual headphone-based experiences. Such systems include Dolby Atmos, which has expanded from cinemas to home and personal use, as well as Apple Spatial Audio.

“The perfect metaverse could benefit tremendously from acoustic simulation and 3D audio. The acoustics of a space affects the listening experience and the reception of music even more than we think,” she suggests.

“In a way, VR is further advanced with audio than with visuals. When we listen to well-made 3D audio through a good speaker system or headphones and close our eyes, the experience can teleport us to another space.”

Featured photo: Holonic Systems / Ove Holmqvist