The field of visual music has historically been constrained within two-dimensional screens, yet recent developments in virtual reality (VR) technology have the potential to revolutionise the medium by allowing new levels of integration between multi-sensory data streams. The relationship between sound and sight forms the theoretical basis for visual music composition, yet now, as VR systems proliferate, it is necessary to ask anew fundamental questions regarding the audio-visual relationship.

​

VR has been part of the popular consciousness since the 1990s, yet even a few years ago virtual reality systems (VRS), due to technological limitations and cost overheads, were still considered something of a novelty (Dascal et al, 2017; Riva, Wiederhold and Gaggioli, 2016). Despite this, the potential impact of virtual reality on the field of visual music has been a topic of some discussion; Francisco Kröpfl’s thoughts on the matter are worth quoting at length:

“One of the main drawbacks to integrating sound and visual image into an art form is the fact that while sound matter develops in ‘real’ space, visual image so far is projected on a bi-dimensional surface in a ‘virtual’ space. My point of view is that until a consistent development of holographic, that is to say, three-dimensional techniques are attained, integration will not be satisfactory enough.” (Kröpfl, 2007)

​

It has previously been shown that the human auditory and visual processing systems are highly interrelated (Talsma, Doty and Woldorff, 2007) and that audio and visual material, if congruent, can become mutually reinforcing and take on a gestalt quality, enhancing the effectiveness of both audio and visual streams (Chion, 1994). Audio-visual pairings in traditional media formats have been explored at length but how such relationships function within the context of virtual reality is not yet well understood.

​

Now, rapid advancements in hardware and software, spearheaded by the gaming and media industries, have led to the availability of sophisticated VRS competitively priced for average consumers (Ripton, Prasuethsut, 2015; Macedonia, Rosenbloom, 2001). The current generation of VRS represents a significant technological step forward (Dascal et al, 2017) and in fact departs so drastically from traditional methods of media consumption (Ryan, 2015) that it necessitates serious discussion of how virtual reality environments (VRE) may affect audio-visual relationships.

​

What’s so special about virtual reality?

​

Much as the transition from mono to stereo recording introduced a wealth of spatial possibilities into audio production, so too does the transition from 2D screens to VRE offers an exponential increase in how visual stimuli may be experienced. As Yan Breuleux succinctly states:

​

“By abolishing the frame of the image and offering audience members the possibility of choosing their own point of view, immersive approaches considerably change the process of designing and receiving works of art.” (Breuleux, 2015)

​

However, three-dimensionality is just one factor in what makes current generation VRS such an attractive medium: the functionality to include motion tracking, haptic feedback and, of course, integrated audio spatialisation form the basis of what has been termed ‘Multi-modal VR’ (Wilson, Soranzo, 2015). It is this ‘multi-modality’ that is perhaps the most exciting and powerful aspect of the current generation of VRS.

​

At this point it is necessary to introduce, and clarify, two terms that relate directly to the efficacy of a VRS/VRE: immersion and presence. Immersion, in this context, refers to a VRS’s ability to render realistic, detailed virtual environments and is directly impacted by a systems technical specifications. Presence, in this context, refers to the psychological effect of the VRE upon a user – the greater the presence the more a user will feel and act as if they are “really there” (Wilson, Soranzo, 2015). Immersion and presence are closely related and the immersive qualities of a system have been shown to positively correlate with the sense of presence (Diemer et al, 2015; Wilson, Soranzo, 2015).

​

Experimental research has characterised high levels of presence as resulting in an increased focus on stimuli within the virtual environment and a decreased focus for stimuli not relevant to the simulation (Burns, Fairclough, 2015). This occlusion of irrelevant stimuli is strong enough that it has been shown to decrease perception of acute and chronic pain and to relieve anxiety and stress in patients dealing with phobias (Dascal et al, 2017; Hoffman et al, 2014; Wiederhold et al, 2014). I refer to this process of masking stimuli that does not support the VRE ‘non-world occlusion’. Conversely, the increased focus on VRE-relevant stimuli, which results from higher levels of presence/immersion, has been shown to elicit higher memory recall, increased intensity of emotional responses, faster mental processing times and improved performance during abstract mental activities (Diemer et al, 2015; Maliński et al, 2015; Wilson, Soranzo, 2015; Keshavarz et al, 2014). I refer to this phenomenon as ‘in-world focus’.

​

In-world focus and non-world occlusion are powerful psychological effects and the scope of these phenomena, and what positive or negative impacts they might have for visual music composers, is yet to be determined. Based on the findings outlined above it is reasonable to infer that multi-modal VRS, aided by in-world focus, can induce a heightened sensitivity to stimuli occurring within a VRE. Accordingly, we must ask ourselves: what could such a heightened sensitivity mean for visual music composers and the audio-visual relationships?

​

The Audio-Visual relationship in VRE

​

My own experiential research in VRE has led me to identify the following areas, and their possible positive and negatives, for further investigation.

​

Concomitance

• Possible positive – The human tendency to integrate remote heterogeneous textures (Coulter, 2010) may be further enhanced by multi-modal VR. In-world focus may allow the creations of concomitant relationships between extremely disparate textures – as anything that impedes this integration may be masked by the process of non- world occlusion.

​

• Possible negative – It is possible that an increased sense of presence may place a higher burden upon the VRE to conform with real-world physical archetypes. Spatial positioning of audio in relation to visual textures is a key example: if a user is presented with audio and visual textures that cannot be reconciled with the physical properties of the VRE, then will this incongruence be masked by non-world occlusion or be highlighted by in-world focus?

Isomorphism

• Positive – It has been experimentally shown (Coulter, 2010) that, through the process of parametric mapping, it may be possible to create satisfying isomorphic relationships by linking as few as three simultaneous parameters – even if these parameters are selected randomly. Due to the heightened sensitivity and in-world focus elicited by multi-modal VR it may be possible to reduce the number of parametrically mapped parameters required to achieve isomorphic relationships. Such a reduction could lead to more convincing isomorphism between disparate textures, with lower technical overheads.

​

• Negative – Conversely a state of heightened sensitivity may instead cause a user to more easily perceive deficiencies in any parametric mapping strategies adopted. This could require more tightly integrated mapping strategies, leading to higher technical overheads.

  The failing of audio-visual relationships is a key concern as it could lead to what I describe as VR-related dissociation. The psychological effect of high-immersion, high-presence VREs can be profound – with research showing that users can experience a dissociative response to the real world once exiting the simulated environment, perceiving reality to be less ‘real’ than the VRE (Aardema et al, 2010). My own experiences with VREs have led me to conclude that feelings of dissociation can also be directed toward the VRE itself. When confronted with incongruent stimuli that defy the ‘natural laws’ perceived within the VRE then a sudden drop in presence has been observed. Such an effect naturally has serious implications for visual music works that frequently rely on creating a continuous atmosphere.

  With sight, sound, spatial constructs and physical sensations all exposed as modifiable parameters, Multimodal VR offers composers a level of control over sensory data unlike anything that has come before (Wilson, Soranzo, 2015). This paper will form the basis for further research into this topic and an audio-visual work will be created, as a vehicle for deeper exploration of these issues.

​

​

References

Breuleux, Y., 2015. VISUAL MUSIC: DISPLAY FORMATS. FULL-DOME PROJECTS. THE LANGUAGE OF A/V SPATIALISATION. In PROCEEDINGS OF UNDERSTANDING VISUAL MUSIC 2015 SYMPOSIUM (p. 7).

Burns, C.G. and Fairclough, S.H., 2015. Use of auditory event-related potentials to measure immersion during a computer game. International Journal of Human-Computer Studies, 73, pp.107-114.

Chion, M. 1994. Audio-Vision: Sound On Screen. (Claudia Gorbman Ed.) New York: Columbia University Press.

Coulter, J., 2010. Electroacoustic Music with Moving Images: the art of media pairing. Organised Sound, 15(01), pp.26-34.

Dascal, J., Reid, M., IsHak, W.W., Spiegel, B., Recacho, J., Rosen, B. and Danovitch, I., 2017. Virtual Reality and Medical Inpatients: A Systematic Review of Randomized, Controlled Trials. Innovations in Clinical Neuroscience, 14(1-2), p.14.

Diemer, J., Alpers, G.W., Peperkorn, H.M., Shiban, Y. and Mühlberger, A., 2015. The impact of perception and presence on emotional reactions: a review of research in virtual reality. Frontiers in psychology, 6, p.26.

F. Aardema, K. O’Connor, S. Cˆot ́e, and A. Taillon, 2010. Virtual reality induces dissociation and lowers sense of presence in objective reality. Cyberpsychology, Behavior, and Social Networking, 13(4), pp. 429–435.

Hoffman, H.G., Meyer III, W.J., Ramirez, M., Roberts, L., Seibel, E.J., Atzori, B., Sharar, S.R. and Patterson, D.R., 2014. Feasibility of articulated arm mounted Oculus Rift Virtual Reality goggles for adjunctive pain control during occupational therapy in pediatric burn patients. Cyberpsychology, Behavior, and Social Networking, 17(6), pp.397-401.

Keshavarz, B., Hettinger, L.J., Vena, D. and Campos, J.L., 2014. Combined effects of auditory and visual cues on the perception of vection. Experimental brain research, 232(3), pp.827-836.

Kröpfl, F. 2007. Integrating sound and visual image as artform. In Relationships Between Audition and Vision in the Creation in Electroacoustic Music. (Barriére, F. and Clozier C. Ed.) Academie Internationale de Musique Electroacoustique / Bourges, Institut International de Musique Electroacoustique de Bourges / IMEB, Bourges cedex, France. Volume VIII (2004-2005), 89-90.

Macedonia, M.R. and Rosenbloom, P., 2001. Entertainment technology and military virtual environments. ARMY SIMULATION TRAINING AND INSTRUMENTATION COMMAND ORLANDO FL.

Malińska, M., Zużewicz, K., Bugajska, J. and Grabowski, A., 2015. Heart rate variability (HRV) during virtual reality immersion. International Journal of Occupational Safety and Ergonomics, 21(1), pp.47-54.

Ripton, J. and Prasuethsut, L., 2015. The vr race: who’s closest to making vr a reality. URL http://www. techradar. com/news/world-of-tech/future-tech/the-vr-race-who-s-closest-to- making-vr-a-reality--1266538.

Riva, G., Wiederhold, B.K. and Gaggioli, A., 2016. Being different. The transfomative potential of virtual reality. Annu Rev Cybertherapy Telemed, 14, pp.1-4.

Ryan, M.L., 2015. Narrative as Virtual Reality II: Revisiting Immersion and Interactivity.

Talsma, D., Doty, T.J. and Woldorff, M.G., 2007. Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?. Cerebral cortex, 17(3), pp.679-690.

Wiederhold, B.K., Gao, K., Sulea, C. and Wiederhold, M.D., 2014. Virtual reality as a distraction technique in chronic pain patients. Cyberpsychology, Behavior, and Social Networking, 17(6), pp.346-352.

Wilson, C.J. and Soranzo, A., 2015. The use of virtual reality in psychology: a case study in visual perception. Computational and mathematical methods in medicine, 2015.