What You See is What You Hear

Jonathan L. Friedmann, Ph.D.

Much has been written on the role of paralinguistic gestures in communicating linguistic content. Hand movements, postures, facial expressions and the like give context to spoken words and shade their meaning in significant ways. The non-verbal amplifies the verbal, conveying emotional information that may or may not be overt in the language alone. Something similar occurs in music performance.

Like verbal interaction, musical communication is a complex activity engaging multiple sensory modalities. Listening by itself does not extract all that a performance can disclose. This is especially so with instrumental music, a category of performance unaided by the (usual) clarity of words. No matter how formulaic or accessible, instrumental music is at best an abstract language. Thus, the full message and impact of a performance often relies on accompanying gestures, body movements and other paramusical signals.

It should be noted that research on music and emotions typically falls into three main categories. The most regularly explored is the influence of culture in shaping emotional responses. Schematic expectations and tonal patterns trigger stereotyped reactions among participants in a specific music-culture. The second most widely explored area is the effect of listening conditions. Settings and circumstances in which music is heard, along with the listener’s mental and physical states, contribute to how sounds are emotionally received. The third most commonly examined aspect is the impression of movement, form and imagery in musical passages. Such symbolism evokes emotions through mimicry, with slow phrases suggesting lethargy, ascending sequences implying elation, etc.

Visual cues deserve a place beside these conventional explanations. Cognitive studies have exposed the limits of emotional conveyance through strictly auditory features, like vibrato, tempo and dynamics. By itself, aural processing can and does open the pathway to music-induced emotions. However, the strength of music’s effect increases considerably and assumes added dimensions when a performance is both heard and seen.

Jane W. Davidson, a musicologist at the University of Western Australia, has examined the extent to which visual communication affects emotional perception. For a 1994 experiment, she had musicians play a piece in three distinct manners: restrained, with little to no physical expression; standard, with natural body and facial movement; and exaggerated, with effusive movement and facial cues. Just listening to the audio of these performances, participants were unable to detect which was played in which fashion. But when the performances were viewed, the intensity of emotional responses was proportional to the amount of gesturing, postural adjustments and facial signals observed. The more demonstrative the playing, the more emotional it seemed. A similar study conducted by Bradley W. Vines, et al. (2011) concludes that emotional ambiguity in atonal music can likewise be resolved through a player’s mannerisms.

Two additional observations deserve mention. The first is that musicians and non-musicians are equally unable to detect changes in performance manner when music is only heard, and are equally swayed by physical displays when performances are both heard and seen. The second is that the less familiar one is with a composition, the more one relies on sight in determining its emotional content. The grand takeaway is this: visuals are an underappreciated and immensely potent medium for enhancing, complementing and clarifying emotions in music.

Visit Jonathan’s website to keep up on his latest endeavors, browse his book and article archives, and listen to sample compositions.

3 thoughts on “What You See is What You Hear

  1. Pingback: What is unusually valuable about me? | power of language blog: partnering with reality by JR Fibonacci

  2. Pingback: SoundEagle in Art, Aphorism and Paramusic | SoundEagle

  3. Pingback: Do You Hear… What I Hear? | The Inspired Verse

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s