Cross-modal similarity between light and timbre

Practices in composition are moving steadily toward the integration of visual and aural elements. Encouraged by advances in technology making multimedia projects easier to both afford and implement effectively, composers have begun serious exploration of the interaction between sight and sound. The current study steps off from previous work in cross-modal perception and cognition in order to involve timbre in our growing understanding of the interaction of light and sound in compound stimuli.

The current study takes as a point of departure Lawrence Marks' 1987 study "On cross-modal similarity: Auditory-visual interaction in speeded discrimination." where subjects were presented with a gray background on a computer screen, on which was displayed centered square images that were either light (white) or dark (black). Each time a square, randomly selected, was displayed, a 70 dB(A) sawtooth waveform tone either at 220 Hz or 360 Hz (and also randomly selected) would simultaneously be played through headphones. Subjects received the compound stimulus comprised of both a sound and light component, one being criterial, and the other accessory. Subjects were asked to identify the criterial stimuli as quickly as possible by pressing one of two keys on a keyboard. The reaction time, stimulus, and response for each trial was recorded.

Marks found intersensory interaction between sound and light. He also found that interaction was greater when the criterial stimulus was light than when the criterial stimulus was sound. He suggested that this might be the case because it is easier to ignore light on a screen than it is to ignore sound being played through headphones. In a related study, Marks also found intersensory interaction when the intensity level was varied and the pitch held constant.

The current study is the first in a series of experiments based on Marks' study but involving timbre as one of the variable qualities of the sound component. In this first study, the setup was the same as in Marks' study except that instead of varying the frequency of the tone using a single sawtooth waveform, the frequency was held constant at 220 Hz and the waveform was either a sawtooth or a sine.

Marks found an interaction on the order of 10 to 35 ms between light levels and sound levels, and a similar interaction when intensity was varied instead of frequency. Results of the current study are very similar, suggesting that timbre is as prominent a component in the interaction of sound and light as either pitch or loudness.

The current series of studies is expected to include examinations of interaction between color and timbre, shape (of visual stimuli) and timbre, and the relationship between the interactions of light and loudness, pitch, and timbre.