The effect of the "sinister valley" in the voice acting of virtual reality

Many artists working in the gaming industry are familiar with the “sinister valley” effect, but sound engineers and composers generally do not face this problem. Nevertheless, given the rapid development of virtual reality, the situation can dramatically change. In this article we will talk about how exactly this effect can affect audio professionals. But first, let's find out what is the "sinister valley"?


To demonstrate the effect of the "sinister valley", this chart is most often used. The idea is this: the more an inanimate object, such as a robot, looks like a person, the prettier it seems. However, this effect grows only to a certain point: when the maximum degree of realism has just been reached, sympathy is sharply replaced by disgust or even fear. In other words, due to the slightest inconsistency of reality, your character turns into a monster. In more detail about the effect of the "sinister valley" is described in the video from the Extra Credits series:



So, we found out what artists mean by “sinister valley”, but what does this have to do with audio?

“Surround sound - an imitation of reality or an illusion?”

The concept of “sinister valley” in the field of audio was first presented by Francis Ramsay in the report “Surround sound - an imitation of reality or an illusion?” At the meeting of the Chicago branch of the Society of Acoustic Engineers (Audio Engineering Society) at headquarters Shure Incorporated's apartment in Niles, Illinois, in May 2014. Francis Ramsey holds a PhD in audio technology from the University of Surrey and is currently Chairman of the AES Technical Council.

In a brilliant one and a half hour presentation (a full recording of which AES members can view here) Francis Ramsey talks about the evolution of surround sound and about numerous attempts to achieve a flawless simulation of natural acoustic space. He also considers various approaches to solving this problem: from an aesthetic point of view, and from a mathematical one - by creating the ideal technology for the synthesis of the wave field. Along the way, he asks himself: what is the purpose of surround sound - the closest approach to reality or the creation of a plausible illusion? This leads us to the main point of the report.

Sound design in virtual reality: “almost” does not count

Ramsey claims that surround sound begins to stimulate more and more areas of the brain as it reaches the maximum degree of realism. Why is this happening? The fact is that hearing is very keen on the "naturalness" of sound. We are well aware of how sound travels in the environment. We know how it reflects, reverbs and how various objects can drown out sound or change its timbre. When the virtual acoustic environment reaches the pinnacle of spatial realism and timbre matching, our brain begins to compare this model with the real environment captured in our memory. And, even if at first glance the simulation is flawless, the slightest flaw will cause us a negative reaction. “The sound is almost like real, but something is wrong with it,” you think. “Something bothers me.”

As an example, let's watch the Oculus virtual reality video demonstrating the operation of the GenAudio AstoundSound 3D RTI surround sound plug-in. Despite the fact that the positioning of the audio is really impressive, in the demo version of the plug-in there are completely no blocking and sound absorption effects (which the creators do not hide). Thus, you can see for yourself what impression these discrepancies make in a realistic acoustic environment. They are especially well observed when the player enters the house, and the murmur of the fountain in the yard remains unchanged, although the sound should have become more muffled.



Voice-over for virtual reality: the “sinister valley" of speech in 3D

The report, Ramsey cites the results of research by Glenn Dickins, technical architect of digital convergence at Dolby Laboratories. Deakins explored the theory of "sinister valley" in relation to voice acting. Human speech is rooted in our consciousness in the same way as the human face and movements, so we easily perceive it in three-dimensional space. Thus, voice acting in 3D is even more at risk of falling into the "sinister valley." The minimum deviation of the voice from the conditional arrangement in space can cause the listener a feeling of deep discomfort.

The relevance of audio compression technology used in the development of traditional video games is questionable when it comes to virtual reality. This is especially true for voice acting. Any gamer can notice that the voices of the characters sound strange, not even knowing that the reason for this is the compression of the recording. The compression process changes the sound of the voice to the smallest, but in virtual reality it is heard quite distinctly. The slightest miscalculation - and the audio is on the edge of the "sinister valley."

Soundtracks in virtual reality: optimistic forecast

In my blog, I already discussed what role music plays in the three-dimensional acoustic space of virtual reality. But Francis Ramsay has an absolutely opposite opinion on this. According to him, it is not at all important for the listeners that the soundtrack should be surround. Composers in the gaming industry can breathe a sigh of relief, because this means that the soundtracks are not in danger of the “sinister valley” effect.

Referring to the results of a study of how inexperienced and experienced listeners perceive surround sound, Ramsey argues that positioning music in space is not so important. All survey participants agreed that the sound quality is the decisive factor for the soundtrack, but the volume does not affect it in any way. Does this mean that listeners are willing to voluntarily believe in this illusion, regardless of its compliance with the rules of virtual reality? If so, sound engineers can safely sacrifice the positioning of off-screen music (i.e., music whose source is not part of the virtual world) in favor of voice dubbing and sound design elements. At the same time, the soundtrack, as before, will contribute to emotional involvement and immersion in the game world. However, in other situations (for example, in films), listeners may perceive volumetric soundtracks differently. In any case, the role of voice-over music in virtual reality requires a more detailed study.

With other types of music, the situation is much simpler. After all, the soundtrack does not always fulfill its usual role of creating an atmosphere in the game. For example, in musical visualizers of virtual reality, positioning audio in space is absolutely not necessary, because it is already in the foreground (as if you were just listening to music). You can see it for yourself by watching the preview of the Harmonix Music VR music visualizer from the developer of such famous series as Rock Band and Dance Central:



Conclusion

In conclusion of the report, Francis Ramsey concludes: in almost everything related to voice acting of virtual reality, “almost” is not considered. In other words, a truly realistic sound should be truly flawless. Otherwise, it’s better to dwell on stylized audio, intentionally distorting and exaggerating aspects of real sound, than trying to recreate an exact copy of it.