Sound effects in virtual worlds

Original author: Ruben Artus
  • Transfer
This article is the third in a series on the quality development of VR projects. Earlier, we already studied the principles of optimization in the article “Rendering and optimization in VR-development” . Links to other materials can be found in the review article .

So far, sound effects have played a secondary role in software development and have hardly been noticeable. But with VR development, they become an integral part of the project.

Correct sound accompaniment of a VR project significantly enhances user immersion and enhances emotions.

A good VR project takes us to virtual worlds and allows us to experience this simulated reality on an emotional level. However, there are certain expectations and, if our projects cannot meet them, immersion and credibility will disappear.

One aspect of these expectations is the sound effects that arise in real life. When a glass is put on a wooden table, they press a doorknob or run a hand over a chair - in all these situations there are sounds that we undoubtedly expect.

In classic games or applications, we are not so much involved, so the lack of sound does not bother us so much. But in VR, this is not so. Therefore, a new problem appears, but also a chance to transfer impressions to a new level.

Here, the fact that we know the exact position and rotation of the head, and, for example, in the case of Vive, also the position of the hands relative to the head, is useful to us. About what cool effects you can implement in your project, based on these data, we will consider at the end of the article.

First, let's go over the hardware and software and find out why even today, when simulating sound, they resort to deception.

Technology


image

The technology of modeling and recognition of sound waves has existed for a long time. Many still remember the 5.1 “surround sound system,” which has been popular for some time.

For a realistic perception of sound in the headphones , we have only two sound receivers (left and right). But the most important is the sound source. In the case of VR simulations, these are sound effects.

Sound waves are not an easy phenomenon, as they are reflected in space and change due to various materials and other influences.

In the real world, until a sound wave has found a way from the source to our ears, it changes significantly and receives certain characteristics, for example, when we are sitting in the bathroom or standing at the train station.

Real life forms certain expectations for the sound, our virtual world must meet these expectations.

This approach is very clearly presented in the next video. Despite the fact that you perceive video in ordinary headphones, you can go directly to the scene thanks to high-quality and realistic sound recording. You can even grasp the location of sound sources, such as a signal from a car.


Because of such complexity, a plausible simulation of sound on a computer costs a fortune, the necessary power to calculate several sound sources significantly exceeds the average computer.

Therefore, when developing software, they resort to deception and at the right time lose pre-recorded audio tracks , for example, when a circle breaks.

Then during the game, these entries will be adjusted depending on the environment. For example, in a large room, a gradual attenuation of sound at multiple reflections (reverb) will be added. In most cases, this substitution is sufficient.

The sensors give us very useful information about the exact position of the head in space. Thanks to this, we can bind sound effects to the corresponding source in the virtual room, and the software will calculate the direction and distortion.

This substitution is often called "Location Based Sound." It allows you to experience a very believable background noise in VR. In practice, high-quality simulations are achieved thanks to the VR SDK from Oculus or Valve.

All we need to do is place the sound effects in a virtual room.

Touching on this topic, it is worth mentioning a report on Oculus Connect 2, in 2015:


Increase immersion



After we introduced sound effects onto the stage when interacting with objects, we can make an even deeper immersion. Add trifles depending on the project or do extensive work with sound.

You can also use the position and rotation of the controller for certain effects. Developers from Hover Junkers used this opportunity, during the rapid movement of a virtual weapon (controller), you can hear "swooosh" or the rattle of metal.

This cool idea is shown in detail in the video:


The platform for which the project is being developed dictates various restrictions . When developing for Cardboard VR, the performance and memory capabilities are scarce compared to Vive VR.

The cool benefits of 3D sound and body positioning can be used to attract attention. Thus, it is possible to highlight important moments of the script or interactivity so that the user does not miss them and continues to navigate normally in the virtual world.

Dive deeper into the topic



In this article, we only slightly touched on this extensive and versatile topic.

For further study of the material, I strongly recommend the report of Nicolas Tsingos with VISION SUMMIT 2016:


Engadget’s article is also very interesting .

In this article, Todd Baker and Manesh Mistry - developers from Land's End and Monument Valley - provide substantive information about sound design in virtual reality. It is worth reading!

In the next article “VR-Design: User Interface” we will deal with the interface in VR (human-machine interaction).

Also popular now: