How to create a unique UrbanDenoiser Player with ambient noise compensation
In this article I want to talk about the process of creating an UrbanDenoiser Player media player. In addition to all the “charms and beauties” (which is better to read about on the almost the same official website ), the player has innovative playback modes:
IPhone version
Naturally, these capabilities would not have been possible without an innovative signal processing algorithm. Its creation will be discussed.
In early 2014, a friendly development team worked on the release of the Petralex Hearing Aid application , which implements a digital hearing aid based on a smartphone / tablet. Initially, the application was addressed to people who are forced (but shy) to wear a hearing aid. After all, the headset of a smartphone for a hearing aid will not be accepted by anyone. In addition, the smartphone / tablet has all the hardware (like the real - and very expensive! - digital hearing aid). It is only a matter of “software” that is capable of processing the audio signal in real time in accordance with the user's personal audiogram (characterizing hearing impairment), the removal of which is also a matter of “software”.
So, in the hottest period of completion of work on Petralex, two people from the development team, after a hard day's work, rode the subway. And one gave a friend an earpiece - to listen to a song with funny content. True, because of the noise (who traveled in the subway, he knows!), A friend did not make out a single word! And in haste, he suggested the idea: to analyze the surrounding noise and process the useful signal in such a way as to increase its intelligibility (despite the external noise!).
The idea of friends inspired. Mathematicians joined in the discussion to create the desired signal processing algorithm (using, among other things, the experience gained with Petralex).
The main problem was the eccentricity of the task - no publications on this topic could be found! At the first moment, the developers did not know about the available analogues at all. Perhaps that is why they solved the problem in a much more original way. And we learned about similar solutions (based on materials from Western conferences) already at the time of filing our own patent.
But that was later. At first, the signal processing algorithm was still very crude. No real time was discussed at all. However, this is normal for initial debugging - the goal was to make the software module work in principle in order to get the desired result.
The first prototype of the application, ready to be installed on the phone, came out five months after the idea itself arose - in June 2014. And immediately its shortcomings were revealed - the mathematical signal processing algorithm turned out to be very "heavy"!
Indeed, the main problem when processing sound in real time is that a simple and effective algorithm is required. So that the mobile phone has time to process a rather rapid data stream (44.1 kHz in our case). Typically, speech processing is limited to 8 or 16 kHz. But since music is also processed in our application, the frequency should be higher.
So, the resulting algorithm refused to work on iPhone 4. The developers had to roll up their sleeves again and optimize the C code. As a result, it was possible to reduce the processor load from 55% to 35% on iPhone 5. At that time, this turned out to be enough (although now the guys are working on an even greater reduction in CPU load so that the application runs on any “seedy Android”).
Testing the player with the updated algorithm proved to be great. The first mode (called Denoiser) was ready. However, further difficulties met again. For example, what degree of speed of the algorithm response to changes in ambient noise should be chosen? If you make the algorithm very “sensitive”, then the player’s sound will change rapidly - when changing the external noise environment. On the one hand, this is good - if the noise has arisen sharply and remains unchanged for a long time. But for "sudden" noises (such as a hammer knock, sharp blows, generally any single "disturbances"), the high sensitivity of the player is not suitable. On the other hand, a very slow reaction is also bad - the listener will have to wait a long time until the player's algorithm adapts to the new situation.
Another problem was the need to take into account the specifics of specific equipment (i.e., the intrinsic noise level of the headset microphone). The processing algorithm should not react to this noise at all! However, the trouble is that each phone has its own “individual” noise level. All this had to be somehow taken into account in the final version of the application. It's good that the sound system of the iPhone is characterized by a fairly low noise level.
At this stage of development, they decided to consider the level of their own noise for all "apple" devices the same. Although in the future (and now this work continues) the task is to “teach” the player not to respond to the user's voice, which is also recorded by the phone’s microphone - along with external noise.
On this, the mathematical sound processing algorithm was completed. In addition to the Denoiser mode (“no noise”), the developers came up with the functions of simultaneously listening to music and ambient sounds (for example, in the subway - it’s useful to hear the names of the transmitted stations).
But the Natural mode ("natural sounding") became a special pride of developers. In this mode, the smoothing of distortions introduced into the audio signal by the hardware of the reproducing device, as well as by the individual characteristics of the user's hearing (except in cases of obvious hearing impairment) is achieved.
The Natural effect is also achieved thanks to the "cunning" mathematical algorithm for processing the audio signal. Its essence is to evaluate the frequency response of the reproducing device relative to the subjective threshold of audibility in silence (for each user this threshold is different) - and enter the correction factor. In practice, this means that after spending a couple of minutes setting up the player (using the built-in procedure for determining the threshold of audibility), you can enjoy the lively coloring of the music. And feel yourself in a real concert hall!
The mathematical algorithm of the player has been debugged. But to turn it into a full-fledged mobile application, a friendly interface was required. And difficulties reappeared. The developers for a very long time were tormented by the synchronization of the drawing of the animation (in particular, the Spectrum - so that it did not “twitch” and coincided with the updates of the mobile screen). The CADisplayLink class is not explicitly defined anywhere. That is, of course, there is documentation. But try to find specific instructions that "this is very important and necessary"!
The next “stepping stone” to success was the struggle with the interface. In particular, in iOS, when you turn on the microphone (when your application is minimized), a red bar is displayed. The microphone can be turned off by calling one function, but the bar does not disappear. To remove it, you have to make a lot of unnecessary, and often interfering, manipulations (stop AudioGraph and restart again, which is not good - you can hear this stop by ear). Well, the common catch was the lack of simple applications (such as “Hello world”, working with the database, working with Audio, Video, etc.). Although in general, nothing could stop the victorious march of the application!
Here is a brief history of the creation of UrbanDenoiser Player - the world's first player with unique playback modes. The result of many months of work by a friendly team of specialists is now available to everyone - the player is distributed freely!
PS: From the latest news - the UrbanDenoiser Player application became the winner of the European Developers Cup held by Microsoft, in the B2C category http://appcup.eu
Thank you for your attention!
- Denoiser - increasing the intelligibility of the audio signal despite external noise;
- Noise Around - simultaneous listening to media content and sounds of the surrounding space;
- Natural - in which the player adapts the output signal not only to the frequency response of specific headphones, but also to the personal characteristics of the user's hearing.
IPhone version
Naturally, these capabilities would not have been possible without an innovative signal processing algorithm. Its creation will be discussed.
From hearing aids to media players
In early 2014, a friendly development team worked on the release of the Petralex Hearing Aid application , which implements a digital hearing aid based on a smartphone / tablet. Initially, the application was addressed to people who are forced (but shy) to wear a hearing aid. After all, the headset of a smartphone for a hearing aid will not be accepted by anyone. In addition, the smartphone / tablet has all the hardware (like the real - and very expensive! - digital hearing aid). It is only a matter of “software” that is capable of processing the audio signal in real time in accordance with the user's personal audiogram (characterizing hearing impairment), the removal of which is also a matter of “software”.
So, in the hottest period of completion of work on Petralex, two people from the development team, after a hard day's work, rode the subway. And one gave a friend an earpiece - to listen to a song with funny content. True, because of the noise (who traveled in the subway, he knows!), A friend did not make out a single word! And in haste, he suggested the idea: to analyze the surrounding noise and process the useful signal in such a way as to increase its intelligibility (despite the external noise!).
The idea of friends inspired. Mathematicians joined in the discussion to create the desired signal processing algorithm (using, among other things, the experience gained with Petralex).
The battle over the algorithm
The main problem was the eccentricity of the task - no publications on this topic could be found! At the first moment, the developers did not know about the available analogues at all. Perhaps that is why they solved the problem in a much more original way. And we learned about similar solutions (based on materials from Western conferences) already at the time of filing our own patent.
But that was later. At first, the signal processing algorithm was still very crude. No real time was discussed at all. However, this is normal for initial debugging - the goal was to make the software module work in principle in order to get the desired result.
The first prototype of the application, ready to be installed on the phone, came out five months after the idea itself arose - in June 2014. And immediately its shortcomings were revealed - the mathematical signal processing algorithm turned out to be very "heavy"!
Indeed, the main problem when processing sound in real time is that a simple and effective algorithm is required. So that the mobile phone has time to process a rather rapid data stream (44.1 kHz in our case). Typically, speech processing is limited to 8 or 16 kHz. But since music is also processed in our application, the frequency should be higher.
So, the resulting algorithm refused to work on iPhone 4. The developers had to roll up their sleeves again and optimize the C code. As a result, it was possible to reduce the processor load from 55% to 35% on iPhone 5. At that time, this turned out to be enough (although now the guys are working on an even greater reduction in CPU load so that the application runs on any “seedy Android”).
The search for "music lovers"
Testing the player with the updated algorithm proved to be great. The first mode (called Denoiser) was ready. However, further difficulties met again. For example, what degree of speed of the algorithm response to changes in ambient noise should be chosen? If you make the algorithm very “sensitive”, then the player’s sound will change rapidly - when changing the external noise environment. On the one hand, this is good - if the noise has arisen sharply and remains unchanged for a long time. But for "sudden" noises (such as a hammer knock, sharp blows, generally any single "disturbances"), the high sensitivity of the player is not suitable. On the other hand, a very slow reaction is also bad - the listener will have to wait a long time until the player's algorithm adapts to the new situation.
Another problem was the need to take into account the specifics of specific equipment (i.e., the intrinsic noise level of the headset microphone). The processing algorithm should not react to this noise at all! However, the trouble is that each phone has its own “individual” noise level. All this had to be somehow taken into account in the final version of the application. It's good that the sound system of the iPhone is characterized by a fairly low noise level.
At this stage of development, they decided to consider the level of their own noise for all "apple" devices the same. Although in the future (and now this work continues) the task is to “teach” the player not to respond to the user's voice, which is also recorded by the phone’s microphone - along with external noise.
On this, the mathematical sound processing algorithm was completed. In addition to the Denoiser mode (“no noise”), the developers came up with the functions of simultaneously listening to music and ambient sounds (for example, in the subway - it’s useful to hear the names of the transmitted stations).
But the Natural mode ("natural sounding") became a special pride of developers. In this mode, the smoothing of distortions introduced into the audio signal by the hardware of the reproducing device, as well as by the individual characteristics of the user's hearing (except in cases of obvious hearing impairment) is achieved.
The Natural effect is also achieved thanks to the "cunning" mathematical algorithm for processing the audio signal. Its essence is to evaluate the frequency response of the reproducing device relative to the subjective threshold of audibility in silence (for each user this threshold is different) - and enter the correction factor. In practice, this means that after spending a couple of minutes setting up the player (using the built-in procedure for determining the threshold of audibility), you can enjoy the lively coloring of the music. And feel yourself in a real concert hall!
Some more rainbow colors and ...
The mathematical algorithm of the player has been debugged. But to turn it into a full-fledged mobile application, a friendly interface was required. And difficulties reappeared. The developers for a very long time were tormented by the synchronization of the drawing of the animation (in particular, the Spectrum - so that it did not “twitch” and coincided with the updates of the mobile screen). The CADisplayLink class is not explicitly defined anywhere. That is, of course, there is documentation. But try to find specific instructions that "this is very important and necessary"!
The next “stepping stone” to success was the struggle with the interface. In particular, in iOS, when you turn on the microphone (when your application is minimized), a red bar is displayed. The microphone can be turned off by calling one function, but the bar does not disappear. To remove it, you have to make a lot of unnecessary, and often interfering, manipulations (stop AudioGraph and restart again, which is not good - you can hear this stop by ear). Well, the common catch was the lack of simple applications (such as “Hello world”, working with the database, working with Audio, Video, etc.). Although in general, nothing could stop the victorious march of the application!
Here is a brief history of the creation of UrbanDenoiser Player - the world's first player with unique playback modes. The result of many months of work by a friendly team of specialists is now available to everyone - the player is distributed freely!
PS: From the latest news - the UrbanDenoiser Player application became the winner of the European Developers Cup held by Microsoft, in the B2C category http://appcup.eu
Thank you for your attention!