Programming low-latency audio in iOS
The article will discuss the features of the low-level API for working with sound in iOS, which were encountered when developing Viber. It will be about choosing the size of the hardware buffer and the behavior of AudioUnit when changing the sampling frequency.
For software work with sound in iOS, Apple provides 4 groups of APIs, each of which is designed to solve a specific class of tasks:
Quite a lot has been written about the initialization and basic use of AudioUnit, including examples in the official documentation. Let's consider not absolutely trivial features of its configuration and use. For interaction with sound as close as possible to the equipment, the RemoteIO and VoiceProcessingIO modules are responsible. VoiceProcessingIO adds to RemoteIO the ability to control additional sound processing at the OS level to improve voice quality and automatic signal level correction (AGC). From the point of view of the programmer, both of these modules have an “input” and an “output”, to which 2 buses are connected.
The programmer can set and request the audio stream format on these buses. By asking the format of the bus 1 stream at the AudioUnit input, you can find out the parameters of the stream received from the microphone at the hardware level, and setting the bus 1 format at the output, you can determine in what format the audio stream will be transferred to the application. Accordingly, by setting the format of the bus 0 for the AudioUnit input, we inform the format of the audio stream that we will provide for playing, and requesting the format of the bus 0 for the output, we find out what format the equipment uses for playing. Buffers are exchanged with AudioUnit in 2 callbacks with a signature:
InputCallback is called when the module is ready to provide us with a buffer of data recorded from the microphone. To get this data into the application, you need to call the AudioUnitRender function in this callback. RenderCallback is called when the module asks the application for playback data to be written to the ioData buffer. These callbacks are called in the context of the internal AudioUnit stream and should execute as soon as possible. Ideally, their work should be limited to copying ready-made data buffers. This introduces additional difficulties in the organization of audio signal processing in terms of stream synchronization. In addition to buffers, a timestamp is sent to these callbacks in the form:
This timestamp can (and should) be used to detect missing samples during recording and playback. The main reasons for the loss of samples:
In the ideal case, to obtain the minimum delay when recording sound, any intermediate buffering is absent. However, in the real world, hardware and software are more optimized for working with groups of sequential samples, rather than single samples. At the same time, iOS provides the ability to adjust the size of the hardware buffer. The PreferredHardwareIOBufferDuration audio session property allows you to request the required buffer duration in seconds, and CurrentHardwareIOBufferDuration to get the real one. Possible durations depend on the current sampling rate. For example, by default, when playing through the built-in speakers and recording through the built-in microphone, the equipment will work with a sampling frequency of 44100Hz. The minimum buffer used by the audio subsystem is 256 bytes, the size is usually equal to the power of two (the values are obtained experimentally and do not appear in the documentation). Therefore, a buffer may have a duration of:
256/44100 = 5.805ms
512/44100 = 11.61ms
1024/44100 = 23.22ms
If you use a bluetooth headset with a sampling frequency of 16000Hz, then the size of the hardware buffer can be:
256/16000 = 16ms
512/16000 = 32ms
1024/16000 = 64ms
The duration of the hardware buffer affects not only the delay, but also the number of samples that AudioUnit exchanges with the application each time the callback is called. If the sampling frequency coincides at the input and output of AudioUnit, a buffer equal in duration to the hardware will be transmitted and requested in the callback, and the callback will be called at regular intervals. Accordingly, if the application algorithms are designed to work with sequences of 10ms each, intermediate buffering on the application side will be necessary in any case, since AudioUnit cannot be configured to work with buffers of arbitrary duration. The size of the hardware buffer is best chosen experimentally, taking into account the performance of specific devices. Decreasing improves latency
In applications for VoIP communication, it does not always make sense to process sound with a sampling frequency above 16000Hz. In addition, it is easier to ignore the hardware sampling rate, since it can change at any time when switching the sound source. By configuring AudioUnit, you can set the sample rate of the audio stream when exchanging data with AudioUnit. Consider how this will work for sound recording in the following example:
After resampling, the output will be an integer number of samples, and the fractional remainder will be stored as the coefficients of the resampling filter. The behavior of AudioUnit is quite different in iOS5 and iOS6.
In iOS5, AudioUnit modules exchange buffers in size that are multiples of the power of two, so during the first call the application will receive 256 samples (16ms @ 16kHz). The remaining 371-256 = 115 will remain inside AudioUnit.
At the second callback call, the application will again receive a buffer of 256 samples: part of the data in it will be from the previous hardware buffer, and part from the new one.
At the third call, the balance accumulated after resampling will allow transferring 512 samples to the application right away.
Then, again, the application receives 256 samples.
Thus, when recording with resampling, a callback will be called at regular intervals, and the size of the buffer it receives will not be constant (but equal to a power of two).
The CoreAudio APIs are hard to call understandable and well-documented. Many features of the work have to be recognized experimentally, but you need to remember that the behavior may vary in different versions of the OS. For those who are interested in the topic of real-time audio processing, in addition to the Apple documentation, the “iZotope iOS Audio Programming Guide“ can be recommended .
For software work with sound in iOS, Apple provides 4 groups of APIs, each of which is designed to solve a specific class of tasks:
- AVFoundation allows you to play and record files and buffers in memory with the ability to use the hardware or software implementations of some audio codecs provided by the platform. It is recommended to use when there are no strict requirements for low latency of playback and playback.
- The OpenAL API is intended for rendering and reproducing three-dimensional sound as well as the use of sound effects. It is mainly used in games. Provides low playback delay, but does not provide the ability to record sound.
- AudioQueue is a basic API for recording and playing audio streams with the ability to use the codecs provided by the platform. Using this API, you can’t get the minimum delay, but using it is extremely simple.
- And finally, AudioUnit , a powerful and rich API for working with audio streams. Compared with Mac OS X on iOS, the programmer is not fully accessible, but is best suited for recording and playing sound as close to the hardware as possible.
AudioUnit
Quite a lot has been written about the initialization and basic use of AudioUnit, including examples in the official documentation. Let's consider not absolutely trivial features of its configuration and use. For interaction with sound as close as possible to the equipment, the RemoteIO and VoiceProcessingIO modules are responsible. VoiceProcessingIO adds to RemoteIO the ability to control additional sound processing at the OS level to improve voice quality and automatic signal level correction (AGC). From the point of view of the programmer, both of these modules have an “input” and an “output”, to which 2 buses are connected.
The programmer can set and request the audio stream format on these buses. By asking the format of the bus 1 stream at the AudioUnit input, you can find out the parameters of the stream received from the microphone at the hardware level, and setting the bus 1 format at the output, you can determine in what format the audio stream will be transferred to the application. Accordingly, by setting the format of the bus 0 for the AudioUnit input, we inform the format of the audio stream that we will provide for playing, and requesting the format of the bus 0 for the output, we find out what format the equipment uses for playing. Buffers are exchanged with AudioUnit in 2 callbacks with a signature:
OSStatus AURenderCallback(void * inRefCon,
AudioUnitRenderActionFlags * ioActionFlags,
const AudioTimeStamp * inTimeStamp,
Int32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData);
InputCallback is called when the module is ready to provide us with a buffer of data recorded from the microphone. To get this data into the application, you need to call the AudioUnitRender function in this callback. RenderCallback is called when the module asks the application for playback data to be written to the ioData buffer. These callbacks are called in the context of the internal AudioUnit stream and should execute as soon as possible. Ideally, their work should be limited to copying ready-made data buffers. This introduces additional difficulties in the organization of audio signal processing in terms of stream synchronization. In addition to buffers, a timestamp is sent to these callbacks in the form:
struct AudioTimeStamp {
Float64 mSampleTime; // метка в количестве семплов
UInt64 mHostTime; // метка в абсолютном системном времени
// поля не имеющие отношения к звуку
// ...
UInt32 mFlags; // маска указывающая какие из полей заполнены
};
This timestamp can (and should) be used to detect missing samples during recording and playback. The main reasons for the loss of samples:
- Switch recording and playback devices (speaker / headphone / Bluetooth). It is impossible to get rid of the loss of part of the samples in this case. The time stamp can be used to adjust further processing of the sound, for example, synchronization with the video stream or recalculation of the “Timestamp” field of the RTP packet.
- Too much CPU load, in which the AudioUnit thread does not have enough time to work, is fixed by optimizing the algorithms or by refusing to support insufficiently powerful devices.
- Errors in the implementation of stream synchronization when working with audio data buffers. In this case, the correct use of lock-free structures, circular buffers, GCDs will help (however, GCDs are not always a good solution for tasks close to real-time). You can use System Trace from Instruments to identify the causes of thread synchronization problems.
Hardware buffer size
In the ideal case, to obtain the minimum delay when recording sound, any intermediate buffering is absent. However, in the real world, hardware and software are more optimized for working with groups of sequential samples, rather than single samples. At the same time, iOS provides the ability to adjust the size of the hardware buffer. The PreferredHardwareIOBufferDuration audio session property allows you to request the required buffer duration in seconds, and CurrentHardwareIOBufferDuration to get the real one. Possible durations depend on the current sampling rate. For example, by default, when playing through the built-in speakers and recording through the built-in microphone, the equipment will work with a sampling frequency of 44100Hz. The minimum buffer used by the audio subsystem is 256 bytes, the size is usually equal to the power of two (the values are obtained experimentally and do not appear in the documentation). Therefore, a buffer may have a duration of:
256/44100 = 5.805ms
512/44100 = 11.61ms
1024/44100 = 23.22ms
If you use a bluetooth headset with a sampling frequency of 16000Hz, then the size of the hardware buffer can be:
256/16000 = 16ms
512/16000 = 32ms
1024/16000 = 64ms
The duration of the hardware buffer affects not only the delay, but also the number of samples that AudioUnit exchanges with the application each time the callback is called. If the sampling frequency coincides at the input and output of AudioUnit, a buffer equal in duration to the hardware will be transmitted and requested in the callback, and the callback will be called at regular intervals. Accordingly, if the application algorithms are designed to work with sequences of 10ms each, intermediate buffering on the application side will be necessary in any case, since AudioUnit cannot be configured to work with buffers of arbitrary duration. The size of the hardware buffer is best chosen experimentally, taking into account the performance of specific devices. Decreasing improves latency
Buffering when sampling rate changes
In applications for VoIP communication, it does not always make sense to process sound with a sampling frequency above 16000Hz. In addition, it is easier to ignore the hardware sampling rate, since it can change at any time when switching the sound source. By configuring AudioUnit, you can set the sample rate of the audio stream when exchanging data with AudioUnit. Consider how this will work for sound recording in the following example:
SampleRateHW = 44100 // аппаратная частота дискретизации
buffSizeHW = 1024 // размер аппаратного буфера (1024 / 44100 = 23.22ms)
mSampleRateAPP = 16000 // частота дискретизации приложения
buffSizeAPP = 1024 * 16000/44100 = 371.52 // размер одного после ресемплирования
After resampling, the output will be an integer number of samples, and the fractional remainder will be stored as the coefficients of the resampling filter. The behavior of AudioUnit is quite different in iOS5 and iOS6.
iOS5
In iOS5, AudioUnit modules exchange buffers in size that are multiples of the power of two, so during the first call the application will receive 256 samples (16ms @ 16kHz). The remaining 371-256 = 115 will remain inside AudioUnit.
At the second callback call, the application will again receive a buffer of 256 samples: part of the data in it will be from the previous hardware buffer, and part from the new one.
At the third call, the balance accumulated after resampling will allow transferring 512 samples to the application right away.
Then, again, the application receives 256 samples.
Thus, when recording with resampling, a callback will be called at regular intervals, and the size of the buffer it receives will not be constant (but equal to a power of two).
iOS6
In iOS6, the restriction on the size of the buffer transferred between the application and AudioUnit was removed, thus eliminating the intermediate buffering during resampling and, accordingly, reducing the delay. The application will receive buffers of 371 and 372 sample sizes alternately.The CoreAudio APIs are hard to call understandable and well-documented. Many features of the work have to be recognized experimentally, but you need to remember that the behavior may vary in different versions of the OS. For those who are interested in the topic of real-time audio processing, in addition to the Apple documentation, the “iZotope iOS Audio Programming Guide“ can be recommended .
