Using short-term characteristics in speech processing

The following is a free translation of a record from the Sakshat Virtual Labs
Need for Short Term Processing of Speech website .
The article contains information about one of the methods for collecting the characteristics of a speech signal and about the three main characteristics that underlie many algorithms for processing audio signals and speech.

Most signal processing tools work in stationary systems, i.e. imply a stationary signal. Speech is reproduced by the vocal tract system, and therefore it is inherently non-stationary. Therefore, conventional means that are used for signal processing are not suitable for speech processing. Using them directly violates the underlying assumptions. And even if you blindly use them, the result will still not be of practical importance. For example, a means of calculating the total energy is fundamental in the field of signal processing:
Suppose you can use this formula to calculate speech energy. Undoubtedly, this will give us the energy present in the speech signal. However, the resulting value will not give us anything. The reason is the nature of speech - we know that it has amplitude and energy varying over time, therefore we need a tool that would provide information on changes in energy over time.

A solution for speech processing was proposed, which consisted of using already known methods from the field of signal processing with a slight modification. That is, the processing tools used still assumed a stationary signal. A stationary speech signal is obtained when viewed in small blocks of 10-30ms. Therefore, for speech processing by various signal processing tools, it is considered in blocks of 10-30ms (hereinafter, this section will be called a speech signal). This processing is called Short Term Processing (STP).

STP speech can be performed in the time or frequency domains. The choice of area depends on what information we want to extract from speech. For example, parameters such as short term energy, short term zero crossing rate and short term autocorrelation can be calculated in the time domain, and Fourier transforms can be calculated in the frequency domain. Each of these parameters gives some information about speech, and can be used for processing.

Short term energy

We call energy the abstract quantity that characterizes the signal. The energy of speech changes in time due to its nature and therefore, for any automatic processing of it, it is important to know how this energy changes in time. By origin, the speech signal consists of speech / non-speech sections / silence. The energy of a speech section is larger than the energy of a non-speech section, while the energy of silence is close to zero. Thus, the short term energy characteristic can be used to classify voice / non-voice portions based on the presence of speech or silence.

The formula for finding short-term energy can be derived from the formula for the total energy defined in the signal processing area. There, the total signal energy is calculated as follows:
To calculate short-term energy, we consider a section of speech with a duration of 10-30ms. Assume that the samples in the frame are listed as “n = 0 to n = N-1”, where N is the frame duration (the number of samples). Beyond the boundaries of the frame, the energy will be zero. Thus we get:

That is, the formula gives full energy in a block of speech.
where w (n) is a window function - several such functions are mentioned in the signal processing literature. The most commonly used
rectangular window is:
Hannah window:
or Hamming window:
For all characteristics calculated in the time domain, we will use a rectangular window because of its simplicity.
Now you can completely write down the short term energy calculation formula:
where n is the shift in samples. Since the energy changes in the case of speech are insignificant, it makes no sense to consider short term energy with a small shift. Therefore, most often it is set equal to or less than half the frame.

The last thing worth noting about short-term energy is the frame size. Since speech becomes approximately stationary in blocks of 10 to 30ms, usually a frame size of 20ms is chosen. If you choose a larger size, we will get a smoother picture of the energy and we may not notice how it changes.

Short Term Zero Crossing Rate (ZCR)

Zero Crossing Rate gives information about the number of changes in the sign of the function (intersections by the function of the OX axis). If the number of intersections is large in some signal, then the signal contains high-frequency information and vice versa. In this way, ZCR provides information on the frequency content of the signal.
In the case of a stationary signal, the ZCR is calculated as follows:
This formula can be corrected for an unsteady signal like speech and can be called as short term ZCR:
By the nature of speech, the signal changes over time after a few ms. In order to get some information, ZCR needs to be calculated on frames with the same duration of 10-30ms and a shift equal to half the frame. Above is a schedule for processing the recorded sentence “she had your suit in your greasy wash water all year”. On the vowel sound "s", the value of the characteristic significantly exceeds the value on the vowel "a".

Short term autocorrelation

In signal processing, cross-correlation can be used to find similarities between two sequences, and autocorrelation requires only one sequence and determines how much the signal resembles itself in time.
For a non-stationary signal, autocorrelation is calculated by the following formula:
where s _w = s (m) w (nm) is the window version of s (n). As a result, we get a short term autocorellation sequence. The nature of this sequence is different for sections with and without speech.

And although the topic of autocorrelation is not completely disclosed, it would be inappropriate not to mention it in the context of this topic.

PS In the next article, the implementation of the calculation of some of the characteristics is finally foreseen in order to make the material more understandable.