Introduction to the concept of entropy and its diversity
It may seem that the analysis of signals and data is a topic well studied and has been spoken hundreds of times. But there are some failures in it. In recent years, the word "entropy" has been rushed by all and sundry, plainly and not understanding what they are talking about. Chaos - yes, disorder - yes, it is used in thermodynamics - like yes too, as applied to signals - and here yes. I would like to at least clarify this point a little and give direction to those who want to learn a little more about entropy. Let's talk about entropy data analysis.
In Russian-language sources there is very little literature on this subject. And it’s almost impossible to get a complete picture. Fortunately, my supervisor turned out to be just a connoisseur of entropy analysis and the author of a fresh monograph [1], where everything is painted "from and to." Fortunately there was no limit, and I decided to try to convey my thoughts on this subject to a wider audience, so I will take a couple of excerpts from the monograph and supplement them with my research. Maybe someone will come in handy.
So, let's start from the beginning. In 1963, Shannon proposed the concept of a measure of the average informativeness of a test (unpredictability of its outcomes), which takes into account the probability of individual outcomes (there was Hartley before him, but this is omitted). If we measure the entropy in bits, and take base 2, we get the formula for the Shannon entropy
, where Pi is the probability of the i-th outcome. That is, in this case, entropy is directly related to the "unexpectedness" of the occurrence of the event. And from this follows its informativeness - the more predictable the event, the less informative. So its entropy will be lower. Although the question remains open about the relationship between the properties of information, the properties of entropy and the properties of its various estimates. It is precisely with estimates that we deal in most cases. All that can be investigated is the information content of various indices of entropy with respect to controlled changes in the properties of processes, i.e. in essence, their usefulness for solving specific applied problems.
The entropy of a signal described in some way (i.e., deterministic) tends to zero. For random processes, entropy increases the more, the higher the level of “unpredictability”. Perhaps it is from such a bunch of interpretations of entropy that the probability-> unpredictability-> informativeness implies the concept of “randomness”, although it is rather vague and vague (which does not interfere with its popularity). There is also the identification of entropy and the complexity of the process. But again, this is not the same thing.
We are going further.
Entropy is different
- thermodynamic
- algorithmic
- informational
- differential
- topological
They all differ on the one hand, and have a common foundation on the other. Of course, each type is used to solve certain problems. And, unfortunately, even in serious work there are errors in the interpretation of calculation results. And all this is due to the fact that in practice in 90% of cases we are dealing with a discrete representation of a signal of a continuous nature, which significantly affects the estimation of entropy (in fact, a correction factor appears in the formula, which is usually ignored).
In order to briefly describe the application of entropy to data analysis, we consider a small applied problem from the monograph [1] (which is not in digital form, and most likely will not).
Let there be a system that switches between several states every 100 clock cycles and generates a signal x (Figure 1.5), the characteristics of which change upon transition. But we don’t know which ones.
By breaking x into implementations of 100 samples, you can construct an empirical distribution density and calculate the Shannon entropy value from it. We will get the values “spaced” by levels (Figure 1.6).
As can be seen, transitions between states are clearly observed. But what to do if we don’t know the transition time? As it turned out, the calculation by a sliding window can help and the entropy also “spreads” to the levels. In a real study, we used this effect to analyze the EEG signal (multi-colored pictures about it will be further).
Now about one more interesting property of entropy - it allows you to evaluate the degree of connectedness of several processes . If they have the same sources, we say that the processes are connected (for example, if an earthquake is recorded at different points on the Earth, then the main component of the signal on the sensors is common). In such cases, correlation analysis is usually used, but it only works well for detecting linear relationships. In the case of nonlinear (generated by time delays, for example), we suggest using entropy.
Consider a model of 5 hidden variables (their entropy is shown in the figure below on the left) and 3 observables that are generated as a linear sum of hidden variables taken with time shifts according to the scheme shown below on the right. Numbers are coefficients and time shifts (in samples).
So, the trick is that the entropy of connected processes converges when their connection is strengthened. Damn it, how beautiful it is!
Such joys make it possible to extract additional information from almost any of the strangest and most chaotic signals (especially useful in economics and analytics). We pulled them out of the electroencephalogram, considering the now fashionable Sample Entropy and these are the pictures we got.
It can be seen that the jumps in entropy correspond to a change in the stages of the experiment. There are a couple of articles on this subject and the master's thesis is already protected, so if anyone is interested in the details, I will gladly share it. And so, around the world, EEG entropy has long been looking for different things - the stages of anesthesia, sleep, Alzheimer's and Parkinson's disease, consider the effectiveness of treatment for epilepsy and so on.But I repeat, often the calculations are carried out without taking into account correction factors and this is sad, since the reproducibility of studies is a big question (which is critical for science, then).
Summarizing, I will dwell on the universality of the entropy apparatus and its actual effectiveness, if we approach everything taking into account the pitfalls. I hope that after reading you will have a seed of respect for the great and mighty power of Entropy.
PS If there is interest, I can talk in more detail next time about the algorithms for calculating entropy and why Shannon's entropy is now shifted by more recent methods.
PPS Continuation about local-rank coding, see here
Literature
1. Tsvetkov OV Entropy data analysis in physics, biology and technology. SPb .: Publishing house of SPbGETU "LETI", 2015.202 p. www.polytechnics.ru/shop/product-details/370-cvetkov-ov-entropijnyj-analiz-dannyx-v-fizike-biologii-i-texnike.html
2.Abásolo D., Hornero R., Espino P. Entropy analysis of the EEG background activity in Alzheimer's disease patients // Physiological Measurement. 2006. Vol. 27 (3). P. 241 - 253. epubs.surrey.ac.uk/39603/6/Abasolo_et_al_PhysiolMeas_final_version_2006.pdf
3. 28. Bruce Eugene N, Bruce Margaret C, Vennelaganti S. Sample entropy tracks changes in EEG power spectrum with sleep state and aging / / Journal of Clinical Neurophysiology. 2009. Vol. 26 (4). P. 257 - 266. www.ncbi.nlm.nih.gov/pubmed/19590434
4. Entropy analysis as a method to find real bezgipoteznogo (homogenous) social groups (OI Shkaratan, GA Hawks) www.sociologos.ru/metody_i_tehnologii/Razdel_Analiz_dannyh/Statisticheskij_analiz/Entropijnyj_analiz_kak_metod_bezgipoteznogo_poiska_realnyh_gomogennyh_socialnyh
5. Entropy and other system laws: Management Issues complex systems. Prangishvili I.V. apolov-oleg.narod.ru/olderfiles/1/Prangishvili_I.V_JEntropiinye_i_dr-88665.pdf
