April 9, 2019 at 19:38How accurately does Yandex predict precipitation in winter? We analyze the accuracy of predictive services
In November, I published an article “Yandex.Meteum - technology without technology. Marketing accurate to the district ” , which correlated the quality of Yandex forecasts with other services. The emphasis was on temperature, without analyzing other parameters. The conclusion was this: the Yandex temperature forecast does not show any exceptional results compared to the already proven forecasting services. I recommend that you read the full text. This time, it's time to check out another key parameter - precipitation.
The main objective of the study is to understand the quality of the forecast for Yandex precipitation in comparison with other forecasting services. I draw your attention to the fact that the emphasis was on predicting the phenomenon, and not on the precipitation phase or their quantity.
As control forecasts, we used data from the intellicast.com website, which merged with the wundeground service in January , but this did not affect the forecasts, only the interface changed. The second control forecast is a comprehensive forecast from the website of the methodological office of the Hydrometeorological Center of Russia. I talked about this forecast in a previous publication.
The forecast was intercepted manually. The evaluation period is meteorological day, that is, from about 8-9 a.m. to 20-21 a.m. I did not take into account night precipitation. Forecasts were estimated from November 3, 2018 to March 31, 2019, that is, for the period during which climatic winter falls in the European part of Russia and part of the Urals, and there is no influence of convection.
Yandex forecasts were also captured only for the daytime. At the same time, I tried to correlate forecasts from the site with those forecasts that Alice sent in push notifications. Later it turned out that the forecasts do not agree with each other, but we will talk about this more.
Cities were chosen so that I could capture part of the south of the Urals and the main part of the European territory of Russia. I drew a line from Kurgan to Moscow, and chose the cities on this site at approximately equal intervals. These are Kurgan, Ufa, Kazan and Moscow. The average distance between the points is 591 km.
The Manual on Short- Range General Weather Forecasts , which is used by Roshydromet to evaluate forecasts up to 72 hours, was used as an assessment technique . As I said, it was important for me to evaluate the fact of precipitation, and not their amount or phase. Therefore, I entered two parameters in the tables: dry (no precipitation) or precipitation.
If the site predicted any precipitation at any hour of the day, then the forecast showed the gradation of "Precipitation", if there was no precipitation in the forecast, then "Dry".
The official ranks of Roshydromet stations were used as factual information. If at a weather station the precipitation meter recorded at least 0.1 mm in 12 hours, then the status “Precipitation” entered into the table. Further, the prognostic and actual gradations were correlated. I give the numbers of weather stations that were used in the study: 28661 (Kurgan), 28722 (Ufa), 27595 (Kazan), 27612 (Moscow, VDNH).
For example, Yandex predicts light snow during the day, and the weather station does not record, then the forecast is set to 0%. If there is any amount of precipitation, then 100%. Thus, all forecasts were placed in equal conditions of assessment.
For study transparency, I publish a link to the source tables with all the data. I also tried not only to evaluate forecasts, but also to correlate them with synoptic maps of zero lead time, which allowed me to identify those synoptic situations in which Yandex gives the lowest results of justification.
Initially, I hypothesized that Yandex will show similar results with other sites, or they will be slightly below the level of competitors. I had to say goodbye to this hypothesis at the end of November, when it turned out that Yandex’s accuracy was comparable to tossing a coin. The results were at the level of random forecasts and ranged from 50 to 60%.
For 5 months, Yandex accuracy in precipitation for 24 hours was 58%. Intellicast / wunderground has 81%, and the integrated forecast of the Hydrometeorological Center has 80%!
Yandex managed to discover an interesting feature - the farther from Moscow, the lower the accuracy. In Kurgan, the average accuracy was 45.6%, and in Moscow - 67.8%. Competitors' forecasts ranged from 75% to 85%, which looks very decent for the winter season.
The main problem of Yandex in the winter is false rainfall. Yandex predicts precipitation, while other services expect dry weather, a typical situation. The saddest thing is that Yandex predicts precipitation even in those territories over which there is an extensive anticyclone. For example, on November 12, Yandex predicted rainfall for Kazan throughout the day, but there shouldn’t be any, because the center of the anticyclone passed over the region, there is clear weather and there can be no rainfall physically.
A similar situation was repeated on December 20, but in Kurgan. Again, an anticyclone was established over the city, but Yandex persistently issued precipitation. There were a lot of such strange situations, I just gave the most vivid examples.
Another problem turned out to be with Alice and her push notifications, which she sends through the main Yandex application. For example, on January 3, the website wrote that there would be no rain on January 4, and Alice sent me: “Snow is beautiful. Tomorrow a little snow. " For all 5 months there were only a few such cases, but the fact of disagreement between the services leads to different thoughts.
Of course, most often Alice reported the same forecast that was listed on the site, but there was no precipitation. The error of Meteum is translated to the work of all other services and something needs to be done.
We figured out the precipitation forecast for 24 hours, but I would like to outline another important problem that is present in the Yandex. Weather".
In the previous article, I already wrote that Yandex publishes the average monthly rainfall in all cities. The problem is the data source. The site indicates the source of NOAA, I can assume that Yandex just use the data of the American computer reanalysis CFSR. The reanalysis is suitable for temperature studies, but it is extremely poor for studying precipitation. The computer model does not reproduce actual precipitation, especially of low intensity, poorly. The precipitation meter at the weather station records the real amount of precipitation, and the computer (reanalysis) calculates the virtual. Because of this, an error may occur.
The difference is especially pronounced in the summer, when convective processes are launched, and the models are very poor friends with them. Frontal precipitation is much easier to calculate than precipitation after passing through a thunderstorm cell (thundercloud). The error increases in mountainous terrain or in a marine climate. Therefore, when studying climate, it is advisable to rely only on data from the nearest weather station . Computer reanalysis should be resorted to only if we simply do not have analog data.
For example, Yandex Ufa counted 803 mm of precipitation per year, although in fact the annual rate is 586 mm. Yandex oversized precipitation by 37%. In Vladivostok, Yandex, by contrast, underestimated precipitation by 40%. According to Yandex, Ufa has more precipitation than in Vladivostok, which is absolutely stupid. I wrote to Yandex about this problem, but my appeal was ignored. Although, downloading ready-made arrays by stations and processing them, for Yandex, I think, is not such a big problem.
The forecast of Yandex on precipitation for 24 hours is much inferior in quality to other forecasting services. For 5 months, Yandex accuracy in precipitation for 24 hours was 58%. Intellicast / wunderground has 81%, and the integrated forecast of the Hydrometeorological Center has 80%! Yandex managed to discover an interesting feature - the farther from Moscow, the lower the accuracy. In Kurgan, the average accuracy was 45.6%, and in Moscow - 67.8%. The forecast parameters do not agree with each other: precipitation is predicted in clear anticyclonal weather.
I strongly do not recommend using Yandex information as the basic or sole source of forecast.
Main image: insidecorp.ru
GFS synoptic cards - meteoweb.ru/alter/prec.php
Verification conditions and control forecasts
The main objective of the study is to understand the quality of the forecast for Yandex precipitation in comparison with other forecasting services. I draw your attention to the fact that the emphasis was on predicting the phenomenon, and not on the precipitation phase or their quantity.
As control forecasts, we used data from the intellicast.com website, which merged with the wundeground service in January , but this did not affect the forecasts, only the interface changed. The second control forecast is a comprehensive forecast from the website of the methodological office of the Hydrometeorological Center of Russia. I talked about this forecast in a previous publication.
The forecast was intercepted manually. The evaluation period is meteorological day, that is, from about 8-9 a.m. to 20-21 a.m. I did not take into account night precipitation. Forecasts were estimated from November 3, 2018 to March 31, 2019, that is, for the period during which climatic winter falls in the European part of Russia and part of the Urals, and there is no influence of convection.
Yandex forecasts were also captured only for the daytime. At the same time, I tried to correlate forecasts from the site with those forecasts that Alice sent in push notifications. Later it turned out that the forecasts do not agree with each other, but we will talk about this more.
Cities were chosen so that I could capture part of the south of the Urals and the main part of the European territory of Russia. I drew a line from Kurgan to Moscow, and chose the cities on this site at approximately equal intervals. These are Kurgan, Ufa, Kazan and Moscow. The average distance between the points is 591 km.
The Manual on Short- Range General Weather Forecasts , which is used by Roshydromet to evaluate forecasts up to 72 hours, was used as an assessment technique . As I said, it was important for me to evaluate the fact of precipitation, and not their amount or phase. Therefore, I entered two parameters in the tables: dry (no precipitation) or precipitation.
If the site predicted any precipitation at any hour of the day, then the forecast showed the gradation of "Precipitation", if there was no precipitation in the forecast, then "Dry".
The official ranks of Roshydromet stations were used as factual information. If at a weather station the precipitation meter recorded at least 0.1 mm in 12 hours, then the status “Precipitation” entered into the table. Further, the prognostic and actual gradations were correlated. I give the numbers of weather stations that were used in the study: 28661 (Kurgan), 28722 (Ufa), 27595 (Kazan), 27612 (Moscow, VDNH).
For example, Yandex predicts light snow during the day, and the weather station does not record, then the forecast is set to 0%. If there is any amount of precipitation, then 100%. Thus, all forecasts were placed in equal conditions of assessment.
For study transparency, I publish a link to the source tables with all the data. I also tried not only to evaluate forecasts, but also to correlate them with synoptic maps of zero lead time, which allowed me to identify those synoptic situations in which Yandex gives the lowest results of justification.
Research results
Initially, I hypothesized that Yandex will show similar results with other sites, or they will be slightly below the level of competitors. I had to say goodbye to this hypothesis at the end of November, when it turned out that Yandex’s accuracy was comparable to tossing a coin. The results were at the level of random forecasts and ranged from 50 to 60%.
For 5 months, Yandex accuracy in precipitation for 24 hours was 58%. Intellicast / wunderground has 81%, and the integrated forecast of the Hydrometeorological Center has 80%!
Yandex managed to discover an interesting feature - the farther from Moscow, the lower the accuracy. In Kurgan, the average accuracy was 45.6%, and in Moscow - 67.8%. Competitors' forecasts ranged from 75% to 85%, which looks very decent for the winter season.
The main problem of Yandex in the winter is false rainfall. Yandex predicts precipitation, while other services expect dry weather, a typical situation. The saddest thing is that Yandex predicts precipitation even in those territories over which there is an extensive anticyclone. For example, on November 12, Yandex predicted rainfall for Kazan throughout the day, but there shouldn’t be any, because the center of the anticyclone passed over the region, there is clear weather and there can be no rainfall physically.
A similar situation was repeated on December 20, but in Kurgan. Again, an anticyclone was established over the city, but Yandex persistently issued precipitation. There were a lot of such strange situations, I just gave the most vivid examples.
Another problem turned out to be with Alice and her push notifications, which she sends through the main Yandex application. For example, on January 3, the website wrote that there would be no rain on January 4, and Alice sent me: “Snow is beautiful. Tomorrow a little snow. " For all 5 months there were only a few such cases, but the fact of disagreement between the services leads to different thoughts.
Of course, most often Alice reported the same forecast that was listed on the site, but there was no precipitation. The error of Meteum is translated to the work of all other services and something needs to be done.
Climate data
We figured out the precipitation forecast for 24 hours, but I would like to outline another important problem that is present in the Yandex. Weather".
In the previous article, I already wrote that Yandex publishes the average monthly rainfall in all cities. The problem is the data source. The site indicates the source of NOAA, I can assume that Yandex just use the data of the American computer reanalysis CFSR. The reanalysis is suitable for temperature studies, but it is extremely poor for studying precipitation. The computer model does not reproduce actual precipitation, especially of low intensity, poorly. The precipitation meter at the weather station records the real amount of precipitation, and the computer (reanalysis) calculates the virtual. Because of this, an error may occur.
The difference is especially pronounced in the summer, when convective processes are launched, and the models are very poor friends with them. Frontal precipitation is much easier to calculate than precipitation after passing through a thunderstorm cell (thundercloud). The error increases in mountainous terrain or in a marine climate. Therefore, when studying climate, it is advisable to rely only on data from the nearest weather station . Computer reanalysis should be resorted to only if we simply do not have analog data.
For example, Yandex Ufa counted 803 mm of precipitation per year, although in fact the annual rate is 586 mm. Yandex oversized precipitation by 37%. In Vladivostok, Yandex, by contrast, underestimated precipitation by 40%. According to Yandex, Ufa has more precipitation than in Vladivostok, which is absolutely stupid. I wrote to Yandex about this problem, but my appeal was ignored. Although, downloading ready-made arrays by stations and processing them, for Yandex, I think, is not such a big problem.
Conclusion
The forecast of Yandex on precipitation for 24 hours is much inferior in quality to other forecasting services. For 5 months, Yandex accuracy in precipitation for 24 hours was 58%. Intellicast / wunderground has 81%, and the integrated forecast of the Hydrometeorological Center has 80%! Yandex managed to discover an interesting feature - the farther from Moscow, the lower the accuracy. In Kurgan, the average accuracy was 45.6%, and in Moscow - 67.8%. The forecast parameters do not agree with each other: precipitation is predicted in clear anticyclonal weather.
I strongly do not recommend using Yandex information as the basic or sole source of forecast.
Main image: insidecorp.ru
GFS synoptic cards - meteoweb.ru/alter/prec.php