April 24, 2018 at 11:01Whose weather forecast is more accurate?
Many people think from time to time - how accurate are the weather forecasts? Whose weather forecast is more accurate? Which direction do the forecasts for my city most often make mistakes?
Math adventure time!
Over the past few years, Yandex has been actively developing its weather service. A couple of years ago, Metium technology was introduced and the precipitation forecast accurate to minutes.
In the discussion of these publications, fantast8 comments stood out . The amateur meteorologist attacked Yandex technology in a rather aggressive way and if you ignore the communication style, the essence of the complaints was the lack of texture and closed statistics.
And then I thought: does anyone even consider the accuracy of the forecasts?
The ForecastAdvisor project gave quick google , however, forecasts there were evaluated solely for US cities, and, of course, there were no Russian-language providers (Yandex, Hydromet, etc.).
In RuNet, for some time there was a MyCli.me project, now closed. However, he had persistent problems with data collection and the rating system (choosing the best forecast) was not based on a scientific method.
In addition, I found many initiatives, for example, from OLS and lucius , but all of them were limited either in time, or in the city, or in the provider. For example, one of the most amazing projects from slimyhorroranalyzes forecast forecasts for more than 5000 days in a row! But only forecasts from the BBC are evaluated and only for cities in the UK.
Thus, it was decided to do their own project. The point is small - to collect forecasts, to collect actual weather and to evaluate the marketability based on a scientifically based formula. We’ll start with her.
In 2009, the Hydrometeorological Center of Russia issued “Manuals on Short-Term Forecasts”. According to this manual, a forecast is considered to be fulfilled if it differs from the actual observed temperature by no more than 3 ° C in any direction. If the forecast differs from the actual weather by more than 3 °, then such a forecast is considered unfulfilled (paragraph 7.3.1.4 of the manual).
Weather forecasts should be estimated separately for day and night (7.2.3). The viability of the daily forecast is determined as the average of the values of the viability of forecasts for the night and the day. Assessment of the success of weather forecasts for a calendar period (month, season, year, etc.) consists in calculating their average justification for the corresponding period (9.1.1).
Okay, it seems not difficult - we take the forecast for tomorrow (the day after tomorrow, etc.), collect data on actual weather, compare separately for day and night. If
then the forecast did not come true (Pt = 0%), if <= 3 then Pt = 100%. We consider the average justification for the corresponding period and obtain the necessary data.
I immediately refused parsing sites with weather, because HTML parsing is not stable and this would inevitably increase the number of errors in the final statistics. Fortunately, many providers provide APIs to access their forecasts. In the first step, I added forecasts from accuweather.com, darksky.net, openweathermap.org and wunderground.com. All of them provide a JSON API. At the second stage, Norwegian yr.no with XML-API, the Russian Hydrometeorological Center (had to parse RSS) and Yandex.Weather were added.
To date, there is an agreement with foreca.com and negotiations are underway with gismeteo.ru, intellicast.com and the English metoffice. If you have friends in these projects who are ready to help with the API, write!
In theory, with the collection of actual weather, there should not have been any problems. Every major city has a weather monitoring station, usually in the center of the city, for example in Moscow it is VDNKh . National hydrometeorological services transmit data from these stations to the World Weather Watch , where it is processed and stored. Observations are transmitted as a SYNOP code every 3 or 6 hours.
In addition, there is a weather monitoring station at each major airport, which transmits an METAR format weather report every hour (or half an hour).
Raw SYNOP and METAR data can be found on the NOAA website; processed data on ogimet.com. The stations themselves are convenient to search on weatherobs.com.
However, there are several problems. Firstly, the quality of the transmitted data - typos are often misprinted in bulletins (for example, the letter O instead of the number 0). Secondly, data stability. In 2017, Aviamettelecom for the whole month stopped transferring meteorological reports from more than fifty airfields of Russia to the international exchange. Thirdly, there are cities in which I simply could not find a working station. For example, in Rostov-on-Don, there is no data either for station 34730 (URRR), or for station 34731.
In such situations, I use information from the private network of stations The Weather Company (now owned by IBM) to collect actual weather .
As a result, the site Meteotrust.com turned out .
At the moment, data is being collected for 10 cities of Russia from 7 providers. In the near future I plan to add more cities, more providers and more periods (long-term forecasts from hydrometcenter and accuweather, forecasts for 10 days). In addition, precipitation statistics are being prepared. Estimating the accuracy of precipitation forecasts is a little more complicated than temperature. This will be a separate post.
Openweathermap - bottom.
Accuweather, yr.no, darksky - strong middle peasants.
Yandex and Wunderground have excellent predictions.
The hydrometeorological center often fights Yandex for first place in the quality of short-term forecasts. However, there is an assumption that with an increase in the number of cities, especially foreign ones, the overall success of the forecasts of the Hydrometeorological Center will fall.
Math adventure time!
Over the past few years, Yandex has been actively developing its weather service. A couple of years ago, Metium technology was introduced and the precipitation forecast accurate to minutes.
In the discussion of these publications, fantast8 comments stood out . The amateur meteorologist attacked Yandex technology in a rather aggressive way and if you ignore the communication style, the essence of the complaints was the lack of texture and closed statistics.
And then I thought: does anyone even consider the accuracy of the forecasts?
Kingdom of fire
The ForecastAdvisor project gave quick google , however, forecasts there were evaluated solely for US cities, and, of course, there were no Russian-language providers (Yandex, Hydromet, etc.).
In RuNet, for some time there was a MyCli.me project, now closed. However, he had persistent problems with data collection and the rating system (choosing the best forecast) was not based on a scientific method.
In addition, I found many initiatives, for example, from OLS and lucius , but all of them were limited either in time, or in the city, or in the provider. For example, one of the most amazing projects from slimyhorroranalyzes forecast forecasts for more than 5000 days in a row! But only forecasts from the BBC are evaluated and only for cities in the UK.
Thus, it was decided to do their own project. The point is small - to collect forecasts, to collect actual weather and to evaluate the marketability based on a scientifically based formula. We’ll start with her.
Banana guard
In 2009, the Hydrometeorological Center of Russia issued “Manuals on Short-Term Forecasts”. According to this manual, a forecast is considered to be fulfilled if it differs from the actual observed temperature by no more than 3 ° C in any direction. If the forecast differs from the actual weather by more than 3 °, then such a forecast is considered unfulfilled (paragraph 7.3.1.4 of the manual).
More about graduation.
Theoretically, the provider would be able to predict the weather from -60 to +60 every day and thus get 100% marketability. However, the correct gradation of the forecast is considered to be a deviation of not more than 1 degree, for example, “Temperature in the daytime from 2 ° to 4 °”. Thus, a deviation of 2 degrees from the interval or 3 degrees from the value is allowed.
Weather forecasts should be estimated separately for day and night (7.2.3). The viability of the daily forecast is determined as the average of the values of the viability of forecasts for the night and the day. Assessment of the success of weather forecasts for a calendar period (month, season, year, etc.) consists in calculating their average justification for the corresponding period (9.1.1).
Okay, it seems not difficult - we take the forecast for tomorrow (the day after tomorrow, etc.), collect data on actual weather, compare separately for day and night. If
Math.abs(predict - actual) > 3then the forecast did not come true (Pt = 0%), if <= 3 then Pt = 100%. We consider the average justification for the corresponding period and obtain the necessary data.
Dark cloud
I immediately refused parsing sites with weather, because HTML parsing is not stable and this would inevitably increase the number of errors in the final statistics. Fortunately, many providers provide APIs to access their forecasts. In the first step, I added forecasts from accuweather.com, darksky.net, openweathermap.org and wunderground.com. All of them provide a JSON API. At the second stage, Norwegian yr.no with XML-API, the Russian Hydrometeorological Center (had to parse RSS) and Yandex.Weather were added.
To date, there is an agreement with foreca.com and negotiations are underway with gismeteo.ru, intellicast.com and the English metoffice. If you have friends in these projects who are ready to help with the API, write!
Increased oddity
In theory, with the collection of actual weather, there should not have been any problems. Every major city has a weather monitoring station, usually in the center of the city, for example in Moscow it is VDNKh . National hydrometeorological services transmit data from these stations to the World Weather Watch , where it is processed and stored. Observations are transmitted as a SYNOP code every 3 or 6 hours.
In addition, there is a weather monitoring station at each major airport, which transmits an METAR format weather report every hour (or half an hour).
Raw SYNOP and METAR data can be found on the NOAA website; processed data on ogimet.com. The stations themselves are convenient to search on weatherobs.com.
However, there are several problems. Firstly, the quality of the transmitted data - typos are often misprinted in bulletins (for example, the letter O instead of the number 0). Secondly, data stability. In 2017, Aviamettelecom for the whole month stopped transferring meteorological reports from more than fifty airfields of Russia to the international exchange. Thirdly, there are cities in which I simply could not find a working station. For example, in Rostov-on-Don, there is no data either for station 34730 (URRR), or for station 34731.
In such situations, I use information from the private network of stations The Weather Company (now owned by IBM) to collect actual weather .
Science rules
As a result, the site Meteotrust.com turned out .
At the moment, data is being collected for 10 cities of Russia from 7 providers. In the near future I plan to add more cities, more providers and more periods (long-term forecasts from hydrometcenter and accuweather, forecasts for 10 days). In addition, precipitation statistics are being prepared. Estimating the accuracy of precipitation forecasts is a little more complicated than temperature. This will be a separate post.
What are the findings?
Openweathermap - bottom.
Accuweather, yr.no, darksky - strong middle peasants.
Yandex and Wunderground have excellent predictions.
The hydrometeorological center often fights Yandex for first place in the quality of short-term forecasts. However, there is an assumption that with an increase in the number of cities, especially foreign ones, the overall success of the forecasts of the Hydrometeorological Center will fall.