February 1, 2008 at 22:01Forecast Quality Assessment Methods
Often when making any forecast, they forget about ways to evaluate its results. Because as often happens, there is a forecast, but there is no comparison with the fact. Even more errors happen when there are two (or more) models and it is not always obvious which one is better, more precisely. As a rule, one digit (R 2 ) is difficult to do. As if you were told this guy is wearing a blue T-shirt . And at once everything became clear about him to you)
In articles on forecasting methods, when evaluating the resulting model, I constantly used such abbreviations or notation.
I’ll try to explain what I had in mind.
So, in order. The main value through which the forecast accuracy is estimated is the residuals (sometimes: errors, error, e). In general terms, this is the difference between the predicted values and the source data (or actual values). Naturally, the more residuals, the stronger we were mistaken. To calculate the comparative coefficients, the residuals are transformed: either they are taken modulo or squared (see table, columns 4,5,6 ). In raw form, they are almost never used, since the sum of negative and positive residues can reduce the total error to zero. And this is stupid, you understand.
When we need to fit the curve to our data, the accuracy of this fit will be evaluated by the mean squared error (MSE) program . It is calculated by the straightforward formula
where n is the number of observations.
Accordingly, the program, calculating the fit curve, seeks to minimize this coefficient. The squares of the remainder in the numerator are taken precisely for the reason that the pros and cons are not mutually destroyed. MSE does not have physical meaning , but the closer to zero, the better the model.
The second abstract quantity is R 2 - coefficient of determination. It characterizes the degree of similarity between the source data and the predicted ones. Unlike MSE, it does not depend on the units of data, so it can be compared. The coefficient is calculated by the following formula:
where Var (Y) is the variance of the source data.
Of course, the coefficient of determination is an important criterion for choosing a model. And if the model does not correlate well with the source data, it is unlikely to have high predictive power.
Statistical methods for evaluating models like MSE and R 2 , unfortunately, are difficult to interpret, so the bright minds came up with lightweight, but easy to compare coefficients.
Mean absolute deviation (MAD) is defined as the quotient of the sum of residuals modulo the number of observations. That is, the average remainder modulo. Conveniently? It seems yes, but it seems not so. In my example, MAD = 43. Expressed in absolute units, the MAD indicates how much the forecast will make an average error.
MAPE is designed to give the model even more visual meaning. The expression is decrypted as mean percentage absolute error ( MAPE ) .
where Y is the value of the original series.
MAPE is expressed as a percentage, and in my case means that the model can be mistaken on average by 16%. Which, you see, is perfectly acceptable.
Finally, the last absolutely synthetic quantity is Bias, or simply bias . The fact is that in the real world deviations in one direction are often much more painful than in the other. For example, with conditionally unlimited storage facilities, it is more important to take into account real demand surges up from the predicted values. Therefore, cases where residuals are positive relate to the total number of observations. In my case, 44% of the predicted values were below the baseline. And you can sacrifice other evaluation criteria to minimize this Bias.
You can try it yourself in
Excel and 
Numbers.
It is interesting to know - what methods of assessing the quality of forecasting do you use in your work? Blog
Details
In articles on forecasting methods, when evaluating the resulting model, I constantly used such abbreviations or notation.
- R 2
- MSE
- MAPE
- Mad
- Bias
I’ll try to explain what I had in mind.
Leftovers
So, in order. The main value through which the forecast accuracy is estimated is the residuals (sometimes: errors, error, e). In general terms, this is the difference between the predicted values and the source data (or actual values). Naturally, the more residuals, the stronger we were mistaken. To calculate the comparative coefficients, the residuals are transformed: either they are taken modulo or squared (see table, columns 4,5,6 ). In raw form, they are almost never used, since the sum of negative and positive residues can reduce the total error to zero. And this is stupid, you understand.
Severe MSE and R 2
When we need to fit the curve to our data, the accuracy of this fit will be evaluated by the mean squared error (MSE) program . It is calculated by the straightforward formula
where n is the number of observations.
Accordingly, the program, calculating the fit curve, seeks to minimize this coefficient. The squares of the remainder in the numerator are taken precisely for the reason that the pros and cons are not mutually destroyed. MSE does not have physical meaning , but the closer to zero, the better the model.
The second abstract quantity is R 2 - coefficient of determination. It characterizes the degree of similarity between the source data and the predicted ones. Unlike MSE, it does not depend on the units of data, so it can be compared. The coefficient is calculated by the following formula:
where Var (Y) is the variance of the source data.
Of course, the coefficient of determination is an important criterion for choosing a model. And if the model does not correlate well with the source data, it is unlikely to have high predictive power.
MAPE and MAD to compare models
Statistical methods for evaluating models like MSE and R 2 , unfortunately, are difficult to interpret, so the bright minds came up with lightweight, but easy to compare coefficients.
Mean absolute deviation (MAD) is defined as the quotient of the sum of residuals modulo the number of observations. That is, the average remainder modulo. Conveniently? It seems yes, but it seems not so. In my example, MAD = 43. Expressed in absolute units, the MAD indicates how much the forecast will make an average error.
MAPE is designed to give the model even more visual meaning. The expression is decrypted as mean percentage absolute error ( MAPE ) .
where Y is the value of the original series.
MAPE is expressed as a percentage, and in my case means that the model can be mistaken on average by 16%. Which, you see, is perfectly acceptable.
Finally, the last absolutely synthetic quantity is Bias, or simply bias . The fact is that in the real world deviations in one direction are often much more painful than in the other. For example, with conditionally unlimited storage facilities, it is more important to take into account real demand surges up from the predicted values. Therefore, cases where residuals are positive relate to the total number of observations. In my case, 44% of the predicted values were below the baseline. And you can sacrifice other evaluation criteria to minimize this Bias.
You can try it yourself in
Excel and Numbers.It is interesting to know - what methods of assessing the quality of forecasting do you use in your work? Blog
Details
