Home > Uncategorized > Macroeconomic forecasting

Macroeconomic forecasting

from Lars Syll

Macroeconomic forecasts produced with macroeconomic models tend to be little better than intelligent guesswork. That is not an opinion – it is a fact. It is a fact because for decades many reputable and long standing model based forecasters have looked at their past errors, and that is what they find. It is also a fact because we can use models to generate standard errors for forecasts, as well as the most likely outcome that gets all the attention. Doing so indicates errors of a similar magnitude as those observed from past forecasts. In other words, model based forecasts are predictably bad …

Sales-forecast-guessing-dart_0I think it is safe to say that this inability to accurately forecast is unlikely to change anytime soon. Which raises an obvious question: why do people still use often elaborate models to forecast? …

It makes sense for both monetary and fiscal authorities to forecast. So why use the combination of a macroeconomic model and judgement to do so, rather than intelligent guesswork? (Intelligent guesswork here means some atheoretical time series forecasting technique.) The first point is that it is not obviously harmful to do so …

Many other organisations, not directly involved in policy making, produce macro forecasts. Why do they bother? Why not just use the policy makers’ forecast? A large part of the answer must be that the media shows great interest in these forecasts. Why is this? I’m tempted to say it’s for the same reason as many people read daily horoscopes. However I think it’s worth adding that there is a small element of a conspiracy to deceive going on here too …

The rather boring truth is that it is entirely predictable that forecasters will miss major recessions, just as it is equally predictable that each time this happens we get hundreds of articles written asking what has gone wrong with macro forecasting. The answer is always the same – nothing. Macroeconomic model based forecasts are always bad, but probably no worse than intelligent guesses.

Simon Wren-Lewis

Hmm …

Strange. On the one hand Wren-Lewis says that “macroeconomic forecasts are always bad,” but, on the other hand, since they are “probably no worse than intelligent guesses” and anyway are “not obviously harmful,” we have no reason to complain.

But Wren-Lewis is wrong. These forecasting models and the organization and persons around them do cost society billions of pounds, euros and dollars every year. And if they do not produce anything better than “intelligent guesswork,” I’m afraid most taxpayers would say that they are certainly not harmless at all!

Mainstream neoclassical economists often maintain – usually referring to the methodological individualism of Milton Friedman – that it doesn’t matter if the assumptions of the models they use are realistic or not. What matters is if the predictions are right or not. But, if so, then the only conclusion we can make is – throw away the garbage! Because, oh dear, oh dear, how wrong they have been!

When Simon Potter a couple of years ago analyzed the predictions that the Federal Reserve Bank of New York did on the development of real GDP and unemployment for the years 2007-2010, it turned out that the predictions were wrong with respectively 5.9% and 4.4% – which is equivalent to 6 millions of unemployed:

Economic forecasters never expect to predict precisely. One way of measuring the accuracy of their forecasts is against previous forecast errors. When judged by forecast error performance metrics from the macroeconomic quiescent period that many economists have labeled the Great Moderation, the New York Fed research staff forecasts, as well as most private sector forecasts for real activity before the Great Recession, look unusually far off the mark.

One source for such metrics is a paper by Reifschneider and Tulip (2007). They analyzed the forecast error performance of a range of public and private forecasters over 1986 to 2006 (that is, roughly the period that most economists associate with the Great Moderation in the United States).

On the basis of their analysis, one could have expected that an October 2007 forecast of real GDP growth for 2008 would be within 1.3 percentage points of the actual outcome 70 percent of the time. The New York Fed staff forecast at that time was for growth of 2.6 percent in 2008. Based on the forecast of 2.6 percent and the size of forecast errors over the Great Moderation period, one would have expected that 70 percent of the time, actual growth would be within the 1.3 to 3.9 percent range. The current estimate of actual growth in 2008 is -3.3 percent, indicating that our forecast was off by 5.9 percentage points.

Using a similar approach to Reifschneider and Tulip but including forecast errors for 2007, one would have expected that 70 percent of the time the unemployment rate in the fourth quarter of 2009 should have been within 0.7 percentage point of a forecast made in April 2008. The actual forecast error was 4.4 percentage points, equivalent to an unexpected increase of over 6 million in the number of unemployed workers. Under the erroneous assumption that the 70 percent projection error band was based on a normal distribution, this would have been a 6 standard deviation error, a very unlikely occurrence indeed.

In other words — the “rigorous” and “precise” macroeconomic mathematical-statistical forecasting models were wrong. And the rest of us have to pay.

Potter is not the only one who lately has criticized the forecasting business. John Mingers comes to essentially the same conclusion when scrutinizing it from a somewhat more theoretical angle:

It is clearly the case that experienced modellers could easily come up with significantly different models based on the same set of data thus undermining claims to researcher-independent objectivity. This has been demonstrated empirically by Magnus and Morgan (1999) who conducted an experiment in which an apprentice had to try to replicate the analysis of a dataset that might have been carried out by three different experts (Leamer, Sims, and Hendry) following their published guidance. In all cases the results were different from each other, and different from that which would have been produced by the expert, thus demonstrating the importance of tacit knowledge in statistical analysis.

Magnus and Morgan conducted a further experiment which involved eight expert teams, from different universities, analysing the same sets of data each using their own particular methodology. The data concerned the demand for food in the US and in the Netherlands and was based on a classic study by Tobin (1950) augmented with more recent data. The teams were asked to estimate the income elasticity of food demand and to forecast per capita food consumption. In terms of elasticities, the lowest estimates were around 0.38 whilst the highest were around 0.74 – clearly vastly different especially when remembering that these were based on the same sets of data. The forecasts were perhaps even more extreme – from a base of around 4000 in 1989 the lowest forecast for the year 2000 was 4130 while the highest was nearly 18000!

The empirical and theoretical evidence is clear. Predictions and forecasts are inherently difficult to make in a socio-economic domain where genuine uncertainty and unknown unknowns often rule the roost. The real processes that underly the time series that economists use to make their predictions and forecasts do not confirm with the assumptions made in the applied statistical and econometric models. Much less is a fortiori predictable than standardly — and uncritically — assumed. The forecasting models fail to a large extent because the kind of uncertainty that faces humans and societies actually makes the models strictly seen inapplicable. The future is inherently unknowable — and using statistics, econometrics, decision theory or game theory, does not in the least overcome this ontological fact. The economic future is not something that we normally can predict in advance. Better then to accept that as a rule “we simply do not know.”

So, to say that this counterproductive forecasting activity is harmless, simply isn’t true. Spending billions after billions of hard-earned money on an activity that is no better than “intelligent guesswork,” is doing harm to our economies.

A couple of years ago Lars E. O. Svensson — former deputy governor of the Swedish Riksbank — was able to show that the bank had conducted a monetary policy — based to a large extent on forecasts produced by its macroeconomic models — that led to far too high unemployment according to Svensson’s calculations. Unharmful? Hardly!

In New York State, Section 899 of the Code of Criminal Procedure provides that persons “Pretending to Forecast the Future” shall be considered disorderly under subdivision 3, Section 901 of the Code and liable to a fine of $250 and/or six months in prison. Although the law does not apply to “ecclesiastical bodies acting in good faith and without fees,” I’m not sure where that leaves macroeconomic model-builders and other forecasters …

  1. BC
    November 11, 2015 at 8:29 pm

    What is the actual purpose of economic modeling? It can’t be to accurately forecast economic activity and recessions, because what they model doesn’t actually exist and effectively the abstract “economy” at, or around, equilibrium is non-recessionary, so there is no need to be able to forecast recessions.

    That Establishment economics virtually ignores the financial sector (that’s what the bankster oligarchs and rentier Power Elite require), then neither can economic models anticipate financial crises.


    Economics is politics. Politics is war with other means. War is the business of empire, and empire is good business for imperialists.

    Therefore, economics is politics is the intellectual rationalization and justification for imperial wars for business expansion, resource and labor expropriation, and profits, often resulting in ecocide and genocide.

    Empire has little consideration for human or ecological well-being; rather, the overarching imperatives are growth of lending, investment, markets, profits, and securing same in perpetuity by the most efficient means possible.

    Therefore, in the first instance, economists are imperial ministerial politicians and sophists, not scientists, owing their intellectual basis and class status to the rentier Power Elite imperialists’ values and objectives.

    Thus, to expect sophists to practice science, as do physicists and biologists, is a fallacious first premise.

    If one assumes that the “best” economists are those who enjoy favor from the principal Power Elite imperialists, then it follows that the “best” economists are the best political sophists for the Power Elite’s global imperial values and objectives.

    Therefore, we should not expect the “best” economists to practice science in attempting to construct models of the real-world economy experienced by the vast majority of us.

  2. November 11, 2015 at 8:56 pm

    BC: quite exaggerated, indeed. Things don´t work so lineally. I tend to think that mistake and false creed are the reason of incompetence. Obviusly there is political bias, conspiracies exist, but they don’t always end well!

  3. November 12, 2015 at 8:36 am

    BC: good question! “What is the actual purpose of economic modelling”? That depends on whose purpose! Merijn Knibbe’s statistical purpose, for example, seems to be to alert us to the magnitude of problems we already have. That justifies my systems analysis style of modelling: to see how the system is operating so that one can see the points at which changes can be made and thus evaluate possible operational or system changes. Quantitative exactitude is ridiculous when the tolerances of the human components are huge, but it does matter whether immensely as to whether feedback is positive or negative: adding fuel to the fire or dampening it down.

  4. November 12, 2015 at 1:46 pm

    Prediction does not work? Try retrodiction first
    Comment on Lars Syll on ‘Macroeconomic forecasting’

    You point out that “In New York State, Section 899 of the Code of Criminal Procedure provides that persons ‘Pretending to Forecast the Future’ shall be considered disorderly … and liable to a fine of $250 and/or six months in prison.” (See intro)

    According to this law we should see all Wall Street gurus in prison and some academic economists, too.

    New York State law is in full accordance with science because “The future is unpredictable” (Feynman, 1992, p. 147). Feynman was a scientist and what he told the world is that science makes no predictions.

    Hold on, is that not the whole purpose of science?

    This is a slight misunderstanding that stems from astrology/astronomy. If, in some special cases, motion turns out to repeat periodically as with comets, then the next occurrence can be predicted with high precision. This prediction, though, does not require the knowledge of the law of gravitation or other laws of motion. The Mayas and others were good at predicting astronomical events long before Newton. This theory-free predictions were based on meticulous observations over long time spans and a bit of extrapolation.

    Apart from these periodically recurring phenomena, physics does not predict single historical events. The physicists’s predictions are of a different sort. For example: from the equivalence of energy and mass, his famous E=mc2, Einstein deduced that the path of light is deflected by large bodies. He calculated the deflection and asserted that it would be observable on occasion of solar eclipses. The rest of the story is well-known: “This eclipse was photographed from the expedition of Sir Arthur Eddington to the island of Principe. Positions of star images within the field near the Sun were used to test Albert Einstein’s prediction of the bending of light around the Sun from his general theory of relativity.” *

    The point is that scientists use the word prediction in a quite different sense from everyday usage. And this leads to the paradox that while the future is ‘unpredictable’ certain aspects may be ‘predictable’ with high precision. Loosely speaking, the laws of physics allow for conditional predictions.

    What does this mean for economics? As soon as we have economic laws, we are in the position to make conditional predictions. There is a snag here which is specific to economics because there is no such thing as laws of human behavior, yet there are structural laws of the economic system.

    To make a long analysis short, this is the First Economic Law for the elementary consumption economy

    This fundamental economic law contains the interrelation between the consolidated business sector’s cost/profit situation rhoF, the real rhoX and the nominal rhoE side of the product market, and the income distribution rhoD. So, the interrelations between firm, market, and the income distribution are encapsuled in the formula.

    This law allows for retrodiction (see Suppe, 1977, p. 621), that is, when we measure the ratios rho for past periods and insert them into the formula then the result must be 1 for all past periods. The formula holds in every single period from past to the future (2014, eq. (12)). In other words, we have a testable economic law. Test it twenty years hence and you will find out that it is true. Where, then, does the difficulty with prediction come in? The crucial point is that the variables that underlay the four rhos are random variables.

    Roughly speaking we can make a precise conditional prediction but the future conditions are not known. This allows for retrodiction but not for prediction except for one case. If we know/control three of the four ratios the forth can be predicted with absolute certainty — under the condition that the retrodiction tests have been successful.**

    Conclusion: When economic theory is built upon a behavioral axiom like constrained optimization then no prediction of any sort will ever be possible, but when the theory is built upon objective structural axioms, then conditional prediction becomes possible.

    Conditional predictions meet all scientific criteria and therefore the New York Code of Criminal Procedure is not applicable. It is still applicable, though, to the forecasts of standard economics which are derived from a logically and materially inconsistent theory, which in turn has been based on unacceptable green cheese behavioral assumptions.

    Egmont Kakarot-Handtke

    Feynman, R. P. (1992). The Character of Physical Law. London: Penguin.
    Kakarot-Handtke, E. (2014). The Synthesis of Economic Law, Evolution, and History. SSRN Working Paper Series, 2500696: 1–22. URL
    Suppe, F. (1977). Afterword–1977. In F. Suppe (Ed.), The Structure of Scientific Theories, pages 615–730. Urbana, IL, Chicago, IL: University of Illinois Press.

    * See Wikipedia
    ** For an advanced application see the following equation which conditionally predicts the unemployment rate and is therefore useful for employment policy

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s