Home > The Economics Profession > Econometric forecasting — a retrospective assessment

Econometric forecasting — a retrospective assessment

from Lars Syll

There have been over four decades of econometric research on business cycles … The formalization has undeniably improved the scientific strength of business cycle measures …

But the significance of the formalization becomes more difficult to identify when it is assessed from the applied perspective, especially when the success rate in ex-ante forecasts of recessions is used as a key criterion. The fact that the onset of the 2008 financial-crisis-triggered recession was predicted by only a few ‘Wise Owls’ … while missed by regular forecasters armed with various models serves us as the latest warning that the efficiency of the formalization might be far from optimal. Remarkably, not only has the performance of time-series data-driven econometric models been off the track this time, so has that of the whole bunch of theory-rich macro dynamic models developed in the wake of the rational expectations movement, which derived its fame mainly from exploiting the forecast failures of the macro-econometric models of the mid-1970s recession.

The limits of econometric forecasting has, as noted by Qin, been critically pointed out many times before.

Trygve Haavelmo — with the completion (in 1958) of the twenty-fifth volume of Econometrica – assessed the the role of econometrics in the advancement of economics, and although mainly positive of the “repair work” and “clearing-up work” done, Haavelmo also found some grounds for despair:

We have found certain general principles which would seem to make good sense. Essentially, these principles are based on the reasonable idea that, if an economic model is in fact “correct” or “true,” we can say something a priori about the way in which the data emerging from it must behave. We can say something, a priori, about whether it is theoretically possible to estimate the parameters involved. And we can decide, a priori, what the proper estimation procedure should be … But the concrete results of these efforts have often been a seemingly lower degree of accuracy of the would-be economic laws (i.e., larger residuals), or coefficients that seem a priori less reasonable than those obtained by using cruder or clearly inconsistent methods.

Haavelmo-intro-2-125397_630x210There is the possibility that the more stringent methods we have been striving to develop have actually opened our eyes to recognize a plain fact: viz., that the “laws” of economics are not very accurate in the sense of a close fit, and that we have been living in a dream-world of large but somewhat superficial or spurious correlations.

And as the quote below shows, even Ragnar Frisch shared some of Haavelmo’s — and Keynes’s — doubts on the applicability of econometrics:

sp9997db.hovedspalteI have personally always been skeptical of the possibility of making macroeconomic predictions about the development that will follow on the basis of given initial conditions … I have believed that the analytical work will give higher yields – now and in the near future – if they become applied in macroeconomic decision models where the line of thought is the following: “If this or that policy is made, and these conditions are met in the period under consideration, probably a tendency to go in this or that direction is created”.

Ragnar Frisch

Econometrics may be an informative tool for research. But if its practitioners do not investigate and make an effort of providing a justification for the credibility of the assumptions on which they erect their building, it will not fulfill its tasks. There is a gap between its aspirations and its accomplishments, and without more supportive evidence to substantiate its claims, critics will continue to consider its ultimate argument as a mixture of rather unhelpful metaphors and metaphysics. Maintaining that economics is a science in the “true knowledge” business, I remain a skeptic of the pretences and aspirations of econometrics. So far, I cannot really see that it has yielded very much in terms of relevant, interesting economic knowledge. And, more specifically,  when it comes to forecasting activities, the results have been bleak indeed.

The marginal return on its ever higher technical sophistication in no way makes up for the lack of serious under-labouring of its deeper philosophical and methodological foundations that already Keynes complained about. The rather one-sided emphasis of usefulness and its concomitant instrumentalist justification cannot hide that the legions of probabilistic econometricians who give supportive evidence for their considering it “fruitful to believe” in the possibility of treating unique economic data as the observable results of random drawings from an imaginary sampling of an imaginary population, are scating on thin ice. After having analyzed some of its ontological and epistemological foundations, I cannot but conclude that econometrics on the whole has not delivered “truth,” nor robust forecasts. And I doubt if it has ever been the intention of its main protagonists.

Our admiration for technical virtuosity should not blind us to the fact that we have to have a more cautious attitude towards probabilistic inference of causality in economic contexts. Science should help us penetrate to — as Keynes put it — “the true process of causation lying behind current events” and disclose “the causal forces behind the apparent facts.”  We should look out for causal relations, but econometrics can never be more than a starting point in that endeavour, since econometric (statistical) explanations are not explanations in terms of mechanisms, powers, capacities or causes. Firmly stuck in an empiricist tradition, econometrics is only concerned with the measurable aspects of reality, But there is always the possibility that there are other variables – of vital importance and although perhaps unobservable and non-additive not necessarily epistemologically inaccessible – that were not considered for the model. Those who were can hence never be guaranteed to be more than potential causes, and not real causes.

A rigorous application of econometric methods in economics really presupposes that the phenomena of our real world economies are ruled by stable causal relations between variables. A perusal of the leading econom(etr)ic journals shows that most econometricians still concentrate on fixed parameter models and that parameter-values estimated in specific spatio-temporal contexts are presupposed to be exportable to totally different contexts. To warrant this assumption one, however, has to convincingly establish that the targeted acting causes are stable and invariant so that they maintain their parametric status after the bridging. The endemic lack of predictive success of the econometric project indicates that this hope of finding fixed parameters is a hope for which there really is no other ground than hope itself.

This is a more fundamental and radical problem than the celebrated “Lucas critique” have suggested. This is not the question if deep parameters, absent on the macro-level, exist in “tastes” and “technology” on the micro-level. It goes deeper. Real world social systems are not governed by stable causal mechanisms or capacities. It is the criticism that Keynes — in Essays in Biography — first launched against econometrics and inferential statistics already in the 1920s:

The atomic hypothesis which has worked so splendidly in Physics breaks down in Psychics. We are faced at every turn with the problems of Organic Unity, of Discreteness, of Discontinuity – the whole is not equal to the sum of the parts, comparisons of quantity fails us, small changes produce large effects, the assumptions of a uniform and homogeneous continuum are not satisfied. Thus the results of Mathematical Psychics turn out to be derivative, not fundamental, indexes, not measurements, first approximations at the best; and fallible indexes, dubious approximations at that, with much doubt added as to what, if anything, they are indexes or approximations of.

The kinds of laws and relations that econom(etr)ics has established, are laws and relations about entities in models that presuppose causal mechanisms being atomistic and additive. When causal mechanisms operate in real world social target systems they only do it in ever-changing and unstable combinations where the whole is more than a mechanical sum of parts. If economic regularities obtain they do it (as a rule) only because we engineered them for that purpose. Outside man-made “nomological machines” they are rare, or even non-existant. Unfortunately that also makes most of the achievements of econometrics – as most of contemporary endeavours of economic theoretical modeling – rather useless.

  1. BC
    August 21, 2014 at 10:07 pm

    ” . . . Unfortunately that also makes most of the achievements of econometrics – as most of contemporary endeavours of economic theoretical modeling – rather useless.”

    The conclusion bears repeating.

    Even before econometrics, one of the first principles of economics is hopelessly flawed, namely, that the ecological system on which all life on the planet depends is assumed to be a sub-set of “the economy” rather than the precise converse.

    Secondly, resource scarcity is said to always be overcome by technological advances, efficiency, and substitution, ignoring net energy (EROEI) and resource services and exergetic flow rates per capita that are now constraining growth or real final sales per capita worldwide with total world petroleum extraction (energy basis for civilization) per capita continuing to decline since 2005-08, i.e., Peak Oil.

    Economics is useful mostly in the sense that one should learn the principles so one can conclude for oneself the flaws and fallacies of the discipline so as to unlearn economics to understand what’s really happening.

  2. davetaylor1
    August 22, 2014 at 11:20 am

    Yes, BC.

    I’ve been going back to the roots of marginalism in Jevon’s 1871 “The Theory of Political Economy”, and finding the mathematics logically flawed: in effect taking logarithms to simplify his calculation but not taking anti-logarithms before interpreting the result, and as a result identifying the value of a labourer with that of the widgets he produces.

    Having Jevons’ 1870 “Elementary Lessons in Logic”, I’ve been studying that to see where he is coming from. This is readable and very good, forming a useful link in the chain of development in the understanding of logic from ancient to recent times, and incidentally providing a fascinating summary of the scientific understanding of the time. The problem is in the economics, where Jevons does not practice what he preaches.

    “The first requisite of a philosophical language”, he says [Lesson XXXIII], is that “every general name must have a certain and knowable meaning”. … General names are a more difficult subject because, as we have seen in Lesson V, they have a double meaning in denotation or extension, and connotation or intension. Of the two meanings the connotation is the one which must be fixed; the other cannot as a general rule be limited and defined”.

    See http://www.fh-jena.de/~kleine/history/languages/Algol68R-UserGuide for how this was achieved a hundred years later, by defining things in terms of types of [internalised] procedures and extent of applicability as parts within parts of multilayered procedures.

    Hence, anyway, Jevons and his successor’s reduction of logic to 1870’s mathematics in their economics: merely assuming their procedures are both defined and relevant, and that “teacher” understands what he himself has learned by example.

  3. August 22, 2014 at 3:03 pm

    ” I cannot but conclude that econometrics on the whole has not delivered ‘truth’, nor robust forecasts.”
    ” The endemic lack of predictive success of the econometric project indicates that this hope of finding fixed parameters is a hope for which there really is no other ground than hope itself.”
    Indeed, all scientifically minded economists would certainly acquiesce, but econometrics goes forth since it is now a wide spread industry. This post should be a required reading for young economists. However, I would still add:
    http:// mpra.ub.uni-muenchen.de/41407 and 46451.

  4. Norman L. Roth
    August 22, 2014 at 4:10 pm

    August 22, 2014

    “BC”…Whoever you may be; I think you’ve had at least some exposure to, and have grasped some of the core content of “TELOS and TECHNOS, The Teleology of Economic Activity and the Origins of Markets.”,,,The 197 page edition ? Or the 256 page one? Or have you simply read the back jacket of the book? It’s not worth it any more to echolalia on & on about what’s wrong with ‘models’. It’s about paradigm supercession. Can’t get away from it ! The modern descendant of Kant’s immortal practical wisdom: [Forgive the translation]

    “Empirical observation without [good] theory is directionless. Theory without empirical observation is hollow”.

    Norman L. Roth, Toronto Canada.
    Please GOOGLE: {1} Origins of Markets, Norman Roth [2] Economics of Technos, Norman Roth {3} telos & technos roth

  5. davetaylor1
    August 23, 2014 at 2:28 pm

    BC says “Economics is useful mostly in the sense that one should learn the principles so one can conclude for oneself the flaws and fallacies of the discipline so as to unlearn economics to understand what’s really happening”.

    http://www.chesterton.org/category/discover-chesterton/chesterton-101/ lecture 3 says:

    “But from the essay [in ‘Varied Types’] on Queen Victoria comes one of Chesterton’s most prophetic and profound lines: “We are learning to do a great many clever things. Unless we are much mistaken the next great task will be to learn not to do them.”

    Try also lecture 35 on shooting the messenger.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.