Home > Uncategorized > Econometric testing

Econometric testing

from Lars Syll

Debating econometrics and its short-comings yours truly often gets the response from econometricians that “ok, maybe econometrics isn’t perfect, but you have to admit that it is a great technique for empirical testing of economic hypotheses.”

But is econometrics — really — such a great testing instrument?

ecokEconometrics is supposed to be able to test economic theories but to serve as a testing device you have to make many assumptions, many of which themselves cannot be tested or verified. To make things worse, there are also only rarely strong and reliable ways of telling us which set of assumptions is to be preferred. Trying to test and infer causality from (non-experimental) data you have to rely on assumptions such as disturbance terms being ‘independent and identically distributed’; functions being additive, linear, and with constant coefficients; parameters being’ ‘invariant under intervention; variables being ‘exogenous’, ‘identifiable’, ‘structural and so on. Unfortunately, we are seldom or never informed of where that kind of ‘knowledge’ comes from, beyond referring to the economic theory that one is supposed to test. Performing technical tests is of course needed, but perhaps even more important is to know — as David Colander recently put it — “how to deal with situations where the assumptions of the tests do not fit the data.”

That leaves us in the awkward position of having to admit that if the assumptions made do not hold, the inferences, conclusions, and testing outcomes econometricians come up with simply do not follow from the data and statistics they use.

The central question is “how do we learn from empirical data?” Testing statistical/econometric models is one way, but we have to remember that the value of testing hinges on our ability to validate the — often unarticulated technical — basic assumptions on which the testing models build. If the model is wrong, the test apparatus simply gives us fictional values. There is always a strong risk that one puts a blind eye on some of those non-fulfilled technical assumptions that actually makes the testing results — and the inferences we build on them — unwarranted.

Haavelmo’s probabilistic revolution gave econometricians their basic framework for testing economic hypotheses. It still builds on the assumption that the hypotheses can be treated as hypotheses about (joint) probability distributions and that economic variables can be treated as if pulled out of an urn as a random sample. But as far as I can see economic variables are nothing of that kind.

I still do not find any hard evidence that econometric testing uniquely has been able to “exclude a theory”. As Renzo Orsi once put it: “If one judges the success of the discipline on the basis of its capability of eliminating invalid theories, econometrics has not been very successful.”

testMost econometricians today … believe that the main objective of applied econometrics is the confrontation of economic theories with observable phenomena. This involves theory testing, for example testing monetarism or rational consumer behaviour. The econometrician’s task would be to find out whether a particular economic theory is true or not, using economic data and statistical tools. Nobody would say that this is easy. But is it possible? This question is discussed in Keuzenkamp and Magnus 􏰄1995􏰀. At the end of our paper we invited the readers to name a published paper that contains a test which, in their opinion, significantly changed the way economists think about some economic proposition. Such a paper, if it existed, would be an example of a successful theory test. The most convincing contribution, we promised, would be awarded with a one week visit to CentER for Economic Research, all expenses paid. What happened? One 􏰄Dutch􏰀 colleague called me up and asked whether he could participate without having to accept the prize. I replied that he could, but he did not participate. Nobody else responded. Such is the state of current econometrics.

Jan Magnus

  1. ghholtham
    March 25, 2020 at 5:58 pm

    The problem is not that theories cannot be rejected by data. It is that economic theorists take no notice of rejections when they occur. It is perverse of Lars to encourage them by attacking one of the few methods we have for empirical testing with a mixture of misunderstanding and impossible counsels of perfection.. It is even more perverse to persistently refuse to explain how he proposes to test economic propositions when he rejects both statistical.analysis and controlled experiment. He joins the neo-classicists in refusing to regard economics as an empirical discipline.

  2. March 25, 2020 at 6:47 pm

    I do think economics ought to be a much more ’empirical’ science than the predominant axiomatic-deductive mainstream ilk. What I do question is the ‘perverse’ privileged position econometrics as a research tool has come to have during the last 30-40 years — especially when considering the rather poor performance of it in terms of producing substantial real-world knowledge. Econometrics certainly is NOT the only approach around (surveys, interviews, case studies, experimentation, economic history, are examples that easily come to my mind) and it certainly is NOT self-evidently applicable to all economic research problems.

    • March 26, 2020 at 7:35 am

      Gerard, in support of Lars for once, the issue as I see it is that there are two concepts of what an ’empirical’ science is: Bacon’s original “take things to bits to see how they work” (and use instruments like x-ray that enable one to see what one cannot otherwise see), and Hume’s “the only things you can know are what (by majority vote) you can agree on seeing”. In the original, econometric testing is just that: what from my own statistical training I recognise as quality control. In the second (and what Lars rightly objects to) it has become not a test but a research tool.

  3. ghholtham
    March 27, 2020 at 2:30 pm

    Dave,

    I think we all agree that a hypothesis has to be framed before being tested and does not emerge from data. There is a subsidiary use of econometrics: developing ways to forecast processes that we don’t understand. I would not glorify that as a research tool but if businesses are compelled to forecast, they tend to use simple moving averages of data series or some form of exponential smoothing. Using a bit of econometrics in that situation leads to somewhat better forecasts. Of course that is just relying on stability(not stationarity) of stochastic time-series which usually fails eventually..

    I’m not sure why Lars thinks econometrics enjoys a privileged position. It is a long time since a Nobel prize was awarded for econometrics. The last one was awarded for careful conduct of pilot studies, a form of experimentation. Mind you, Lars then devoted at least one blog to denouncing that approach as well, in similar terms to his criticism of econometrics – that it can’t be perfect. You can never guarantee that you have taken everything relevant into account or that you are not dealing with a unique sample or circumstance. That is pure nihilism since no empirical method is immune to such criticism.

    Currently eminent economists think they have to ” strike a balance between realism (flexibility), adherence to restrictions from economic theory, and connections between individual behavior and aggregate statistics” (quote). Those restrictions of economic theory are repeatedly rejected when subjected to test, econometric or otherwise. So we are dealing with an establishment that thinks they have to “balance” evidence and failed empirical assumptions that are treated as axioms. To me, that is the principal battleground. Lars needs to decide which side he is on and then recognise friends when he sees them.

  4. March 28, 2020 at 3:58 am

    I’d like to address my comment on the criticisms of econometrics in another way. “The ‘perverse’ privileged position econometrics”, in my view, actually reflects the weakness or decline of economic theories. When economists have little to do with theories, they would inevitably resort to other work — such as econometrics. However, how to revive theories from empirical data? How to revive qualitative analyses from numbers? How to revive verbal narratives from mathematical functions? How to revive the classical humanistic genres of economics from deadly “science”? Criticisms are far from enough, and a constructive and lump-sum answer shall be: Algorithm Framework Theory.

    • March 28, 2020 at 2:38 pm

      “Algorithmic Framework Theory” is a pretty good description of what I wrote a moment ago on “Differences in Theoretical Methodology”. What do YOU mean by the phrase, BinLi? I see you use the word ‘Algorithm’, suggesting you may be thinking of three distinct things/phases as against my linguistic narrowing down of the concept of “theory”. In which case, three illustrative examples?

      Interesting thought: your “When economists have little to do with theories, they would inevitably resort to other work — such as econometrics.”

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.