Home > Uncategorized > Econometrics — fictions masquerading as science

Econometrics — fictions masquerading as science

from Lars Syll

rabbitIn econometrics one often gets the feeling that many of its practitioners think of it as a kind of automatic inferential machine: input data and out comes casual knowledge. This is like pulling a rabbit from a hat. Great — but first you have to put the rabbit in the hat. And this is where assumptions come into the picture.

As social scientists — and economists — we have to confront the all-important question of how to handle uncertainty and randomness. Should we equate randomness with probability? If we do, we have to accept that to speak of randomness we also have to presuppose the existence of nomological probability machines, since probabilities cannot be spoken of – and actually, to be strict, do not at all exist – without specifying such system-contexts.

Accepting a domain of probability theory and a sample space of “infinite populations” — which is legion in modern econometrics — also implies that judgments are made on the basis of observations that are actually never made! Infinitely repeated trials or samplings never take place in the real world. So that cannot be a sound inductive basis for a science with aspirations of explaining real-world socio-economic processes, structures or events. It’s not tenable.

In his book Statistical Models and Causal Inference: A Dialogue with the Social Sciences David Freedman touches on this fundamental problem, arising when you try to apply statistical models outside overly simple nomological machines like coin tossing and roulette wheels:

61TOS8PyfLLRegression models are widely used by social scientists to make causal inferences; such models are now almost a routine way of demonstrating counterfactuals. However, the “demonstrations” generally turn out to depend on a series of untested, even unarticulated, technical assumptions. Under the circumstances, reliance on model outputs may be quite unjustified. Making the ideas of validation somewhat more precise is a serious problem in the philosophy of science. That models should correspond to reality is, after all, a useful but not totally straightforward idea – with some history to it. Developing appropriate models is a serious problem in statistics; testing the connection to the phenomena is even more serious …

In our days, serious arguments have been made from data. Beautiful, delicate theorems have been proved, although the connection with data analysis often remains to be established. And an enormous amount of fiction has been produced, masquerading as rigorous science.

  1. ghholtham
    February 11, 2020 at 2:07 pm

    “In econometrics one often gets the feeling that many of its practitioners think of it as a kind of automatic inferential machine: input data and out comes casual knowledge.”

    Please name names. Who or what gives you this feeling? Or have some shame about repeating this generalised calumny. No-one I know thinks like that. The researcher proposes, and data disposes.

  2. February 11, 2020 at 2:23 pm

    No, no, no, Gerry. The only one who should have some shame here is you. In the excellent journal that hosts this blog, Judea Pearl and Bryant Chen has already done what you ask for — and give plenty of names! So do stop repeating that no-one “thinks like that”. Read the article:

  3. Yoshinori Shiozawa
    February 11, 2020 at 5:45 pm

    Lars, why do you intervene in this time? You have kept silent for many other important (at least in my understanding) objections. If you keep this new attitude, it is a change that should be recommended. If you post your articles in this blog, it should be interactive dialogue. I am not asking you to respond all Relies. Perhaps you may choose two or three good Replies to which you answer.

    • Meta Capitalism
      February 12, 2020 at 9:12 am

      A blog is not the same thing as a seminar or online WEA conference. There is no expectation that the authors of a blog will either post your comment or respond to comments thereon.

      • Yoshinori Shiozawa
        February 12, 2020 at 6:34 pm

        If this RWER blog is a personal one, Meta Capitalism must be right. But isn’t this blog a public forum?

      • Meta Capitalism
        February 12, 2020 at 10:31 pm

        [I]sn’t this blog a public forum?

        I am not sure exactly what you mean by public forum. If you mean anyone on the Internet can read it, well, sure. But it is not a Town Hall where people come expecting to have a dialogue with the speaker or a question and answer session.
        The bottom line is your expectation that any author on this blog “should” dialogue is off base, simply not what I have experienced on any blog. That is not what blogs are all about. They are nothing more than a place for authors to post their viewpoints and opinions.
        You can comment with your viewpoint and opinion, but don’t expect or demand an answer from the author. That is not how blogs work.

      • Yoshinori Shiozawa
        February 13, 2020 at 4:03 am

        I have learned much that we cannot expect much from this blog site. I had a much more expectation when I first came to know the existence of this blog, because it is a part of RWER movement.

        As you know well, RWER started as a movement of economics students (2000 in France) who demanded more pluralistic economics education for their curriculum. The movement inspired economics professors and RWER was organized by those economists (with other activities). I wonder why this RWER blog (not the REWR itself) does not work as a forum to envision various possibilities that may contribute to the re-organization and re-building of economics. The reality is far from what I have expected. It is shame that this blog does not work as an open forum for the purpose that I have expected.

        I came to understand that reconstruction of economics is very difficult, not because of the inertia of mainstream economists, but also of bad customs and attitudes among those who ask renovation of economics.

      • Meta Capitalism
        February 13, 2020 at 4:27 am

        Have you given even the slightest thought Shiozawa how you have in your own comments engaged in “bad customs and attitudes”?

      • Yoshinori Shiozawa
        February 13, 2020 at 1:44 pm

        Thank you. We have to reflect on it.

      • Meta Capitalism
        February 13, 2020 at 1:59 pm

        That is hopeful. Now I see some humility and zange.

  4. Meta Capitalism
    February 12, 2020 at 8:39 am

    If you post your articles in this blog, it should be interactive dialogue. ~ Shiozawa Being Ridiculous

    It’s a blog, not a seminar, your being ridiculous. You have a penchant for the ludicrous.

  5. ghholtham
    February 12, 2020 at 1:42 pm

    Lars, nothing in the article by Pearl and Chen justifies your opening sentence. Perhaps you should read it again yourself,. more carefully. Econometric analysis can be used for calibrating mere forecasting equations, which cannot be interpreted as “structural” or a basis for policy intervention. Alternatively, it can be used to test (though not “prove”) explicitly causal hypotheses, where error terms are interpreted as the effect of omitted influences. Pearl and Chen complain that econometric texts do not always clarify which activity is being discussed. There is no example given of a researcher inferring causation from a statistical relationship without benefit of a prior causal hypothesis, let alone “many”. Such an activity is known in the trade as data snooping, fishing or data mining – activities regarded with general disapproval.

  6. ghholtham
    February 12, 2020 at 1:55 pm

    Meta, I often agree with you, sometimes not, but I have noticed a deterioration in your mood recently leading to n increase in unnecessary rudeness.
    Authors should not respond to abuse but if salient points are made in a reasonable way I think there is some expectation that they should address them, either in correspondence or in a subsequent post. Otherwise we have a dialogue of the deaf.

  7. ghholtham
    February 12, 2020 at 2:29 pm

    My difference with Lars can be summarised briefly from my point of view. We agree that there are too-frequent examples of bad practice where econometrics is used as top-dressing for some unsubstantiated position. In other words, crime exists. Lars appears to go from there to argue that crime is inevitable; econometrics is somehow illegitimate and cannot be used to advance knowledge. There I cannot follow. No general causal hypothesis can be proven by analysis of a finite data set; that’s a logical inevitability not a fault of econometrics. But such a hypothesis can be disproven. An example is David Hendry’s disproof of key propositions by Friedman and Schwartz that undermined the claimed empirical basis for monetarism. The fact that economic theoreticians ignore disproofs that econometrics provides is their fault, not that of the econometricians. If econometrics results were taken seriously much of economic theory would collapse and be dismissed as irrelevant to the world in which we live.
    Lars is fighting, unnecessarily, on two fronts. He wants economics to be more empirical. I wholeheartedly agree. Yet he wants to dismiss the necessarily imperfect techniques we have for empirical testing and discrimination. He wants us instead to, in his words, confront causality. I admit to not knowing exactly what he means by that if we can’t do it either by attempting controlled experiments or careful statistical analysis. I hope he will enlighten me.

    • February 12, 2020 at 2:53 pm

      Gerry, one of the founding fathers of modern econometrics, Trygve Haavelmo, once wrote:
      “What is the use of testing, say, the significance of regression coefficients, when maybe, the whole assumption of the linear regression equation is wrong?”
      Real-world social systems are usually not governed by stable causal mechanisms or capacities. The kinds of ‘laws’ and relations that econometrics has established, are laws and relations about entities in models that presuppose mechanisms and variables — and the relationship between them — being linear, additive, homogenous, stable, invariant and atomistic. But — when those mechanisms operate in the real world they only do it in ever-changing and unstable combinations where the whole is more than a mechanical sum of parts. Since statisticians and econometricians haven’t been able to convincingly warrant their assumptions of homogeneity, stability, invariance, independence, additivity, and so on, as being ontologically isomorphic to real-world economic systems, I have to admit I still think, as Keynes put it in a letter to Ragnar Frisch, “nothing emerges at the end which has not been introduced expressively or tacitly at the beginning.” So — I remain doubtful of the scientific aspirations of econometrics. And I certainly also remain doubtful of econometrics as a “testing instrument”. To my knowledge, not a single economic hypothesis has been irrefutably falsified by an econometric regression.

      • February 12, 2020 at 3:07 pm

        Agreeing with Lars, this seems to me another argument for fundamental science being about working out where to look, not about providing ready-made answers.

  8. ghholtham
    February 13, 2020 at 1:03 pm

    I think we are getting somewhere. “Irrefutably” is a big word. Popperian refutation can always be resisted by throwing in auxiliary hypotheses but effective refutation is possible. I have already mentioned Hendry’s demolition of monetarism. Here’s another: take any DSGE model and ask the question: does its mean squared error for forecasts out of sample beat that of its own unrestricted reduced form? The answer is always no. In my world that means the restrictions on coefficients implied by the so-called structural model are not consistent with data. That should be the end of the model and “rational” expectations based on it. How can it be rational to believe a model that is inferior in forecasting to its unrestricted reduced form? The refutation is there; it just needs to be taken seriously.

    I do not know what laws and relations economics can be said to have established but it surely can’t have established them if they are not perceptible in data. You seem perilously close to saying that economic generalisations can not be tested at all conclusively. Does that mean we can never resolve disagreements?.

    It is true that hypothesized relations have to be given a degree of concreteness or specificity to be tested. It is true then that it is the specific form of the hypothesis that is being tested not something vaguer but then what is economics for? If it is to be quantitative and used for policy or conditional forecasting, vague relations have to be made more specific. The more specific you get and the more you are explicit about other conditioning variables, the greater the risk of over-fitting to a particular historical episode with subsequent instability – yet the more testable you are. The more complex the model, though, the more you are testing joint hypotheses not a single one and that gives the theoreticians room to duck and dive. I think you are supposed to push the boundaries and see how much you can say, rather than reside with generalisations so vague as to be quite untestable. And we have to accept that in an evolving system not much will be stable for ever.

    Finally, Haavelmo wrote a long time ago. We do not have to suppose that relations are linear. We have non-nested tests for functional form and can specify non-linear relations. The only limit on the intricacy of hypothesized functions is data – you have to have enough data to identify the parameters. We are supposed to test for independence of variables and there are various ways to tackle systems of related variables using instrumental variables or simultaneous equations approaches. There are numerous ways to test for stability. We have tests for spurious correlation of trending series and where there is two-way causation we have tests for causal ordering. The subject has moved on since 1938. But on one point we agree – none of that generates a hypothesis. The causal hypothesis logically precedes any econometric test and nothing can be tested that has not been hypothesized. Finding correlations in data never establishes causation. I do not believe in induction, abduction, retroduction or any such fanciful notion.,

    Sometimes you can find forecasting equations that are stable for a time, even though there is no causal model underlying them. They are pure exercises in plotting conditional probabilities and you may not know why. That’s a different exercise – and has its uses. I have made money for clients by deploying it, which is how our commercial society values activities. I do not pretend that that advances science.

    • Meta Capitalism
      February 13, 2020 at 1:12 pm

      You could of fooled me! But that is not your fault :-)

  9. Yoshinori Shiozawa
    February 15, 2020 at 4:39 am

    As a result, heterodox economics is perceived as centred on
    criticising given assumptions typical of mainstream economics and emphasizig
    their weaknesses and failures, rather than on displaying any aspirations to create an alternative influential economic approach. That is why heterodox economics acquired much less appreciation and prestige. It is commonly treated as an attitude of secondary importance, incapable of constructing an alternative research
    program better than that offered by mainstream economics.

    Citation from Izabela Bludnik (2015) Keynesianism: mainstream economics or heterodx economics?

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.