Home > Uncategorized > Econometrics — formal modelling that has failed miserably

Econometrics — formal modelling that has failed miserably

from Lars Syll

An ongoing concern is that excessive focus on formal modeling and statistics can lead to neglect of practical issues and to overconfidence in formal results … Analysis interpretation depends on contextual judgments about how reality is to be mapped onto the model, and how the formal analysis results are to be mapped back into reality. But overconfidence in formal outputs is only to be expected when much labor has gone into deductive reasoning. First, there is a need to feel the labor was justified, and one way to do so is to believe the formal deduction produced important conclusions. Second, there seems to be a pervasive human aversion to uncertainty, and one way to reduce feelings of uncertainty is to invest faith in deduction as a sufficient guide to truth. Unfortunately, such faith is as logically unjustified as any religious creed, since a deduction produces certainty about the real world only when its assumptions about the real world are certain …

What should we do with econometrics? | LARS P. SYLLUnfortunately, assumption uncertainty reduces the status of deductions and statistical computations to exercises in hypothetical reasoning – they provide best-case scenarios of what we could infer from specific data (which are assumed to have only specific, known problems). Even more unfortunate, however, is that this exercise is deceptive to the extent it ignores or misrepresents available information, and makes hidden assumptions that are unsupported by data …

Despite assumption uncertainties, modelers often express only the uncertainties derived within their modeling assumptions, sometimes to disastrous consequences. Econometrics supplies dramatic cautionary examples in which complex modeling has failed miserably in important applications …

Much time should be spent explaining the full details of what statistical models and algorithms actually assume, emphasizing the extremely hypothetical nature of their outputs relative to a complete (and thus nonidentified) causal model for the data-generating mechanisms. Teaching should especially emphasize how formal ‘‘causal inferences’’ are being driven by the assumptions of randomized (‘‘ignorable’’) system inputs and random observational selection that justify the ‘‘causal’’ label.

Sander Greenland

Yes, indeed, econometrics fails miserably over and over again. One reason why it does — besides those discussed by Greenland — is that the error term in the regression models used are thought of as representing the effect of the variables that were omitted from the models. The error term is somehow thought to be a ‘cover-all’ term representing omitted content in the model and necessary to include to ‘save’ the assumed deterministic relation between the other random variables included in the model. Error terms are usually assumed to be orthogonal (uncorrelated) to the explanatory variables. But since they are unobservable, they are also impossible to empirically test. And without justification of the orthogonality assumption, there is as a rule nothing to ensure identifiability.

Without sound justification of the assumptions made, the formal models used in econometric analysis is of questionable value. Failing to take unmodelled uncertainty (not stochastic risk) into serious consideration has made most econonometricians ridiculously overconfident in the reach of the (causal) inferences they make.

  1. April 19, 2021 at 12:01 am

    Lately how many research papers do not make any sense at all, or just established what sheer common sense would do?
    Do we have a serious mutual admiration club profiteering peer review problem?
    I scratch your back and you scratch mine!

  2. Gerald Holtham
    April 20, 2021 at 5:35 pm

    “But since they (error terms) are unobservable, they are also impossible to empirically test. ”
    They are generated when an attempt is made to fit a model to real data. There they are – entirely observable. Not only are they possible to test but it is obligatory to test them to ensure the model is not nonsense. Orthogonality is one of the things that can and should be tested.
    Aren’t we supposed to learn from our mistakes and not repeat them?

    • April 20, 2021 at 5:39 pm

      Aren’t we confusing errors and residuals here?

  3. Gerald Holtham
    April 21, 2021 at 11:39 am

    Residuals are the differences between actual values of variable and the fitted or predicted values of a model equation. Errors are the difference between forecasts made using the equation, assuming a zero residual and the actual outturn of the variable in question. Both are observable, though forecast errors only so ex post.
    You appear to be asserting the existence of some metaphysical unobservable errors different from either of these. I can’t see the point of such ghostly entities and recommend the application of Occam’s razor.
    When we specify an equation as stochastic we are simply acknowledging that there will be residuals. If the equation is an adequate model, those residuals will have certain characteristics. Since they are observable we can test for the characteristics. Of course, even if the tests are passed, the model may still fail when new data become available or when circumstances change. C’est la vie. It is pointless to deplore things that could not be otherwise.

    • April 21, 2021 at 12:33 pm

      In econometrics, the error term is conceptualized as a THEORETICAL, NON-OBSERVABLE, term that lays behind the difference between an observed value of a variable and the THEORETICAL value we ASSUME in the econometric model. The true values of our model parameters are UNKNOWN and so, of course, we can’t know the value of the error. Residuals, on the other hand, can be observed since they are the estimated differences between our observed variable values and the values we estimate with our econometric model. Sorry Gerry, but I really thought this was common knowledge …

      In one of the textbooks I use in my own econometrics courses – Marno Verbeek’s A Guide to Modern Econometrics, 5th ed, 2017: 15 – it is perhaps more succinctly formulated: “The distinction between the error terms and the residuals is important. Error terms are unobserved …The properties of the error terms and the residuals are not the same and occasionally very different.”

      I hope we have cleared this up now and can move forward in our ongoing discussion to more interesting questions.

    • April 21, 2021 at 12:46 pm

      And in case you think the made distinction is some kind of weird Syll-Verbeek idiosyncrasy, this is what David Freedman writes (Statistical models and causal inference, CUP, 2010:12): “In more standard terminology, the error terms are assumed to be ‘independent and identically distributed, with mean 0’. Such assumptions can present difficult scientific issues, because error terms are not observable.”

  4. Gerald Holtham
    April 21, 2021 at 1:25 pm

    I don’t think we have arrived at clarity quite yet. What you are saying is that when we specify a model the parameters and the residuals will be determined at the same time. Different parameters give different residuals, of course. How do we know the model is adequate and the parameters appropriate? By the character of the residuals.
    What you appear to mean by the errors is an idealisation of the residuals which would have all the properties we would wish the residuals to have. OK but those errors are unobservable because they don’t exist. Implement the model and study the residuals. If they don’t have the right characteristics, the model is most unlikely to be adequate and you must embark on an iterative process of model respecification both of economic content and, perhaps, method of estimation. (In your preferred language, if the residuals don’t have the same properties as your errors the model isn’t good. I’d just say if the residuals don’t have the right properties it’s not good. Why multiply entities unnecessarily?)

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.