## The pitfalls of econometrics

from **Lars Syll**

Ed Leamer’s Tantalus on the Road to Asymptopia is one of my favourite critiques of econometrics, and for the benefit of those who are not versed in the econometric jargon, this handy summary gives the gist of it in plain English:

Most work in econometrics and regression analysis is made on the assumption that the researcher has a theoretical model that is ‘true.’ Based on this belief of having a correct specification for an econometric model or running a regression, one proceeds as if the only problem remaining to solve have to do with measurement and observation.

When things sound to good to be true, they usually aren’t. And that goes for econometric wet dreams too. The snag is, as Leamer convincingly argues, that there is pretty little to support the perfect specification assumption. Looking around in social science and economics we don’t find a single regression or econometric model that lives up to the standards set by the ‘true’ theoretical model — and there is pretty little that gives us reason to believe things will be different in the future.

To think that we are being able to construct a model where* all* relevant variables are included and *correctly* specify the functional relationships that exist between them, is not only a belief without support, but a belief *impossible* to support. The theories we work with when building our econometric models are insufficient. No matter what we study, there are always some variables missing, and we don’t *know* the correct way to functionally specify the relationships between the variables we choose to put into our models.

*Every* econometric model constructed is misspecified. There are always an endless list of possible variables to include, and endless possible ways to specify the relationships between them. So every applied econometrician comes up with his own specification and parameter estimates. The econometric Holy Grail of consistent and stable parameter-values is nothing but a dream.

A rigorous application of econometric methods presupposes that the phenomena of our real world economies are ruled by stable causal relations between variables. Parameter-values estimated in specific spatio-temporal contexts are *presupposed* to be exportable to totally different contexts. To warrant this assumption one, however, has to convincingly establish that the targeted acting causes are stable and invariant so that they maintain their parametric status after the bridging. The endemic lack of predictive success of the econometric project indicates that this hope of finding fixed parameters is a hope for which there really is no other ground than hope itself.

The theoretical conditions that have to be fulfilled for regression analysis and econometrics to really work are *nowhere* even closely met in reality. Making outlandish statistical assumptions does not provide a solid ground for doing relevant social science and economics. Although regression analysis and econometrics have become the most used quantitative methods in social sciences and economics today, it’s still a fact that the inferences made from them are, strictly seen, invalid.

Econometrics is basically a deductive method. Given the assumptions (such as manipulability, transitivity, separability, additivity, linearity, etc) it delivers deductive inferences. The problem, of course, is that we will *never* completely know when the assumptions are right. Conclusions can only be as certain as their premises. That also applies to econometrics.

I like your Leamer-based Pitfalls of Empirics diagram. Much better than long-winded op-eds and journal articles (often not accessible) for encouraging students, young professionals, and emerging civic leaders to get their feet wet with the real essence of what econ and related fields can offer. I hope that you and others in this circuit feel encouraged to do more of such.

Lars Sylls has discovered Karl Popper. No general statement is verifiable; all scientific “laws” are provisional and are held until refuted. No generalisation can exhaustively list all the conditions under which it holds and no formulation of a relationship can be sure that all potentially relevant variables have been controlled for. This is as true in economics as in any other discipline with the added problem that the system is evolving faster than biological or physical systems.

Testing whether a formulation is consistent with past data (including testing error independence etc) is a necessary but not, of course, a sufficient condition for its being useful to describe or project future data. What is the alternative: NOT testing a formulation on past data? As the old war cartoon put it: if someone points out a better hole I’ll go to it. I don’t think Lars has a better hole; he is a methodological nihilist.

“What is the alternative: NOT testing a formulation on past data?”

No, Gerald: testing the logical justification of the formula. Lars portrays perhaps more intelligibly than I did the problem calling for Kuhn’s “revolutionary science” which I addressed in our discussion of “‘cats and dogs”:

“Most work in econometrics and regression analysis is made on the assumption that the researcher has a theoretical model that is ‘true.’ Based on this belief of having a correct specification for an econometric model or running a regression, one proceeds as if the only problems remaining to solve have to do with measurement and observation”.

What if it is NOT ‘true’? What if it portrays the economy as stationary, or as changing at a constant speed? How can it be ‘true’ into the future if it fails to take account of accelerations, the effect of diversions, and the passage of time? One looks for a theory which does: which won’t give you singular answers but will offer you guidance on where to look and a paradigm example one can actually look at, i.e. navigation, in which progress is plotted not along a line but on a complex number plane. The theory of Cybernetics has been around for 70 years, but even heterodox economists show no sign of having noticed it.

Your suggestion of stationary can not be found in this way. All they produce is a concrete equation describing the empirical data. When a theoretically valid relationship is used the difference is enormous as can be seen in my paper “Transient Development”, RWER-81.

You must be an econometrician Gerald. They are apparently good at jumping foxholes (i.e., from hedge fund finance to banking stock market casinos to commodities market manipulations to …) for their bread and butter.

Until economists find relationships which conform to the quantity calculus, they will achieve nothing.

Until economists and economic pundits start thinking and perceiving on the level of the paradigm/pattern they will rely upon the lesser level of analysis and insight of econometrics.

Excellent. Lars. I find the “counters” to your presentation interesting. Some form of refutationism is offered as a solution. As is “new” mathematics, or just a reworking of the old mathematics. Or, just a new theory. Bottom line, however is that there is an endless array of data that can be fitted to an endless array of theories. Choosing among the options is a judgment humans make. For all the subtle, uncertain, unspecifiable, and emotional reasons humans always make the final decision. Even the ones about the extinction of their species.

Does a 50% discount to consumers on every consumer item from a package of chewing gum to autos to a house…every cent of which is rebated back to the retail enterprise giving the discount by the monetary authority…not immediately double the earned income/purchasing power of everyone, double the free and clear money actually available for every enterprise’s goods and services and beneficially integrate price and asset deflation into profit making economic systems?

Please, anyone here….yes or no? Don’t be afraid. Let’s have your answers.

Economic theories normally imply restrictions on the parameters of an equation, albeit loose ones. A variable may be posited to have an effect or not to have an effect. Econometrics can be used to test whether the theoretical restrictions are met. If they are not met the theory fails. If they are met econometrics can estimate the size of the parameter falling within the theoretical range. There is no assurance that the parameter will be stable or that the theoretical restrictions will continue to be met in other times and places. Fortunately there is enough inertia in human affairs that such equations can prove useful for a time. In general the more elaborate the theory the less robust is the equation. As KZ says, there are usually more theories than one that can pass any such test.

Elaborate theories that depend on assuming optimisation by economic agents generally ignore aggregation problems and underplay the effects of uncertainty. . I understand the impatience that some people feel when some economists devote a lot of resource to working out the implications of individual choice theory under certainty equivalence and then try and impose the result on a model of the macroeconomy. It is indeed implausible to think the economy behaves “as if” there is a representative consumer optimising consumption patterns over a forecast lifetime. It is not implausible, however, to suppose that if people’s income goes up they will spend more. The extent to which they spend more will depend on a lot of things some of which we can control for and others we can’t. But simply dealing with large numbers of people tends to confer its own temporal stability as uncontrolled effects can net out across a large population. This is all far short of “proving” a general theory but it is better than sucking propositions out of your thumb and refusing to test them because of arcane scruples about statistical theory..

Gerald, theories have three sources, experience, imagination, or more commonly experience and imagination combined. But none of the three has to necessarily be related directly to the subject of the theory. For example an economist might conclude a theory that strays too far from the dominant accepted theories of the field might be harmful to their future status and salary in the profession. That’s the first thing that needs to be sorted. Second, efforts to “test” any theory are subject to the same concerns, but also to one other. Specifying the theory in testable terms. Sometimes the term “operationalization” is used to summarize this process. Depending on the theory, its history, and the team or teams doing the testing, these steps may involve dozens or even hundreds of decisions (judgments). Each of which is a choice among different options, which may be in terms of testing near equally applicable. To say this plays into the hands of uncertainty and can be challenged for good and ill reasons is an understatement. Next, tools must be selected for the testing. And agreements reached on how the data from each tool is to be “read” and used. Finally, since it’s unlikely the results from the application of each tool will be identical, how are differences to be reconciled? And which differences are non-confirmation of our theory and which are not? No matter how you word these steps, scientists, even economists are often “flying by the seats of their pants” in this work. The best we can do is the best we can do. And expect challenges on the flying and the pants. The bigger danger is there won’t be enough challenges. Which is often the case with statistical tools. In other words, in the culture of some social sciences, statistics has become tacitly accepted.