Home > Uncategorized > What should we do with econometrics?

What should we do with econometrics?

from Lars Syll

Econometrics … is an undoubtedly flawed paradigm. Even putting aside the myriad of technical issues with misspecification and how these can yield results that are completely wrong, after seeing econometric research in practice I have become skeptical of the results it produces.

deb6e811f2b49ceda8cc2a2981e309f39e3629d8ae801a7088bf80467303077bReading an applied econometrics paper could leave you with the impression that the economist (or any social science researcher) first formulated a theory, then built an empirical test based on the theory, then tested the theory. But in my experience what generally happens is more like the opposite: with some loose ideas in mind, the econometrician runs a lot of different regressions until they get something that looks plausible, then tries to fit it into a theory (existing or new) … Statistical theory itself tells us that if you do this for long enough, you will eventually find something plausible by pure chance!

This is bad news because as tempting as that final, pristine looking causal effect is, readers have no way of knowing how it was arrived at. There are several ways I’ve seen to guard against this:

(1) Use a multitude of empirical specifications to test the robustness of the causal links, and pick the one with the best predictive power …

(2) Have researchers submit their paper for peer review before they carry out the empirical work, detailing the theory they want to test, why it matters and how they’re going to do it …

(3) Insist that the paper be replicated. Firstly, by having the authors submit their data and code and seeing if referees can replicate it (think this is a low bar? Most empirical research in ‘top’ economics journals can’t even manage it). Secondly — in the truer sense of replication — wait until someone else, with another dataset or method, gets the same findings in at least a qualitative sense …

All three of these should, in my opinion, be a prerequisite for research that uses econometrics (and probably statistics more generally … Naturally, this would result in a lot more null findings and probably a lot less research. Perhaps it would also result in fewer attempts at papers which attempt to tell the entire story: that is, which go all the way from building a new model to finding (surprise!) that even the most rigorous empirical methods support it.

Unlearning Economics

Good advise, underlining the importance of never letting our admiration for technical virtuosity blind us to the fact that we have to have a cautious attitude towards probabilistic inferences in economic contexts.  

Science should help us disclose causal forces behind apparent ‘facts.’ We should look out for causal relations, but econometrics can never be more than a starting point in that endeavour since econometric (statistical) explanations are not explanations in terms of mechanisms, powers, capacities or causes. Firmly stuck in an empiricist tradition, econometrics is only concerned with the measurable aspects of reality. But there is always the possibility that there are other variables – of vital importance and although perhaps unobservable and non-additive, not necessarily epistemologically inaccessible – that were not considered for the model. Those who were can hence never be guaranteed to be more than potential causes, and not real causes. A rigorous application of econometric methods in economics really presupposes that the phenomena of our real-world economies are ruled by stable causal relations between variables. A perusal of the leading econom(etr)ic journals shows that most econometricians still concentrate on fixed parameter models and that parameter-values estimated in specific spatiotemporal contexts are presupposed to be exportable to totally different contexts. To warrant this assumption one, however, has to convincingly establish that the targeted acting causes are stable and invariant so that they maintain their parametric status after the bridging. The endemic lack of predictive success of the econometric project indicates that this hope of finding fixed parameters is a hope for which there really is no other ground than hope itself.

Real world social systems are seldom governed by stable causal mechanisms or capacities. The kinds of ‘laws’ and relations that econometrics has established, are laws and relations between entities in models that presuppose causal mechanisms being atomistic and additive. When causal mechanisms operate in real-world social target systems they only do it in ever-changing and unstable combinations where the whole is more than a mechanical sum of parts. If economic regularities obtain they do it (as a rule) only because we engineered them for that purpose. Outside man-made ‘nomological machines’ they are rare, or even non-existent. Unfortunately, that also makes most of the achievements of econometrics – as most of the contemporary endeavours of mainstream economics – rather useless.

Maintaining that economics is a science in the ‘true knowledge’ business, yours truly remains a skeptic of the pretences and aspirations of econometrics. So far, I cannot see that it has yielded much in terms of relevant, interesting economic knowledge. Over all the results have been bleak indeed.

  1. Edward Ross
    January 22, 2018 at 2:11 am

    Lars Sill’s “Reading an applied econometrics paper could leave you with the impression that the economic (0r any social science researcher first formulated a theory , then built on empirical test based on theory, then tested the theory. But in my experience what generally happens is the opposite;”

    Before going further I am not an economist or academic rather I am only a mature age BA graduate who studied archaeology, anthropology and third world development. My reason for replying to your blog is firstly because your blog resonates with my experiences as a volunteer in Papua New Guinea and secondly because although some economists are acknowledging that without involving and understanding people economics would be irrelevant to the real world. Thus I find it encouraging to see that some economists are advocating the importance of including anthropology in the conversations and have actually done some anthropological study including field work. Then returning to your above quote the question is did they do the field work before theory or after theory in which case in spite of the best intentions theory cold have formed a subconscious theoretical bias.

    Now having raised this question I recall my experience as a volunteer in Papua New throughout the 1970s where my wife and I worked on helping the people of an isolated area to learn how to raise cattle and grow soy and mung beans in order to improve their chronic health problems. Obviously this required us to learn pidgin English the main language for expatriates to communicate with the local people as well as between some of the local language groups.
    Thus our challenge was without any knowledge of anthropology we knew that the key to helping the people was to establish a working relationship with them where we both could respect one and other. To this end we lived in an old mission built bush house at the edge of a main village where my wife ran an aid post and we had a primary school with one coastal and one local teacher. Then during the day I worked with some of the local young men clearing the forest for the agricultural. Furthermore in those early years my wife and I seldom saw English speaking visitors because there was only walking tracks that could sometimes be negotiated by motor bicycle if it was not to wet. Because of our closeness to the village the school and the aid post at the back of the house we were reasonably aware of some of the events in the village as they were aware of us. here I would to reiterate from experience that even with our involvement with our village and other nearby villages in some aspects the peoples culture was not static while in other aspects many traditional beliefs did not change.
    For example it appeared that the Australian Government followed Margaret Mead or her disciples views that witchcraft and sorcery had disappeared and yet even in our early days it was not until you really got to know the people that you became aware that these practises simply went underground as proved after independence particularly in the Highlands where there some traditional killing of presumed witches. Another criticism were that many of her assumptions were based on lack of in depth information, because of limited understanding of local language and pidgin English but also local villages colloquial use of those languages. As well the use of informants from my wife and my experience it was not very helpful to answer a direct question because invariably you were given the answer on what the person being questioned wanted you to hear. For example one day we were having I cup of tea with a patrol officer and the young fellow who stoked the fire was listening to our conversation in pigin, when the patrol officer asked who was the leader in the cargo cult. The young fellow quickly replied ‘me no savey’ I do not know, so I said all e ‘ calim’ Stephan the young fellows name, quick as a flash he gave the culprits name to clear his own name. As for the use of informants particularly the young they were neither simple or stupid , in fact when you got to know some of them they were instantly able ascertain the physiological attitude of the patrol officer. The proof of this was if they were in a situation where there was no fear of any reprisal they would have great fun ridiculing the victim. Thus from experience I am sure that Margeret Meads young female informants sensed her sensual attitude and interest and must have had great fun feeding her fantasies.

    I give another example of terrible consequences of the lack of understanding of the peoples culture can have. This personally observed story is about a n extremely well intentioned young Canadian couple who may or may not have received any anthropological information before becoming volunteers. The young woman was given the task of supervising the trade store that suppled the local teachers and government employees where there was already a good young high school graduate serving in the store. The Canadian young woman not knowing the local culture, ignored any advice not to give the young store attendant the keys to the store an the safe because it would place him in the situation where he could not refuse his people the benefits of his position. Ultimately the store collapsed and a good young fellow was physiologically destroyed. In some areas of the West Sepik this sort of shame was sufficient to cause the shamed to commit suicide.

    Now moving on from my PNG experiences I began my mature age studies at Deakin University where I found that the whole anthropology department very informative fleshing out my understanding. Here I will take a few lines of Bill Geddes from a larger important introduction;
    “This distinctively Western view of the nature of human activity and community plays down the significance of ethnic identity. It is assumed that people are primarily self-interested, acquisitive individuals who will readily adopt new ways of behaving and organising life if these enhance their material wealth.— No body is, ever , an isolated individual with his or own , privately and independently developed sets of understandings about the world and about self. Rather. each person thinks and acts in terms of understanding learned from the community in which he or she lives. Bill Geddes(19940) Ethnicity and history Zaire and Europeans cited Anthropology and Third World Development

    The whole point of this attempt to describe why I am convinced of the impotence of a first hand experience of culture before making what may amount to what may amount to false assumptions about peoples culture. Furthermore I am aware that from my experience Margaret Mead have missed the boat on culture but I some ways was a victim of circumstance’s in that if she ignored her tutor Fanz Boaz and his connections to Malinofsky and Darwin she might not get her PhD. What I find interesting here is that this describes Lars Syll,s account of what amounts to beginning with theory and then adjusting empirical evidence to suit that theory.
    Now connected to this I add Dave Taylor January 17, 2018at 9;38pm” Yes why invest social science at all, when it tells you nothing you can’t see better with your own eyes with their kinesthetic sensing”
    . Here i respectfully ask is this guy for real or is he fishing for a repose because if you put this question to most members of the public they would most likely say many people may view a subject but come up with different descriptions, Or is he afraid that the social sciences may challenge his theories.

  2. Edward Ross
    January 23, 2018 at 8:27 pm

    answer a direct question obviously should have been ask. Another a couple of months ago Ken Zimmerman wrote “for example in hunter gatherer societies (in which Homo Sapiens have lived during 90% of their history on earth )all knowledge is shared between all members of the group.”
    Here I accept that you may have omitted to write more on a short response, my comment is firstly is did you intend to imply that this indicated some sort of democracy in those early societies. Secondly in our first assignment the people were in a transitional stage between hunter gather and gardeners for example a large part of their existence was from hunting and gathering and their idea of a garden was to cut down a couple of trees and plant in the debris. IN this situation from observation and experience I agree with your above statement however what I would like to add is that the village was definitely not run in a democratic manner. It was run by self appointed powerful big men enlisted the support of the spirit or magic men. Thus fear of offending the spirits played a large part in how the village functioned it also created a situation where individual village people were reluctant to seek to improve themselves above the rest of the village .

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s