Home > Uncategorized > The tools economists use

The tools economists use

from Lars Syll

In their quest for statistical “identification” of a causal effect, economists often have to resort to techniques that answer either a narrower or a somewhat different version of the question that motivated the research.

Policy Methods Toolbox - Observatory of Public Sector Innovation  Observatory of Public Sector InnovationResults from randomized social experiments carried out in particular regions of, say, India or Kenya may not apply to other regions or countries. A research design exploiting variation across space may not yield the correct answer to a question that is essentially about changes over time …

Economists’ research can rarely substitute for more complete works of synthesis, which consider a multitude of causes, weigh likely effects, and address spatial and temporal variation of causal mechanisms. Work of this kind is more likely to be undertaken by historians and non-quantitatively oriented social scientists …

Economists would not even know where to start without the work of historians, ethnographers, and other social scientists who provide rich narratives of phenomena and hypothesize about possible causes, but do not claim causal certainty.

Economists can be justifiably proud of the power of their statistical and analytical methods. But they need to be more self-conscious about these tools’ limitations. Ultimately, our understanding of the social world is enriched by both styles of research. Economists and other scholars should embrace the diversity of their approaches instead of dismissing or taking umbrage at work done in adjacent disciplines.

Dani Rodrik

As Rodrik notes, ‘ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. Causes deduced in an experimental setting still have to show that they come with an export-warrant to their target populations.

the-right-toolThe almost religious belief with which its propagators — like 2019’s ‘Nobel prize’ winners Duflo, Banerjee and Kremer — portray it, cannot hide the fact that randomized controlled trials, RCTs, cannot be taken for granted to give generalisable results. That something works somewhere is no warranty for us to believe it to work for us here or even that it works generally.

Believing there is only one really good evidence-based method on the market — and that randomisation is the only way to achieve scientific validity — blinds people to searching for and using other methods that in many contexts are better. Insisting on using only one tool often means using the wrong tool.

‘Randomistas’ like Duflo et consortes think that economics should be based on evidence from randomised experiments and field studies. They want to give up on ‘big ideas’ like political economy and institutional reform and instead go for solving more manageable problems the way plumbers do. But that modern time ‘marginalist’ approach sure can’t be the right way to move economics forward and make it a relevant and realist science. A plumber can fix minor leaks in your system, but if the whole system is rotten, something more than good old fashion plumbing is needed. The big social and economic problems we face today is not going to be solved by plumbers performing RCTs.

  1. A.J. Sutter
    April 26, 2021 at 2:46 am

    Apropos of “Economists can be justifiably proud of the power of their statistical and analytical methods.”: is “power” apt, or is it something more like technical sophistication?

  2. Gerald Holtham
    April 26, 2021 at 7:00 pm

    A.J.S : power is indeed the wrong word. Statistical tests in economics often have low power to discriminate or conclusively reject hypotheses because social reality is complicated and the data available are usually sparse or error-ridden or otherwise deficient. For that reason statistical work in the social sciences has had to develop techniques to extract as much information as possible from imperfect data whereas scientists in “hard” subjects have the better option of redesigning or re-running an experiment to get better data. That sophistication in developing statistical theory is probably what Rodrik is referring to. One must agree with his call for a diversity of approaches. All real-world situations, even if we think they instantiate general tendencies, will have their own specificities and it is more than helpful to know about those when analysing the situation or events..

  3. Ken Zimmerman
    May 2, 2021 at 4:59 am

    Missing from the usual account scientists present of their work is any reference to the judgments entailed in the production of scientific knowledge-judgments relating to the acceptability of experimental, randomized survey, observational, etc. data as facts about natural, social, anthropologicial phenomena, and judgments relating to the plausibility of theories. But this lack is only apparent. The scientist’s account avoids any explicit reference to judgments by retrospectively adjudicating upon their validity. By this I mean the following.
    Theoretical entities like molecules, and conceptualisations of natural phenomena like the weak neutral current, are in the first instance theoretical constructs: they appear as terms in theories elaborated by scientists. However, scientists typically make the realist identification of these constructs with the contents of nature, and then use this identification retrospectively to legitimate and make unproblematic existing scientific judgments. Thus, for example, the experiments which discovered the weak neutral current are now represented in the scientist’s account as closed systems just because the neutral current is seen to be real. Conversely, other observation reports which were once taken to imply the non-existence of the neutral current are now represented as being erroneous: clearly, if one accepts the reality of the neutral current, this must be the case. Similarly, by interpreting quarks and so on as real entities, the choice of quark models and gauge theories is made to seem unproblematic: if quarks really are the fundamental building blocks of the world, why should anyone want to explore alternative theories? Social scientists engage in the same charade. Social classes are taken as real before scientists begin to reveal their outlines and features. Otherwise research on them would be judged as unreal by scientists.

    Most scientists think of it as their purpose to explore the underlying structure of reality, and it therefore seems quite reasonable for them to view their history in this way. But from the perspective of the historian the realist idiom is considerably less attractive. Its most serious shortcoming is that it is retrospective. One can only appeal to the reality of theoretical constructs to legitimate scientific judgments when one has already decided which constructs are real. And consensus over the reality of particular constructs is the outcome of a historical process. Thus, if one is interested in the nature of the process itself rather than in simply its conclusion, recourse to the reality of real phenomena and theoretical entities is self-defeating.

    How is one to escape from retrospection in analysing the history of science? To answer this question, it is useful to reformulate the objection to the scientist’s account in terms of the location of agency in science. In the scientist’s account, scientists do not appear as genuine agents. Scientists are represented rather as passive observers. The Prehistory of High-Energy Physics of nature: the facts of natural reality are revealed through experiment; the experimenter’s duty is simply to report what he sees; the theorist accepts such reports and supplies apparently unproblematic explanations of them. The facts of social class are revealed through randomized surveys or direct observation. From which theoreticians create unproblematic theories ‘explaining’ social class reality. One gets little feeling that scientists actually do anything in their day-to-day practice. Inasmuch as agency appears anywhere in the scientist’s account it is ascribed to real phenomena which, by manifesting themselves through the medium of experiment, survey, etc. somehow direct the evolution of science. Seen in this light, there is something odd about the scientist’s account. The attribution of agency to inanimate matter or humans canceled by institutions rather than to human actors is not a routinely acceptable notion. It is my view that agency belongs to actors not phenomena: scientists make their own history, they are not the passive mouthpieces of ‘the real.’
    This perspective has two advantages for the historian. First, while it may be the scientist’s job to discover the structure of nature or society it is certainly not the historian’s. The historian deals in texts, which give the historian access not to reality but to the actions of scientists-scientific practice. The historian’s methods are appropriate to the exploration of what scientists were doing at a given time, but will never lead him to a quark or a neutral current, or to social class. And, by paying attention to texts as indicators of contemporary scientific practice, the historian can escape from the retrospective idiom of the scientist.
    He can, in this way, attempt to understand the process of scientific development, and the judgments entailed in it, in contemporary rather than retrospective terms-but only, of course, if he distances himself from the realist identification of theoretical constructs with the contents of nature and society. This is where the mirror symmetry arises between the scientist’s account and that offered here. The scientist legitimates scientific judgments by reference to the state of reality; the historian attempts to understand them by reference to the cultural context in which they are made. I put scientific practice, which is accessible to the historian’s methods, at the center of my account, rather than the putative but inaccessible reality of theoretical constructs. My goal is to interpret the historical development of particle physics, social class, etc., including the pattern of scientific judgments entailed in it, in terms of the dynamics of research practice.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.