Home > Uncategorized > Evidence-based economics

Evidence-based economics

from Lars Syll

Evidence-based theories and policies are highly valued nowadays. Randomization is supposed to control for bias from unknown confounders. The received opinion is that evidence based on randomized experiments therefore is the best.

More and more economists have also lately come to advocate randomization as the principal method for ensuring being able to make valid causal inferences.

I would however rather argue that randomization, just as econometrics, promises more than it can deliver, basically because it requires assumptions that in practice are not possible to maintain.

Especially when it comes to questions of causality, randomization is nowadays considered some kind of “gold standard”. Everything has to be evidence-based, and the evidence has to come from randomized experiments.

But just as econometrics, randomization is basically a deductive method. Given the assumptions (such as manipulability, transitivity, separability, additivity, linearity, etc.) these methods deliver deductive inferences. The problem, of course, is that we will never completely know when the assumptions are right. And although randomization may contribute to controlling for confounding, it does not guarantee it, since genuine ramdomness presupposes infinite experimentation and we know all real experimentation is finite. And even if randomization may help to establish average causal effects, it says nothing of individual effects unless homogeneity is added to the list of assumptions. Real target systems are seldom epistemically isomorphic to our axiomatic-deductive models/systems, and even if they were, we still have to argue for the external validity of the conclusions reached from within these epistemically convenient models/systems. Causal evidence generated by randomization procedures may be valid in “closed” models, but what we usually are interested in, is causal evidence in the real target system we happen to live in.

When does a conclusion established in population X hold for target population Y? Only under very restrictive conditions!

Ideally controlled experiments (still the benchmark even for natural and quasi experiments) tell us with certainty what causes what effects – but only given the right “closures”. Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. “It works there” is no evidence for “it will work here”. Causes deduced in an experimental setting still have to show that they come with an export-warrant to the target population/system. The causal background assumptions made have to be justified, and without licenses to export, the value of “rigorous” and “precise” methods is despairingly small.

Like us, you want evidence that a policy will work here, where you are. Randomized controlled trials (RCTs) do not tell you that. They do not even tell you that a policy works. What they tell you is that a policy worked there, where the trial was carried out, in that population. Our argument is that the changes in tense – from “worked” to “work” – are not just a matter of grammatical detail. To move from one to the other requires hard intellectual and practical effort. The fact that it worked there is indeed fact. But for that fact to be evidence that it will work here, it needs to be relevant to that conclusion. To make RCTs relevant you need a lot more information and of a very different kind. What kind? That’s what this book is about.

  1. September 15, 2015 at 3:11 pm

    I have always known that randomized trials do not depict the true situation. In fact most works in economics that are in touch with reality are in most cases rejected by mainstream economists. The reasons we don’t know. Great post admin thank. I believe its time EBE System Evidence Based Economics is promoted rather than the none senses we have in textbooks

  2. September 19, 2015 at 8:01 pm

    Randomization is an approach we use when we do not control the environment. Economics is about the transfer of value between humans. We have invented the idea and the mechanisms to do it. We control those mechanisms. Economics should be about adjusting the mechanisms to achieve the agreed objectives of the humans transferring value. We can run experiments on the economic system by changing the mechanisms and see the result.

    We can nowadays build a virtual economic system that is a model of the existing system and then run experiments on the virtual system. What we do at the moment with so called economic modelling is run experiments on a mathematical model of a theory of how the economy works. The economic models used are based on theories of economics not on the economic system as it exists. They are poor at predicting what will happen because the underlying mechanism of the way we repay credit when we transfer value has an exponential term of interest on interest. This must cause chaotic behaviour.

    Remove all interest on interest and all other compounding artifacts in our economic systems and prediction will work well.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s