Home > Uncategorized > Taking the con out of RCTs

Taking the con out of RCTs

from Lars Syll

rcDevelopment actions and interventions (policies/programs/projects/practices) should be based on the evidence. This truism now comes with a radical proposal about the meaning of “the evidence.” In development practice, where there are hundreds of complex, sometimes rapidly changing, contexts seemingly innocuous phrases like “rely on the rigorous evidence” are taken to mean: “Ignore evidence from your context and rely in your context on evidence that was ‘rigorous’ for another place, another time, another implementing organization, another set of interacting policies, another set of local social norms, another program design and do this without any underlying model or theory that guides your understanding of the relevant phenomena.” …

The advocates of RCTs and the use and importance of rigorous evidence, who are mostly full-time academics based in universities, have often taken a condescending, if not outright ad hominem, stance towards development practitioners. They have often treated arguments against exclusive reliance on RCT evidence, like that the world is complex, getting things done in the real world is a difficult craft, that RCTs don’t address key issues, that results cannot be transplanted across contexts, not as legitimate arguments but as the self-interested pleadings of “bureaucrats” who don’t care about “the evidence” or development outcomes. Therefore, it is striking that it is the practitioner objections about external validity that are actually technically right about the unreliability of RCTs for making context-specific predictions and it is the academics that are wrong, and this in the technical domain that supposedly is the acamedicians comparative advantage.

Lant Pritchett

Just as econometrics, the use of randomization often promises more than it can deliver, basically because it requires assumptions that in practice are not possible to maintain. And just like econometrics, randomization is basically a deductive method. Given the assumptions, these methods deliver deductive inferences. The problem, of course, is that we will never completely know when the assumptions are right. And although randomization may contribute to controlling for confounding, it does not guarantee it, since genuine ramdomness presupposes infinite experimentation and we know all real experimentation is finite. And even if randomization may help to establish average causal effects, it says nothing of individual effects unless homogeneity is added to the list of assumptions. Causal evidence generated by randomization procedures may be valid in ‘closed’ models, but what we usually are interested in, is causal evidence in the real target system we happen to live in.

As Pritchett shows, does a conclusion established in population X hold for target population Y only under very restrictive conditions. ‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. “It works there” is no evidence for “it will work here”. Causes deduced in an experimental setting still have to show that they come with an export-warrant to the target population/system. The causal background assumptions made have to be justified, and without licenses to export, the value of ‘rigorous’ and ‘precise’ methods — and ‘on-average-knowledge’ — is despairingly small.

  1. June 30, 2021 at 11:13 pm

    This is a perfect example of functioning exactly backwards. Representative democracy is another exactly backwards functioning system; Government an contracted experts design systems to satisfy their donor market. Completely understandable in a backwards functioning system.

    A functional government exhibits systematic organizational support for what the people decide to do. A functional government organizes accomplishment of policies set out democratically by the distributed intelligence of the nation.

  2. June 30, 2021 at 11:34 pm

    “Ignore evidence” from sometime before any bank crisis “and rely in on evidence ‘rigorously’” collected about what was risky after the crisis”.
    Also known as Monday morning quarterbacking

  3. deshoebox
    July 1, 2021 at 1:31 am

    I’m not sure what verb tense they will use in talking about this kind of thing but there is also the habit of ignoring what will at some point in the foreseeable future become incontrovertible evidence in hindsight. Future perfect retrospective subjunctive, perhaps? In any case, as an example consider how in twenty years looking back most of what observe now, whether quantifiable or not, will obviously support the thesis that people of today should have made and implemented policy decisions that will (or would have) led to reducing climate warming before Florida sank under the waves rather than after. If there are any universities left there is no doubt that smart economists will develop fiendishly clever models that might coulda predicted that outcome with absolute certainty.

  4. Ken Zimmerman
    July 3, 2021 at 10:29 am

    Two important notions in research are generalization and transfer.

    Generalizability is applied by researchers in experimental and academic settings. It can be defined as the extension of research findings and conclusions from a study conducted on a sample population to the population at large. Or, as research from one context (setting) applied or used to assess another setting. While the dependability of this extension is never absolute, it is probable based on the experiences of the researchers. Because sound generalizability requires data on large populations and diverse settings, quantitative research — experimental for instance, deep qualitative research – extensive interviews for instance, and well seasoned researchers provide the best foundation for producing broad generalizability. The larger the base of data, the more one can generalize the results. For example, a comprehensive study of the role computers play in the writing process might reveal that it is noticeably probable that students who do most of their composing on a computer will move chunks of text around more than students who do not compose on a computer.

    Transferability is applied by the readers of research. Although generalizability usually applies only to certain types of methods, transferability can apply in varying degrees to most types of research . Unlike generalizability, transferability does not involve broad claims, but invites readers of research to make connections between elements of a study and their own experience. For instance, teachers at the high school level might selectively apply to their own classrooms results from a study demonstrating that heuristic writing exercises help students at the college level.

    In many ways, generalizability amounts to nothing more than making predictions based on a recurring experience. If something occurs frequently, we expect that it will continue to do so in the future. Researchers use the same type of reasoning when generalizing about the findings of their studies. Once researchers have collected sufficient data to support a hypothesis, a premise regarding the behavior of that data can be formulated, making it generalizable to similar circumstances. Because of its foundation in historical experience, however, such a generalization cannot be regarded as conclusive or exhaustive.

    Like all the rest of science, generalizability is just a special case (both because of who and why it’s performed and what the expected results are) of what humans have been doing for 30 or 40,000 years. Or longer. And each of us still does it every day. In the US each of us drives on the right side of the road and rxpect others to do the same. It’s a cultural and behavioral norm. In the UK the norm is left but we expect all UK citizens to adhere to that norm. This doesn’t change for scientists. For example, scientists expect atomic weights to be fixed baring specific events known (assumed based on experience) to change atomic weights.

    One important practical note.. When writing up the results of a study, it is important that the researcher provide specific information about and a detailed description of their subject(s), location, methods, role in the study, etc. This is commonly referred to as “thick description” of methods and findings; it is important because it allows readers to make an informed judgment about whether they can transfer the findings to their own situation. For example, if an educator conducts an ethnography of their writing classroom, and finds that their students’ writing improved dramatically after a series of student-teacher writing conferences, the educator must describe in detail the classroom setting, the students observed, and the educator’s own participation. If the researcher does not provide enough detail, it will be difficult for readers to try the same strategy in their own classrooms. For example, if the researcher fails to mention that they conducted this research in a small, upper-class private school, readers may attempt to transfer the results to a large, inner-city public school expecting a similar outcome.

  5. July 14, 2021 at 4:04 am

    In much of the medical world ‘evidence based’ means according to the fixed beliefs of biopsychosocial psychologists, to whom, actually looking for real evidence is a sign of psychosisand must be discouaged. Thus, the term has come to mean the opposite of what it appears to say.

    Similarly, in the ‘planning and development’ world, ‘sustainability’, which was intended to be a test conditon that development must be restricted to what environmental resources can safely bear, was repaced with ‘sustainable development’: the term that overrules all others because development *must continue increasing exponentially* to ensure perpetual jobs for planners and developers.

    As for the problems of ‘western’ academics control of ‘developing world’ ‘aid’: This was very thoroughly described years ago, in a masterly work called: ‘Who’s Reality Counts: Putting the First Last’. By Robert Chambers.


  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.