Home > Uncategorized > Of what use are RCTs?

Of what use are RCTs?

from Lars Syll

nancyIn her interesting Pufendorf lectures Nancy Cartwright presents a theory of evidence and explains why randomized controlled trials (RCTs) are not at all the “gold standard” that it has lately often been portrayed as. As yours truly has repeatedly argued on this blog (e.g. here and here), RCTs usually do not provide evidence that their results are exportable to other target systems. The almost religious belief with which its advocates portray it, cannot hide the fact that RCTs cannot be taken for granted to give generalizable results. That something works somewhere is no warranty for it to work for us or even that it works generally.

  1. Gerald Holtham
    November 4, 2021 at 7:57 pm

    I recommend watching the lectures. They are sensible if not exactly ground-breaking. They consider the circumstances in which an RCT result can or cannot be applied in some other time and place other than that of the original trial. The lectures discourage “religious belief” but support thoughtful application, which is all most of RCT’s “advocates” would claim.
    There is no useful tool from a screwdriver to statistical theory that cannot be misused. So what?

  2. Ken Zimmerman
    November 6, 2021 at 6:07 am

    A few thoughts always come to me when RCT pops up.

    First, we need to consider the parable of the blind men and the Elephant. The parable originated in the ancient Indian subcontinent, from where it has been widely diffused. There are several versions of the parable. This is one. It is a story of a group of blind men who have never come across an elephant before and who learn and imagine what the elephant is like by touching it. Each blind man feels a different part of the elephant’s body, but only one part, such as the side or the tusk. They then describe the elephant based on their limited experience and their descriptions of the elephant are different from each other. In some versions, they come to suspect that the other person is dishonest and they come to blows. The moral of the parable is that humans have a tendency to claim absolute truth based on their limited, subjective experience as they ignore other people’s limited, subjective experiences which may be equally true. Reminds me of our curious invention RCT.

    Second, this from neuroscientist Stuart Firestein. “It is difficult to find a black cat in a dark room. Particularly, when there is no cat.” Firestein uses the dark to tell us about science.  Fundamentally, science is about stumbling around in the dark, bumping around trying to figure out what’s happening.  He describes science as “farting around, in the dark,” with no real organized method. Science is not about what we know. It about what we don’t know. About stumbling around in the dark. RCT tells us there is no dark.  Only the grand structures of knowledge that RCT opens to us. A false opening.

    Then we have the troublesome notion of randomness. As a concept that people can grasp, randomness must first be invented. There are no blueprints.  The concept is limited only by human imagination.  Which means humans must make some choices based on their judgments about randomness. Then humans must decide how the concept is practiced. Scientists name these decisions as ‘operationalization.’ Anthropologists name them as ‘performativity. So, there are some basic questions before us. What is a random choice, how are random choices actually performed, how many random selections are appropriate for effective and accurate results, what precisely are effective and accurate, at what point is drawing samples to end, how large should each sample be, etc. Stumbling  and bumbling ‘in the dark.’

  3. Gerald Holtham
    November 7, 2021 at 4:42 pm

    You might imagine from reading Ken Zimmerman that you could not trust your satnav, that aeroplanes regularly crashed, that high yielding plant varieties didn’t exist or always failed. Modern medicine has increased average life expectancy in most rich countries by nearly twenty years in the last century with methods based on RCT. It was RCT that told us the Covid vaccine generally works, at least for a time and has no or limited side effects. Has Ken had his jab? We know there is no such thing as perfect certainty but how about turning down the nihilism and using a bit of common sense?
    I have to agree when people say economics is not empirical enough and propositions are embraced without enough rigorous testing. Testing is harder than in physical sciences and perhaps than in biological sciences but you have to do the best you can. Critiquing specific tests makes sense but a generalised anathema on all known methods of empirical research leaves us in a black hole with no escape.

    • Ken Zimmerman
      November 19, 2021 at 8:47 am

      Gerald, thanks for feedback.

      Let’s begin with textbook descriptions.  “Drawing a simple random sample is accomplished by making a complete list of all the elements in a population, assigning each a number and then drawing a set of random numbers which identifies n members of the population to be sampled.” To make this workable the size of the population must be manageable. What is that? Not over 10,000, 100,000, 1,000,000? Even in the age of computer random number generators, large populations remain difficult to sample randomly.  Stratified and embedded sampling add more difficulty. Work involving small samples (e.g., medical) are still doable. If the population is clearly identifiable.  That’s more problematic today. For example, what is the population of all white American males today?

      Randomness is also an issue. One way of computing random numbers relies on atmospheric noise or simply uses the exact time you press keys on your keyboard as a source of unpredictable data, or entropy. For example, your computer might notice that you pressed a key at exactly 0.23423523 seconds after 2 p.m.. Grab enough of the specific times associated with these key presses and you’ll have a source of entropy you can use to generate what’s called a “true” random number.

      Pseudorandom numbers are an alternative to “true” random numbers. A computer could use a seed value and an algorithm to generate numbers that appear to be random, but that are in fact predictable. The computer doesn’t gather any random data from the environment. This isn’t necessarily a bad thing in every situation. For example, it works well enough for social scientific research. But no mathematician would take them as genuinely random.

      But none of this has anything to do with things like satnav. Satnav, digital computers, telephones, etc. are the result of years, sometimes decades of bumbling around in one dark room after another building technology, testing technology, rebuilding technology. All driven by the ideas, experiences, and judgments of engineers, material scientists, mathematicians, etc. The ‘scientific method’ here is whatever helps to find data that move us from one dark room to the next. Based on the judgments of those involved in the work.

      This is the opposite of nihilism.  It is recognizing that figuring out the world and the parts humans play in it is the essence of what humans do and have done since they first appeared on the planet. Everything humans know, care about, and use to create themselves is self-constructed and thus far also self-correcting by humans. Science is merely a formalization of this. Fundamentally, human existence is interacting with objects around them and working to name, conceptualize, and integrate the objects into their lives. This is reality and the subject matter of every historian, anthropologist, and social scientist (including economists). While not a black hole, humans do live in a series of dark rooms in which they can only bumble and stumble to figure out what’s there and how they want to relate to these things. Experience is sometimes a help but no certain solution. This is the human condition.

      Thrownness is an English translation of the German word ‘Geworfenehit’, a word with the meaning and connotation of a kind of alienation that human beings struggle against what is. It leaves a paradoxical opening for freedom.

      This concept, theorized by German philosopher Martin Heidegger (1889-1976), purports that as human beings we are “thrown” at birth into a world (class, nationality, animals, gender, trees, mountains, oceans, etc.) that we have no control over and must learn to cope with. It further purports that, after being “thrown” into a world that is not of our own choosing, we then have to learn to navigate the challenges that come with that world: frustrations, sufferings, demands, social conventions, and ties of kinship and duty. These challenges, and how we cope with them or rise above them, ultimately becomes our identity. And is the source of human freedom.

      Or, if you don’t like philosophy—”But I don’t have to know an answer. I don’t feel frightened by not knowing things, by being lost in a mysterious universe without any purpose, which is the way it really is, so far as I can tell. It doesn’t frighten me.”  Richard P. Feynman.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: