Home > Uncategorized > The leap of generalization

The leap of generalization

from Lars Syll

Statistician Andrew Gelman has an interesting blogpost up on what inference in science really means:

gelmanI like Don Rubin’s take on this, which is that if you want to go from association to causation, state very clearly what the assumptions are for this step to work. The clear statement of these assumptions can be helpful in moving forward …

Another way to say this is that all inference is about generalizing from sample to population, to predicting the outcomes of hypothetical interventions on new cases. You can’t escape the leap of generalization. Even a perfectly clean randomized experiment is typically of interest only to the extent that it generalizes to new people not included in the original study.

I agree — but that’s also why we so often fail (even when having the best intentions) when it comes to making generalizations in social sciences.

What strikes me again and again when taking part of the results of randomized experiments is that they really are very similar to theoretical models. They all have the same basic problem — they are built on rather artificial conditions and have difficulties with the ‘trade-off’ between internal and external validity. The more artificial conditions, the more internal validity, but also less external validity. The more we rig experiments/models to avoid the ‘confounding factors,’ the less the conditions are reminiscent of the real ‘target system.’ The nodal issue is basically about how scientists using different isolation strategies in different ‘nomological machines’ attempt to learn about causal relationships. I doubt the generalizability of the (randomized or not) experiment strategy because the probability is high that causal mechanisms are different in different contexts and that lack of homogeneity/stability/invariance don’t give us warranted export licenses to the ‘real’ societies.

Evidence-based theories and policies are highly valued nowadays. Randomization is supposed to best control for bias from unknown confounders. The received opinion — including Rubin and Gelman — is that evidence based on randomized experiments therefore is the best.

More and more economists have also lately come to advocate randomization as the principal method for ensuring being able to make valid causal inferences. Especially when it comes to questions of causality, randomization is nowadays considered some kind of “gold standard”. Everything has to be evidence-based, and the evidence has to come from randomized experiments.

But just as econometrics, randomization is basically a deductive method. Given  the assumptions (such as manipulability, transitivity, Reichenbach probability principles, separability, additivity, linearity, etc., etc.)  these methods deliver deductive inferences. The problem, of course, is that we will never completely know when the assumptions are right. [And although randomization may contribute to controlling for confounding, it does not guarantee it, since genuine ramdomness presupposes infinite experimentation and we know all real experimentation is finite. And even if randomization may help to establish average causal effects, it says nothing of individual effects unless homogeneity is added to the list of assumptions.] Real target systems are seldom epistemically isomorphic to our axiomatic-deductive models/systems, and even if they were, we still have to argue for the external validity of  the conclusions reached from within these epistemically convenient models/systems. Causal evidence generated by randomization procedures may be valid in ‘closed’ models, but what we usually are interested in, is causal evidence in the real target system we happen to live in.

Many advocates of randomization want  to have deductively automated answers to  fundamental causal questions. But to apply ‘thin’ methods we have to have ‘thick’ background knowledge of  what’s going on in the real world, and not in (ideally controlled) experiments. Conclusions  can only be as certain as their premises — and that also goes for methods based on randomized experiments.

So yours truly agrees with Gelman that “all inference is about generalizing from sample to population.” But I don’t think randomized experiments — ideal or not — take us very far on that road. Randomized experiments in social sciences are far from being the ‘gold standard’ they so often are depicted as.

  1. February 25, 2021 at 6:19 pm

    Relevant also is identifying whom you are actually making inferences about. This is every bit as important as the validity of the inferences themselves.

    Half of economic theory and 70-90 per cent of its ‘findings’ come out the the part of the theoretic called the Theory of the Consumer. Anyone reading any part of this theoretic would be well-off understanding that the word ‘Consumer’ means Purchaser. Replace the word ‘Consumer’ in ‘Consumer Sovereignty’ and Consumer Welfare and you get ‘Purchaser Sovereignty’ and ‘Purchaser Welfare’, which gravitates towards political dogmas about market rule since only purchasers in this or that market appear to have any ‘Welfare’. The theoretic ignores all non-purchasers in this or that market, and it shrouds from sight the reality that being unable to purchase goods or services in this or that market most certainly impacts upon the welfare of non-purchasers.

    It is, in short, very ‘handy’ to have a theoretic in which it’s impossible to even ask what happens to people’s well-being if they cannot afford to purchase in this or that market, and which blesses the notion that only purchasers in a market should have any say about it or, more broadly, the course of and economic activity. By using the word ‘consumers’, the reality that everyone is affected positively or negatively by what happens in markets is removed from sight because, after all, we are all ‘consumers’, aren’t we?

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.