Home > Uncategorized > On the limits of ‘mediation analysis’ and ‘statistical causality’

On the limits of ‘mediation analysis’ and ‘statistical causality’

from Lars Syll

mediator“Mediation analysis” is this thing where you have a treatment and an outcome and you’re trying to model how the treatment works: how much does it directly affect the outcome, and how much is the effect “mediated” through intermediate variables …

In the real world, it’s my impression that almost all the mediation analyses that people actually fit in the social and medical sciences are misguided: lots of examples where the assumptions aren’t clear and where, in any case, coefficient estimates are hopelessly noisy and where confused people will over-interpret statistical significance …

More and more I’ve been coming to the conclusion that the standard causal inference paradigm is broken … So how to do it? I don’t think traditional path analysis or other multivariate methods of the throw-all-the-data-in-the-blender-and-let-God-sort-em-out variety will do the job. Instead we need some structure and some prior information.

Andrew Gelman

Causality in social sciences — and economics — can never solely be a question of statistical inference. Causality entails more than predictability, and to really in depth explain social phenomena require theory. Analysis of variation — the foundation of all econometrics — can never in itself reveal how these variations are brought about. First, when we are able to tie actions, processes or structures to the statistical relations detected, can we say that we are getting at relevant explanations of causation.

Most facts have many different, possible, alternative explanations, but we want to find the best of all contrastive (since all real explanation takes place relative to a set of alternatives) explanations. So which is the best explanation? Many scientists, influenced by statistical reasoning, think that the likeliest explanation is the best explanation. But the likelihood of x is not in itself a strong argument for thinking it explains y. I would rather argue that what makes one explanation better than another are things like aiming for and finding powerful, deep, causal, features and mechanisms that we have warranted and justified reasons to believe in. Statistical — especially the variety based on a Bayesian epistemology — reasoning generally has no room for these kinds of explanatory considerations. The only thing that matters is the probabilistic relation between evidence and hypothesis. That is also one of the main reasons I find abduction — inference to the best explanation — a better description and account of what constitute actual scientific reasoning and inferences.

In the social sciences … regression is used to discover relationships or to disentangle cause and effect. However, investigators have only vague ideas as to the relevant variables and their causal order; functional forms are chosen on the basis of convenience or familiarity; serious problems of measurement are often encountered.

Regression may offer useful ways of summarizing the data and making predictions. Investigators may be able to use summaries and predictions to draw substantive conclusions. However, I see no cases in which regression equations, let alone the more complex methods, have succeeded as engines for discovering causal relationships.

David Freedman

Some statisticians and data scientists think that algorithmic formalisms somehow give them access to causality. That is, however, simply not true. Assuming ‘convenient’ things like faithfulness or stability is not to give proofs. It’s to assume what has to be proven. Deductive-axiomatic methods used in statistics do no produce evidence for causal inferences. The real causality we are searching for is the one existing in the real world around us. If there is no warranted connection between axiomatically derived theorems and the real world, well, then we haven’t really obtained the causation we are looking for.

If contributions made by statisticians to the understanding of causation are to be taken over with advantage in any specific field of inquiry, then what is crucial is that the right relationship should exist between statistical and subject-matter concerns …
introduction-to-statistical-inferenceThe idea of causation as consequential manipulation is apt to research that can be undertaken primarily through experimental methods and, especially to ‘practical science’ where the central concern is indeed with ‘the consequences of performing particular acts’. The development of this idea in the context of medical and agricultural research is as understandable as the development of that of causation as robust dependence within applied econometrics. However, the extension of the manipulative approach into sociology would not appear promising, other than in rather special circumstances … The more fundamental difficulty is that​ under the — highly anthropocentric — principle of ‘no causation without manipulation’, the recognition that can be given to the action of individuals as having causal force is in fact peculiarly limited.

John H. Goldthorpe

  1. Ken Zimmerman
    June 13, 2021 at 8:44 am

    The identification and evaluation of causes, explaining why one set of events or state of affairs came into being and not another, is a – perhaps the – fundamental task of historians, linked to the formulation and discussion of questions, the selection of subjects and evidence, the definition of concepts and the establishment and contestation of theories. History in this context is the explanation of significant processes of change and instances of stasis.

     
    Such a definition appears to correspond to social science rather than to history, according to one common distinction between the two, in which the social scientist ‘constructs type-concepts and seeks to formulate  general  statements about what happens’ whereas the historian ‘aims to provide a causal analysis and an assessment of  individual culturally significant actions, social systems and persons’ in the words of the sociologist Max Weber.  Historians, so the argument runs, tend to particularize, retrieving and reconstructing individuals’ experiences and points of view and establishing the relations –   sometimes causal, sometimes not – between their actions over time (diachronic). Social scientists generalize, often through an   examination of relations between aggregates of individuals at a single point in time (synchronic). Yet Weber rightly recognized that this distinction is one of degree, with social scientists relying on a knowledge of unique sequences of historical events and historians necessarily referring to all kinds of categories and patterns of repeated actions. Social science is therefore ‘necessarily historical,’ just as history is social-scientific. From this standpoint, the main debate concerns the extent to which historians and other social scientists emphasize the singularity of events or they aim to generalize.
     

    William Sewell in ‘Logics of History’ asks whether there are ‘consequential forms of social mediation’ that cannot be grasped adequately by means of semiotic methods, ‘even if language is a major, or the major, way that interdependence in human relations is mediated.’ His answer begins with the supposition that most social scientists (or ‘at least most social scientists outside of history and anthropology’), given the alleged dominance of quantitative methods and positivist epistemology in American social science, would hold that such methods ‘are far from sufficient for making sense of the social world,’ availing themselves of ‘a very different form of explanation, which I would call mechanistic’ and which ‘specifies not paradigm and performance but cause and effect.’  Such mechanistic explanation can be applied to physical nature, where it implies that ‘the presence of some phenomenon (a cause) determines the appearance of another phenomenon (an effect),’ but it has been extended through analogy to human relations, where ‘laws governing social phenomena,’ in contrast to those concerning natural ones, always take a varying form, thanks to the extraordinary complexity of the determinants of human actions.  Sewell’s   counter-proposal, which implicitly derives from the assumptions and practices of the majority of historians ‘within a   non-theoretical discipline’, presents an ‘interpretivist’ methodology to account for the uniquely semiotic interactions of human beings.
     

    In ‘The Landscape of History’ (2002), John Gaddis advances the intriguing thesis that the methods and theoretical premises of natural sciences and history have begun to converge, espousing simulation, which ‘attempts to illustrate (not replicate) some specific set of past events’, not models, which show how a system has worked in the past, but also how it will work in the future.  Gaddis pits both scientists and historians, who operate with complex and chaotic systems with many variables, against social scientists, who seek to understand reality by breaking it up into its various parts.  The latter seem to have a ‘reductionist’ and the former an ‘ecological’ view of reality. Historians, anthropologists, paleontologists, evolutionary biologists and astronomers generalize, but only from the knowledge of particular outcomes: that’s what Gaddis means by particular generalization. In pursuit of parsimony, stability and universality, social scientists allegedly look for the variable within an equation that determines the value of all the others or, more broadly (and very differently, Gaddis argues), they seek the element whose removal from a causal chain would alter the outcome.  At most, they practice a form of ‘general particularization’, examining particular sets of events to confirm or refute an hypothesis: ‘Theory therefore comes first, with explanation [of facts] enlisted as needed to confirm it.’ Such a   distinction seems redundant: the necessary, reciprocal relationship between theories and facts – or between theories, contexts and facts – matters more than whether theory or fact ‘comes first’. Arguably, historians’ and other social scientists’ awareness of the relationship itself should constitute the philosophical and methodological starting-point of their enquiries

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.