Home > Uncategorized > On the non-applicability of statistical models

On the non-applicability of statistical models

from Lars Syll

Eminent statistician David Salsburg is rightfully very critical of the way social scientists — including economists and econometricians — uncritically and without arguments have come to simply assume that they can apply probability distributions from statistical theory on their own area of research:

9780805071344We assume there is an abstract space of elementary things called ‘events’ … If a measure on the abstract space of events fulfils​ certain axioms, then it is a probability. To use probability in real life, we have to identify this space of events and do so with sufficient specificity to allow us to actually calculate probability measurements on that space … Unless we can identify [this] abstract space, the probability statements that emerge from statistical analyses will have many different and sometimes contrary meanings …

Kolmogorov established the mathematical meaning of probability: Probability is a measure of sets in an abstract space of events. All the mathematical properties of probability can be derived from this definition. When we wish to apply probability to real life, we need to identify that abstract space of events for the particular problem at hand … It is not well established when statistical methods are used for observational studies … If we cannot identify the space of events that generate the probabilities being calculated, then one model is no more valid than another … As statistical models are used more and more for observational studies to assist in social decisions by government and advocacy groups, this fundamental failure to be able to derive probabilities without ambiguity will cast doubt on the usefulness of these methods.

Wise words well worth pondering on.

As long as economists and statisticians cannot really identify their statistical models with real-world phenomena there is no real warrant for taking their statistical inferences seriously.  

Just as there is no such thing as a ‘free lunch,’ there is no such thing as a ‘free probability.’ To be able at all to talk about probabilities, you have to specify a model. If there is no chance set-up or model that generates the probabilistic outcomes or events – in statistics one refers to any process where you observe or measure as an experiment (rolling a die) and the results obtained as the outcomes or events (number of points rolled with the die, being e. g. 3 or 5) of the experiment – there strictly seen is no event at all.

Probability is a relational element. It always must come with a specification of the model from which it is calculated. And then to be of any empirical scientific value it has to be shown to coincide with (or at least converge to) real data generating processes or structures – something seldom or never done!

And this is the basic problem with economic data. If you have a fair roulette-wheel, you can arguably specify probabilities and probability density distributions. But how do you conceive of the analogous ‘nomological machines’ for prices, gross domestic product, income distribution etc? Only by a leap of faith. And that does not suffice. You have to come up with some really good arguments if you want to persuade people into believing in the existence of socio-economic structures that generate data with characteristics conceivable as stochastic events portrayed by probabilistic density distributions!

  1. lobdillj
    December 14, 2017 at 7:59 pm

    Well said!

  2. December 15, 2017 at 11:12 am

    Does this help us understand better why humans invented and use mathematics? Axioms are only axioms because humans made them that way. Does probability serve some needs for humans? It’s been shown that humans only invent what they need.

  3. Rob Reno
    December 16, 2017 at 5:32 am

    “As long as economists and statisticians cannot really identify their statistical models with real-world phenomena there is no real warrant for taking their statistical inferences seriously.” ~ Lars Syll

    “Economics, however is different in its politics from the other social sciences in one important sense. It is tied to the notion that economics is the queen of the social sciences, coming near to a physical science. Economics can literally provide comprehensive explanations of all facets of human existence. Thereby transcending and replacing philosophy and all other sciences.” ~ Ken Zimmerman

    Many of Syll’s arguments and posts reminds me of Whitehead’s ‘Fallacy of Misplaced Concreteness.’ I am currently working my way through two recommended readings Gustavo Marqués (2016, A Philosophical Framework for Rethinking Theoretical Economics and Philosophy of Economics) and Edward Fullbrook (2016, Narrative Fixation in Economics). Having already read Robert H. Nelson’s “Economics as Religion” and Tomas Sedlacek’s “Economics of Good and Evil” it has become obvivious, in my viewpoint, that erroneous philosophy is one of the root causes of the failure of modern economics. The history of economic ideas reveals the fundamental philosophical frameworks and assumptions along with how these have changed over time (e.g., the elimination of morality, ethics, uncertainty, etc., in the secular/materialist quest to make a ‘social mathematics’ in the mathematization of the field). Whether conscious of their philosophical assumptions or not, all scientists _must_ do philosophy in the simple act of doing science (Weinert 2004). The only question in my mind is are they aware of those unexamined philosophical assumptions that are underpinning their endeavors. One of the fist books on economics I read was Robert H. Nelson’s “Economics as Religion,” followed by Tomas Sedlacek’s “Economics of Good and Evil.” These two works expose the Naked Emperor of economic high theory. And of course the classic is Keen’s “Debunking Economics.” Both Nelson and Sedlacek show how religion and philosophical assumptions were historically at the core of modern economic theory, and how these unspoken philosophical assumptions have shifted over time from religious beliefs seeking mathematical/scientific justification to just dropping the religious elements and elevating the pure empiricist/materialist viewpoint exclusively.

    One idea I have come across in my economic readings is the desire by economists to model economic theory and methodology after the philosophical-mechantistic mathematical physics of the nineteenty century ignoring the progress in the field of physics since the birth of quantum mechanics and its philosophical undermining much of mechanistic-reductionistic philosophy underpinning nineteenth century physics. Donald N. McCloskey’s (1991) “The Arrogance of Economic Theorists” (http://www.deirdremccloskey.com/docs/graham/arrogance.pdf) highlights this well: “Economists think of themselves as physicists of the social sciences. But they know nothing about how physics operates as a field, and the physicists themselves are astonished at the mathematical character of economics.” Mathematical rigor and abstract mathematical proofs divorced from reality and that don’t fit the real world and are untested by real-world experimental data are uninteresting to those pursuing real science who want their models and theories to tell them something meaningful about the real world. This theme is repeated over and over, in disparate scientific fields, at different times by different leading thinkers who have reached the top of their field of endeavor.

    This is a common theme in the history of science; naïve philosophical materialism (Nelson calls this “scientific materialism”) wedded to philosophical reductionism practiced with varying degrees of awareness of the philosophical content (i.e., critical assumptions about reality grounded in philosophical choice/assumptions). Such mathematical hubris (e.g., the belief/assumption that the only things that matter are materially quantifiable) denies both the fully human social context, mind-meanings, and spirit-values that makes us truly human. These are the fruits of thoughtless secular materialism and the naïve belief that only “science” matters and offers us truth while philosophy and religion are a waste of time and mental effort. This is “scientism”; pseudo-scientific myth-making. Philosophical reductionism (aka ‘physical reductionism’ of the Newtonian clock-work universe) holds that all scientific explanations may (and should) eventually be recast in terms of physics:

    “To start with, I should say I’m an unashamed reductionist. I believe that the laws of biology can be reduced to those of chemistry. We have already seen this happening with the discovery of the structure of DNA. And I further believe that the laws of chemistry can be reduced to those of physics. I think most chemists would agree with that. ” (Hawking, Stephan. The Objections of an Unashamed Reductionist. In The Large, the Small and the Human Mind. Canto Edition ed. Cambridge: Cambridge University Press; 2005; p. 169.)

    Hawking’s claim that biology can be reduced to chemistry based upon the discovery of the structure of DNA ignores the scientific revolution in biology brought about by the rediscovery of epigenetics and biological complexity revealed therein that refutes genetic reductionism. Syll (2016) writes, “Reducing microeconomics to refinements of hyper-rational Bayesian deductivist models is not a viable way forward. It will only sentence to irrelevance the most interesting real world economic problems. And as someone has so wisely remarked, murder is unfortunately the only way to reduce biology to chemistry — reducing macroeconomics to Walrasian general equilibrium microeconomics basically means committing the same crime.” (Syll 2016, 56)

    Reference List

    1. Whitehead, Alfred North. Science and the Modern World.: The Free Press; 1925; c1967 pp. 50-55.
    Notes: The answer, therefore, which the seventeenth century gave to the ancient question … “What is the world made of?” was that the world is a succession of instantaneous configurations of matter — or material, if you wish to include stuff more subtle than ordinary matter…. Thus the configurations determined there own changes, so that the circle of scientific thought was completely closed. This is the famous mechanistic theory of nature, which has reigned supreme ever since the seventeenth century. It is the orthodox creed of physical science…. There is an error; but it is merely the accidental error of mistaking the abstract for the concrete. It is an example of what I will call the ‘Fallacy of Misplaced Concreteness.’ This fallacy is the occasion of great confusion in philosophy. (Whitehead 1967: 50-51)

    (….) This conception of the universe is surely framed in terms of high abstractions, and the paradox only arises because we have mistaken our abstractions for concrete realities…. The seventeenth century had finally produced a scheme of scientific thought framed by mathematics, for the use of mathematics. The great characteristic of the mathematical mind is its capacity for dealing with abstractions; and for eliciting from them clear-cut demonstrative trains of reasoning, entirely satisfactory so long as it is those abstractions which you want to think about. The enormous success of the scientific abstractions, yielding on the one hand matter with its simple location in space and time, on the other hand mind, perceiving, suffering, reasoning, but not interfering, has foisted onto philosophy the task of accepting them as the most concrete rendering of fact. (Whitehead 1967: 54-55)

    Thereby, modern philosophy has been ruined. It has oscillated in a complex manner between three extremes. These are the dualists, who accept matter and mind as on an equal basis, and the two varieties of monists, those who put mind inside matter, and those who put matter inside mind. But this juggling with abstractions can never overcome the inherent confusion introduced by the ascription of misplaced concreteness to the scientific scheme of the seventeenth century. (Whitehead 1967: 55)

    2. Weinert, Friedel. The Scientist as Philosopher: Philosophical Consequences of Great Scientific Discoveries. Berlin: Springer-Verlag; 2004; p. 95; 100.
    Notes: [W]hen the facts speak against the adequacy of the concepts, something needs to give way. (Weinert 2004: 95) The notion of presupposition is well known to philosophers and historians of science…. It has often been observed that presuppositions play a pivotal role in human thinking…. It is important to realize that presuppositions change. But it is even more important to inquire how and why they change. (Weinert 2004: 100)

    3. Weinert, Friedel. The Scientist as Philosopher: Philosophical Consequences of Great Scientific Discoveries. Berlin: Springer-Verlag; 2004; pp. 278-279.
    Notes: The reason for the everlasting interaction between science and philosophy transpires clearly. The human mind musters an admirable ability to think up equations for physical systems. But equations need to be interpreted in terms of physical models and mechanisms. Science requires conceptual understanding. This understanding employs fundamental philosophical notions.

    (….) The scientific enterprise comes with philosophical commitments, whether the scientist likes it or not. The scientist needs philosophical ideas, simply because amongst the experimental and mathematical tools in the toolbox of the scientist there are conceptual tools, like fundamental notions. The despairing scientist may ask: ‘Will we ever get an answer?’ The philosopher replies: ‘Not a definitive answer, but a few tentative answers.’ Recall that the philosopher (and the scientist qua philosopher) works with conceptual models. At any one time only a few of these models are in circulation. They cannot provide the definitive answers of which the scientist is fond. But this is typical of models even in the natural sciences.

    4. Barbour, Ian G. Nature, Human Nature, and God. Minneapolis: Fortress Press; 2002; pp. 4-5.
    Notes: While I accept the evidence for evolution, as almost all scientists do, I do not accept the philosophy of materialism that is assumed or defended by many scientists. Materialism is the assertion that matter is the fundamental reality in the universe. Materialism is a form of metaphysics (a set of claims concerning the most general characteristics and constituents of reality). It is often accompanied by a second assertion: the scientific method is the only reliable path to knowledge. This is a form of epistemology (a set of claims concerning inquiry and the acquisition of knowledge). The two assertions are linked: if the only real entities are those with which science deals, then science is the only valid path to knowledge. (Barbour 2002: 4-5)

    In addition, many forms of materialism express reductionism. Epistemological reductionism claims that the laws and theories of all the sciences are in principle reducible to the laws of physics and chemistry. Metaphysical reductionism claims that the component parts of any system determine its behavior. The materialist believes that all phenomena will eventually be explained in terms of the actions of material components, which are the only effective causes in the world. In the past, powerful new theories excited the imagination of scientists who sometimes extrapolated them beyond their proper domains. In the eighteenth century many scientists thought that Newtonian physics could in principle account for all phenomena, but in the twentieth century quantum physics has shown the limits of such predictability. Today molecular biology is an immensely fruitful research program, and we may be tempted to think that it will explain the behavior of all living things. But new ideas in the biological sciences encourage a less reductionist view. (Barbour 2002: 5)

    Scientists have often extended scientific concepts beyond their scientific use to support comprehensive materialistic philosophies. The identification of the real with measurable properties that can be correlated by exact mathematical relationships started in the physical sciences, but it influenced scientists in other fields and continues today. I would argue that the quantifiable properties of matter have been abstracted from the real world by ignoring the particularity of events and the nonquantifiable aspects of human experience. We do not have to conclude that matter alone is real or that mind, purpose, and human love are only byproducts of matter in motion. (Barbour 2002: 5)

    • Rob Reno
      December 17, 2017 at 5:24 am

      Polished paper in scientific journals are great, but they are not a real view into the sausage grinder and how science is done on the personal and group level. Symposiums, conferences, etc., that turn into books often reveal what polished papers do not. Carle Woese was to biology what Hawking is to cosmology. The fields of evolutionary developmental biology (evo-devo and devo-evo) are young relatively speaking, but have revolutionized our understanding of evolutionary theory and overturned many a dogma. Hawking is ignorant, which is not insult but to point that even great minds can have huge blind spots. Woese writes,

      Conceptualizing Cells

      We should all take seriously an assessment of biology made by the physicist David Bohm over 30 years ago (and universally ignored):

      “It does seem odd … that just when physics is … moving away from mechanism, biology and psychology are moving closer to it. If the trend continues … scientists will be regarding living and intelligent beings as mechanical, while they suppose that inanimate matter is to complex and subtle to fit into the limited categories of mechanism.” [D. Bohm, “Some Remarks on the Notion of Order,” in C. H. Waddington, ed., Towards a Theoretical Biology: 2 Sketches. (Edinburgh: Edinburgh Press 1969), p. 18-40.]

      The organism is not a machine! Machines are not made of parts that continually turn over and renew; the cell is. A machine is stable because its parts are strongly built and function reliably. The cell is stable for an entirely different reason: It is homeostatic. Perturbed, the cell automatically seeks to reconstitute its inherent pattern. Homeostasis and homeorhesis are basic to all living things, but not machines.

      If not a machine, then what is the cell?

      ~ Woese, Carl R., Author. Evolving Biological Organization. In Microbial Phylogeny and Evolution: Concepts and Controversies. (Jan Sapp, ed.). Oxford: Oxford University Press; 2005: 100.

      I think Woese knows a bit more about the issue than Hawking, who in my view could use some humility.

  4. Luis
    December 16, 2017 at 8:49 pm

    “As long as…” is the best ever parragraph in non-statistical economic analyse!

  5. December 17, 2017 at 8:52 am

    Rob, thank you particularly for your comment of 17th Dec on symposiums and conferences rather than polished papers being where you can see real science being done. See the WEA conference on

  6. December 17, 2017 at 9:07 am

    Sorry, wrong key! Please see the WEA Conference on Complexity in Economic Philosophy, where I’ve struggled to share discoveries and conclusions from a lifetime as a scientidst which you, Rob, here express so eloquently from the viewpoint of biology.

    Thank you again, for the excellent word ‘homeorhesis’, which is spot on but new to me, even ‘homeostasis’ being the supplement to my 1964 dictionary.

    • Rob Reno
      December 17, 2017 at 7:21 pm

      “Sorry, wrong key!” Gosh do I know that one! The older I get the more frequent it happens, especially on those darn soft keyboards not made for old fat fingers ;-)

      My pleasure and thank you for the tip to the conference paper. I cannot claim originality in the concept but my old mind could not remember who or where I had read it. Here is a curious thought I had that you might enjoy. As I study Syll’s (2016, 107) — and assuming I understand it correctly ;-) on the non-ergodicity of utility qua preferences, he writes:

      Daniel Kahneman [2011:113] writes — in Thinking, Fast and Slow — that expected utility theory is seriously flawed since it does not take into consideration the basic fact that people’s choices are influenced by changes in their wealth [and actually, much, much, more]. Where standard microeconomic theory assumes that preferences are stable over time, Kahneman and other behavioural economists have forcefully again and again shown that preferences aren’t fixed, but vary with different reference points. How can a theory that doesn’t allow for people having different reference points from which they consider their options have an almost axiomatic status within economic theory?

      The mystery is how a conception of the utility of outcomes that is vulnerable to such obvious counterexamples survived for so long. I can explain it only by a weakness of the scholarly mind… I call it theory-induced blindness: once you have accepted a theory and used it as a tool in your thinking it is extraordinarily difficult to notice its flaws… You give the theory the benefit of the doubt, trusting the community of experts who have accepted it… But they did not pursue the idea to the point of saying, “This theory is seriously wrong because it ignores the fact that utility depends on the history of one’s wealth, not only present wealth.” [Kahneman [2011:113]]

      On a more economic-theoretical level, information theory — and especially the so called the Kelly criterion — also highlights the problems concerning the neoclassical theory of expected utility.

      ~ Syll, Lars Pålsson. On the Use and Misuse of Theories and Models in Mainstream Economics. UK: College Publications for World Economics Association; 2016; p. 107. (On the use and misuse of theories and models in mainstream economics; v. 4).

      It is hard to overstate the depth and breadth of the ongoing revolution in the biological sciences today. The transformation Woese describes below is profound, yet, he witnesses to the same conceptual straitjacket of _theory-induced blindness_ and its difficulty in thinking outside the box, a kind of unconscious intellectual inertia almost:

      The science of biology enters the twenty-first century in turmoil, in a state of conceptual disarray, although at first glance this is far from apparent. When has biology ever been in a more powerful position to study living systems? The sequencing juggernaut has still to reach full steam, and it is constantly spewing forth all manner of powerful new approaches to biological systems, many of which were previously unimaginable: a revolutionized medicine that reaches beyond diagnosis and cure of disease into defining states of the organism in general; revolutionary agricultural technology built on genomic understanding and manipulation of animals and plants; the age-old foundation of biology, taxonomy, made rock solid, greatly extended, and become far more useful in its new genomic setting; a microbial ecology that is finally able to contribute to our understanding of the biosphere; and the list goes on. (Woese 2005: 99)

      All this is an expression of the power inherent in the methodology of molecular biology, especially the sequencing of genomes. Methodology is one thing, however, and understanding and direction another. The fact is that the understanding of biology emerging from the mass of data that flows from the genome sequencing machines brings into question the classical concepts of organism, lineage, and evolution as the same time it gainsays the molecular perspective that spawned the enterprise. The fact is that the molecular perspective, which so successfully guided and shaped twentieth-century biology, has effectively run its course (as all paradigms do) and no longer provides a focus, a vision of the biology of the future, with the result that biology is wandering will-nilly into that future. This is a prescription for revolution–conceptual revolution. One can be confident that the new paradigm will soon emerge to guide biology in this new century…. Molecular biology has ceased to be a genuine paradigm, and it is now only a body of (very powerful) technique…. The time has come to shift biology’s focus from trying to understand organisms solely by dissecting them into their parts to trying to understand the fundamental nature of biological organization, of biological form. (Woese 2005: 99-100)

      (….) When one has worked one’s entire career within the framework of a powerful paradigm, it is almost impossible to look at that paradigm as anything but the proper, if not the only possible, perspective one can have on (in this case) biology. Yet despite its great accomplishments, molecular biology is far from the “perfect paradigm” most biologists take it to be. This child of reductionist materialism has nearly driven the biology out of biology. Molecular biology’s reductionism is fundamentalist, unwavering, and procrustean. It strips the organism from its environment, shears it of its history (evolution), and shreds it into parts. A sense of the whole, of the whole cell, of the whole multicellular organism, of the biosphere, of the emergent quality of biological organization, all have been lost or sidelined. (Woese 2005: 101)

      Our thinking is fettered by classical evolutionary notions as well. The deepest and most subtle of these is the concept of variation and selection. How we view the evolution of cellular design or organization is heavily colored by how we view variation and selection. From Darwin’s day onward, evolutionists have debated the nature of the concept, and particularly whether evolutionary change is gradual, salutatory, or of some other nature. However, another aspect of the concept concerns us here more. In the terms I prefer, it is the nature of the phase (or propensity) space in which evolution operates. Looked at one way, variation and selection are all there is to evolution: The evolutionary phase space is wide open, and all manner of things are possible. From this “anything goes” perspective, a given biological form (pattern) has no meaning outside of itself, and the route by which it arises is one out of an enormous number of possible paths, which makes the evolution completely idiosyncratic and, thus, uninteresting (molecular biology holds this position: the molecular biologist sees evolution as merely a series of meaningless historical accidents). (p. 101)

      The alternative viewpoint is that the evolutionary propensity space is highly constrained, being more like a mountainous terrain than a wide open prairie: Only certain paths are possible, and they lead to particular (a relatively small set of) outcomes. Generic biological form preexists in the same sense that form in the inanimate world does. It is not the case that “anything goes” in the world of biological evolution. In other words, biological form (pattern) is important: It has meaning beyond itself; a deeper, more general significance. Understanding of biology lies, then, in understanding the evolution and nature of biological form (pattern). Explaining biological form by variation and selection hand-waving argumentation is far from sufficient: The motor does not explain where the car goes. (Woese 2005: 101-102)

      ~ Woese, Carl R. (2005) Evolving Biological Organization. In Microbial Phylogeny and Evolution: Concepts and Controversies (Jan Sapp, ed.). Oxford: Oxford University Press, pp. 99-102.

      • Rob Reno
        December 17, 2017 at 7:31 pm

        It seems between the fields of economics and biology, the exact details may be different but the conceptual difficulties are similar.

  7. Risk Analyst
    December 18, 2017 at 10:55 pm

    Lars’ essay is excellent as usual, short and to the point. However, it is in the category of preferring to curse the darkness rather than lighting a candle. Where do we go with econometrics from here? We have a well tested and studied set of statistical procedures, poor and ambiguous data sets, and researchers unable to come to terms with their bias, acknowledge alternative frameworks or as the paper says to really understand the consequences and behavior of the framework they’ve chosen. I’ve seen worse in the medical field where grand conclusions for the whole US population are made from a tiny sample which was made even smaller after throwing out inconvenient observations that conflicted with the conclusions the researchers wanted. The only econometrics I trust are those I do myself and I usually just skip over the statistical analysis pages in research papers.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.