Home > Uncategorized > Does it — really — take a model to beat a model? No!

Does it — really — take a model to beat a model? No!

from Lars Syll

Many economists respond to criticism by saying that ‘all models are wrong’ … But the observation that ‘all models are wrong’ requires qualification by the second part of George Box’s famous aphorism — ‘but some are useful’ … The relevant  criticism of models in macroeconomics and finance is not that they are ‘wrong’ but that they have not proved useful in macroeconomics and have proved misleading in finance.

kaykingWhen we provide such a critique, we often hear another mantra to which many economists subscribe: ‘It takes a model to beat a model.’ On the contrary, we believe that it takes facts and observations to beat a model … If a model fails to answer the problem to which it is addressed, it should be put back in the toolbox … It is not necessary to have an alternative tool available to know that the plumber who arrives armed only with a screwdriver is not the tradesman we need.

A similar critique yours truly sometimes encounters is that as long as I cannot come up with some own alternative model to the failing mainstream models, I shouldn’t expect people to pay attention.

This is, however, not only wrong for the reasons given by Kay and King, but is also to utterly misunderstand the role of philosophy and methodology of economics!

As John Locke wrote in An Essay Concerning Human Understanding:

19557-004-21162361The Commonwealth of Learning is not at this time without Master-Builders, whose mighty Designs, in advancing the Sciences, will leave lasting Monuments to the Admiration of Posterity; But every one must not hope to be a Boyle, or a Sydenham; and in an Age that produces such Masters, as the Great-Huygenius, and the incomparable Mr. Newton, with some other of that Strain; ’tis Ambition enough to be employed as an Under-Labourer in clearing Ground a little, and removing some of the Rubbish, that lies in the way to Knowledge.

That’s what philosophy and methodology can contribute to economics — clearing obstacles to science by clarifying limits and consequences of choosing specific modelling strategies, assumptions, and ontologies.

unnameadIt takes a model to beat a model has to be one of the stupider things, in a pretty crowded field, to come out of economics. … I don’t get it. If a model is demonstrably wrong, that should surely be sufficient for rejection. I’m thinking of bridge engineers: ‘look I know they keep falling down but I’m gonna keep building them like this until you come up with a better way, OK?’

Jo Michell

  1. Yoshinori Shiozawa
    September 14, 2020 at 3:27 am

    Readers of this article are requested to see evidencebas(evidencebasedeconomics, Mike Joffe)’s posts on September 12, 2020 at 5:14 pm, and September 14, 2020 at 12:19 am, which are comments on Peter Radford’s article “More on what’s’ missing”. My post on September 12, 2020 at 11:36 may be helpful. Evidencebas’s comments start with the post on September 12, 2020 at 5:00 pm.

    Evidencebas has produced several keywords that would be useful to discuss what should be done now in economics. Those are “incremental mode”, “reactive mode”, “(selective) replacement mode”, and “true belief mode”, among which “reactive mode” and “replacement mode” are the most important now. I emphasize the conditional “now”, because these modes are related to different phases of economics development. Phases change through time. In my (not very humble) opinion, the present is the replacement mode phase, where as Lars Syll seems to believe it is still the reactive mode phase. Difference of opinions between Lars and me depends much on the recognition whether we judge the present phase either a reactive mode or replacement mode. If we are in a reactive mode, Lars’s efforts would be the most effective ones. If we are in a replacement mode, another strategy is required even if criticizing mainstream economics looses all its meanings.

    To add a word on Kay and King, the dictum “it takes a theory to beat a theory” (not “model”) was adopted from jurisdiction world to the history and philosophy of science(s), becaue many historians admitted that ” facts and observations” alone never beat a theory. To make accept some facts and observations important and crucial, it needs a theory, which is not necessarily restricted to (mathematical) models. With this regard, Kay and King are from the start wrong because they cannot distinguish theory and models. Models are only a form of theory. In addition, economic facts are not as simple as the fallen down bridge. Quantitative easing (QE) failed almost everywhere where it was tried, but macroeconomic theory (in our case, New Neoclassical Synthesis) has not yet fallen down. It takes a theory to beat a theory.

    I am old enough to know how rational expectation hypothesis jumped up as the most referred topic in 1970’s, there were severe criticisms against it. I myself thought what a ridiculous idea it is. More than forty years of criticism were ineffective. It would be the time to reconsider the strategy for a reconstruction of economics.

    (I have not read Kay and King (2020). Don’t misunderstand me. I am not opposing to the idea of radical uncertainty. Kay and King’s book seems a good book.)

    • Calgacus
      September 18, 2020 at 8:39 am

      Just happened to open an old book, Keynes & the Classics (1964), edited by Robert Lekachman. He ends the book with an essay of his from Encounter (1963) – and speaking of the (neo)classical theory that saw involuntary unemployment as impossible, he says

      This was still the theory, and in economics, as in other subjects, bad theory holds the ground until better theory displaces it, even in the face of the evidence that millions of the unemployed would have gladly accepted work at almost any rate above zero.

      Surely there are earlier expressions of the same sentiment in economics, but there’s one. Of course it is important to distinguish between theories and models (of theories) – wish more economists would.

      • Yoshinori Shiozawa
        September 18, 2020 at 5:30 pm

        Thank you. Example of Robert Lekachman is quite instructive. Any good observers of economics or sciences would arrive to acknowledge the content of the dictum: It takes a theory to beat a theory. We have various expressions.

  2. Yoshinori Shiozawa
    September 14, 2020 at 3:37 am

    “if criticizing mainstream economics looses all its meanings” must be read “if criticizing mainstream economics does not loose all its meanings.”.

    “the most referred topic in 1970’s, there were severe criticisms against it. ”
    “the most referred topic in 1970’s. There were severe criticisms against it.”

  3. Yoshinori Shiozawa
    September 14, 2020 at 5:15 am

    I found an interesting note among Customer Reviews in Amazon.com by Sami Al Suwailem with a title “It takes a theory to beat a theory”. Admitting the basic truth of the dictum, it argues why Adaptive Market Hypothesis (AMH) cannot beat EMH (efficient market hypothesis).

    • Yoshinori Shiozawa
      September 14, 2020 at 5:32 am

      I have some reservations on Sami Al Suwailem’s contention. His criteria of scientific theory is an ideal type. We must admit various ladders of scientific theories. So some theories may not be as strong to beat a theory. It is necessary to have a theory to beat a theory, but it is possible that an alternative theory does not beat a theory. It depends on how the new theory is strong.

  4. September 14, 2020 at 11:32 am

    At the end of Lars’ posting, Jo Mitchell is missing the point that if a bridge model fails, because of the cost one doesn’t build bridges differently until the bridge model DOESN’T fail: only model bridges.

    I’m not sure whether Lars is in fact making an effort, or whether the RWER editor is simply pasting up such of his previous comments that are worthy of discussion. That would at least explain why we cannot get him to offer an informed opinion himself on our differing views.

    The ‘John Locke’ position being advanced is the Critical Realist one (i.e. that of Tony Lawson, Lars and myself: appreciative of advances made by others even when – like Locke’s empiricism, democracy and ‘rights of ownership’ doctrines – they ultimately misfire). Now Locke’s misfire has become obvious the priority needs to shift from the ‘appreciation’ to the ‘advancement’. The new language of Shannon’s Information Science makes it possible to write new answers both to old problems and the new problems arising from their failure (like people brought up in towns only seeing street pictures rather than interacting with nature).

    For fun, I tried the cinema cartoon on my wife. The picture she’d rather watch was “A Reassuring Life”, whereas mine was “An Inconvenient Truth”. Seems reasonable enough, since her job as a home-maker requires life to go on reassuringly around her, whereas mine as a problem fixer requires me to remove the real inconveniences from inconvenient truths. Would that on this job there were more model-builders like me: like Robert Locke’s apprenticed engineers.

    • September 14, 2020 at 1:27 pm

      I’m in general agreement with what Yoshinori says in his first comment, but this is worthy of discussion: “Kay and King are from the start wrong because they cannot distinguish theory and models. Models are only a form of theory”.

      Are they? Or are models embodied theories?.

      The scientific point of that being that – like say a model railway – they are working models with an ‘inside’ that can go wrong and be shown or studied to see how it works (the dynamic effects of causes sometimes being surprising). By comparison, in ‘black box’ theories the effects are static and built into (implied by) their deductive logic and observable (skin-deep, present-day knowledge) parameters. A computer program is then a theory; it only becomes a model when it is activated in a computer. As the computer includes error correction it always does what it is told, but the results being surprising implies that its logic is misconceived. Either we’ve told it to do the wrong thing or what we think we have to do to the data is mistaken, e.g. we are asking it to add and oranges or (more trickily) variables that can (if rarely) take that form. (I’m thinking of a dating algorithm which successfully coped with the scary millenium change but nevertheless managed to fail (just once): as I remember due to changes in the earth’s orbit causing its fixed leap year correction actually to change in length)!

      Working hypotheses, anyway? Theories use static logic. Models use dynamic logic with static premises, which before anything evolved to be measured could only be measuring method conventions, e.g. the orthogonal coordinates of complex number, chosen for their symmetry.

  5. September 14, 2020 at 3:39 pm

    I agree with Yoshinori, especially his first post. Distinguishing models from theories is key, and without that, neither economics nor economic methodology will make any significant progress.

    In the natural sciences like biology, there are many models, but they are nested within broader theories. These theories are empirically based, and are centrally concerned with causation. It works extremely well, e.g. we now *know* how the human circulatory system works, and the causes of many diseases – and there are literally hundreds (or probably thousands) of similar examples. The iterative interplay between evidence and theorizing about the likely causal mechanism is the key; either can come first.

    The models in natural sciences (apart from some branches of physics, which have their own problems by the way) are *embedded* in an existing empirically-based causal theory. Like the pressure relationships within the circulatory system, which are mathematical, but only became possible when the nature of the system was established. Or the modelling of the spread of infectious diseases, much in the news in recent months, which would be impossible without the germ theory of disease as applied to a specific type of infection.

    One decisive advantage of models embedded in empirically-based causal theories over disembodied models is that when you do it this way, you already have a broad map of the various causal relationships and how they interact. Then by doing a model, which is necessarily a simplification, *you can see what is being omitted*, which as all good modelers know is one of the most important attributes of a model.

    For these ideas in a more fleshed out fashion, with descriptions of examples from both biology and economics, see my 2017 paper “Causal theories, models and evidence in economics—some reflections from the natural sciences.” https://www.cogentoa.com/article/10.1080/23322039.2017.1280983.pdf

    To come back to Yoshinori’s post, economic methodologies that *start* with the examination of the strengths and weaknesses of models are in reactive mode. And almost invariably, their restricted focus means that they miss the big picture, as briefly described above. This is unfortunately true of a large proportion of current economic methodology. Lars’s article is guilty of it too, but he is in “good” company, in the sense that this is what economic methodologists mainly talk about. Also economists, when they talk about methodology.

    And anyway, the central problem of economic theory is that the dominant models are a priori, conjured out of the air on the basis of introspection about “what must be true”. It just isn’t a way of representing the real world that actually works. And starting from such a theory can lead to the most obvious explanations being completely missed! – e.g. in trying to explain the massive China-to-US capital flows starting from the Lucas puzzle (see my “Why does capital flow from poor to rich countries? – the real puzzle” at http://www.paecon.net/PAEReview/issue81/Joffe81.pdf).

    PS: I don’t think it is useful to use examples from computers and their programs, or indeed from physics (and still less from philosophy of physics!).

  6. September 14, 2020 at 8:47 pm

    PS to this PS: compare this with https://rwer.wordpress.com/2020/09/10/michael-woodford-on-models/#comment-172645

    (About using computers and their programs – not their uses – as examples).

  7. Gerald Holtham
    September 15, 2020 at 9:26 am

    There would be general agreement among contributors to this blog that many vaunted economic models are not “useful”. Lars Syll has made that point squarely but sometimes he seems to go further. He criticises the practice itself of constructing a model and working out its implications. For example he took exception to some remarks of Michael Woodford about modelling in general. He comes close to arguing that no model can be useful.
    Now I’ll concede that it is legitimate to criticise a given theory or model, indeed it may be an obligation, even if one does not have an alternative theory to propose. But I think if you criticise a whole way of proceeding, a whole methodology if you like, there is some requirement to indicate what people should do instead. Should we all just study economic history and refrain from trying to generalise about economics at all?
    Lars stresses the unreality and empirical irrelevance of many economic models and, again, one has to agree. Yet he also disparages what techniques of empirical testing that we have. He was dismissive of pilot studies and critical of Du Flo and associates when they got a Nobel prize. And he doesn’t like econometric testing of theories. Of course the results of pilot studies can be overstated and there is plenty of sloppy or tendentious econometrics to criticise but Lars seldom criticises a particular study or piece of work. He delvers an anathema on the whole practice, of statistical analysis for example. Surely one is entitled to ask then: how do we decide on the empirical relevance or usefulness of a given proposition?
    By taking potentially valid criticisms of common malpractices in economics and trying to elevate them to criticisms of methodology, Lars is like a man who locks all known doors and then complains you won’t leave the room.

    • September 16, 2020 at 3:07 pm

      On modelling in general in economics I would certainly argue that the kind of detailed historical analysis that (economic) historians conduct contributes much more to explanation and understanding than the kind of modelling mainstream economics builds on. The heuristic and explanatory value of axiomatic-deductive modelling in social sciences is next to nil, at least if what we want to achieve is explaining and/or understanding real-world phenomena.

      • Yoshinori Shiozawa
        September 16, 2020 at 3:43 pm

        Lars, that is your position. It is a possible way of thinking. But, what your are repeatedly emphasizing is that using mathematics inevitably lead to wrong economics. You have not proved it. It is a proposition that nobody can prove nor refute. And yet, you repeat it as if it is an established fact.

  8. Yoshinori Shiozawa
    September 16, 2020 at 5:27 am

    Please remind of what I have argued in my post on February 23, 2020 at 3:03 am in a comment to Lars Syll’s article Paul Romer explains what went wrong with economics. I have an impression that Lars Syll has been trained as econometrician and has no experience to have worked on theoretical questions in which he does not know whether a proposition is true or not in a given situation. Even if it is formulated in a mathematical form, it is not easy to determine whether the proposition is true or false. But in some cases (not all), to know if a conjecture is true or not is a necessary part of economic research.

    (I do not claim it is all or a major part of research. Majority of economists never come to confront with such a situation. But, in some cases, even in a very rare cases, this is an important part of economic research. As a theoretical economist, I have several occasions to encounter such occasions. For example, Morioka’s theorem was what I wanted to solve but failed. See Marc Lavoie’s book review. The theorem was obtained by Morioka and it is explained in Chapter 4. I have given my personal research history in Section 2.7 in Chapter 2. Another example is the new theory of international values. I had a basic idea already in 1985, but could not build a good theory until 2014. Major difficulty lied in mathematics. See Section 2.6 in the book cite above.)

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.