Does it — really — take a model to beat a model? No!
from Lars Syll
Many economists respond to criticism by saying that ‘all models are wrong’ … But the observation that ‘all models are wrong’ requires qualification by the second part of George Box’s famous aphorism — ‘but some are useful’ … The relevant criticism of models in macroeconomics and finance is not that they are ‘wrong’ but that they have not proved useful in macroeconomics and have proved misleading in finance.
When we provide such a critique, we often hear another mantra to which many economists subscribe: ‘It takes a model to beat a model.’ On the contrary, we believe that it takes facts and observations to beat a model … If a model fails to answer the problem to which it is addressed, it should be put back in the toolbox … It is not necessary to have an alternative tool available to know that the plumber who arrives armed only with a screwdriver is not the tradesman we need.
A similar critique yours truly sometimes encounters is that as long as I cannot come up with some own alternative model to the failing mainstream models, I shouldn’t expect people to pay attention.
This is, however, not only wrong for the reasons given by Kay and King, but is also to utterly misunderstand the role of philosophy and methodology of economics!
As John Locke wrote in An Essay Concerning Human Understanding:
The Commonwealth of Learning is not at this time without Master-Builders, whose mighty Designs, in advancing the Sciences, will leave lasting Monuments to the Admiration of Posterity; But every one must not hope to be a Boyle, or a Sydenham; and in an Age that produces such Masters, as the Great-Huygenius, and the incomparable Mr. Newton, with some other of that Strain; ’tis Ambition enough to be employed as an Under-Labourer in clearing Ground a little, and removing some of the Rubbish, that lies in the way to Knowledge.
That’s what philosophy and methodology can contribute to economics — clearing obstacles to science by clarifying limits and consequences of choosing specific modelling strategies, assumptions, and ontologies.
It takes a model to beat a model has to be one of the stupider things, in a pretty crowded field, to come out of economics. … I don’t get it. If a model is demonstrably wrong, that should surely be sufficient for rejection. I’m thinking of bridge engineers: ‘look I know they keep falling down but I’m gonna keep building them like this until you come up with a better way, OK?’
Leave a reply to evidencebasedeconomics Cancel reply
—– look inside —– $5.94 / $20.00
—– look inside —– $4.90 / $8.00
—– look inside —– $15.99
—– look inside —– $5.99 / 12.99
—– look inside —– $5.93 / $12.99
—– look inside —– $4.97 / $9.90
—— Ugarteche, Puyana and Madi ——
Gerson Lima / Maria Alejandra Madi
Edward Fullbrook and Jamie Morgan
————— Michael Hudson ————–
Maria Alejandra Madi / Jack Reardon
————- Edward Fullbrook ————-
—————— Steve Keen —————–
————— Richard Smith —————
————– Gustavo Marques————
– Victor Beker and Beniamino Moro –
————– Lars Pålsson Syll ————-
—————– Stuart Birks —————-
Edward Fullbrook and Jamie Morgan
———— Armando Ochangco ———-
Shimshon Bichler / Jonathan Nitzan
————— Mauro Gallegati ————–
————— Herman Daly —————-
————— Asad Zaman —————
—————– C. T. Kurien —————
————— Robert Locke —————-
Real-World Economics Review Blog 
Readers of this article are requested to see evidencebas(evidencebasedeconomics, Mike Joffe)’s posts on September 12, 2020 at 5:14 pm, and September 14, 2020 at 12:19 am, which are comments on Peter Radford’s article “More on what’s’ missing”. My post on September 12, 2020 at 11:36 may be helpful. Evidencebas’s comments start with the post on September 12, 2020 at 5:00 pm.
Evidencebas has produced several keywords that would be useful to discuss what should be done now in economics. Those are “incremental mode”, “reactive mode”, “(selective) replacement mode”, and “true belief mode”, among which “reactive mode” and “replacement mode” are the most important now. I emphasize the conditional “now”, because these modes are related to different phases of economics development. Phases change through time. In my (not very humble) opinion, the present is the replacement mode phase, where as Lars Syll seems to believe it is still the reactive mode phase. Difference of opinions between Lars and me depends much on the recognition whether we judge the present phase either a reactive mode or replacement mode. If we are in a reactive mode, Lars’s efforts would be the most effective ones. If we are in a replacement mode, another strategy is required even if criticizing mainstream economics looses all its meanings.
To add a word on Kay and King, the dictum “it takes a theory to beat a theory” (not “model”) was adopted from jurisdiction world to the history and philosophy of science(s), becaue many historians admitted that ” facts and observations” alone never beat a theory. To make accept some facts and observations important and crucial, it needs a theory, which is not necessarily restricted to (mathematical) models. With this regard, Kay and King are from the start wrong because they cannot distinguish theory and models. Models are only a form of theory. In addition, economic facts are not as simple as the fallen down bridge. Quantitative easing (QE) failed almost everywhere where it was tried, but macroeconomic theory (in our case, New Neoclassical Synthesis) has not yet fallen down. It takes a theory to beat a theory.
I am old enough to know how rational expectation hypothesis jumped up as the most referred topic in 1970’s, there were severe criticisms against it. I myself thought what a ridiculous idea it is. More than forty years of criticism were ineffective. It would be the time to reconsider the strategy for a reconstruction of economics.
(I have not read Kay and King (2020). Don’t misunderstand me. I am not opposing to the idea of radical uncertainty. Kay and King’s book seems a good book.)
Just happened to open an old book, Keynes & the Classics (1964), edited by Robert Lekachman. He ends the book with an essay of his from Encounter (1963) – and speaking of the (neo)classical theory that saw involuntary unemployment as impossible, he says
Surely there are earlier expressions of the same sentiment in economics, but there’s one. Of course it is important to distinguish between theories and models (of theories) – wish more economists would.
Thank you. Example of Robert Lekachman is quite instructive. Any good observers of economics or sciences would arrive to acknowledge the content of the dictum: It takes a theory to beat a theory. We have various expressions.
Errata:
“if criticizing mainstream economics looses all its meanings” must be read “if criticizing mainstream economics does not loose all its meanings.”.
“the most referred topic in 1970’s, there were severe criticisms against it. ”
>
“the most referred topic in 1970’s. There were severe criticisms against it.”
I found an interesting note among Customer Reviews in Amazon.com by Sami Al Suwailem with a title “It takes a theory to beat a theory”. Admitting the basic truth of the dictum, it argues why Adaptive Market Hypothesis (AMH) cannot beat EMH (efficient market hypothesis).
I have some reservations on Sami Al Suwailem’s contention. His criteria of scientific theory is an ideal type. We must admit various ladders of scientific theories. So some theories may not be as strong to beat a theory. It is necessary to have a theory to beat a theory, but it is possible that an alternative theory does not beat a theory. It depends on how the new theory is strong.
At the end of Lars’ posting, Jo Mitchell is missing the point that if a bridge model fails, because of the cost one doesn’t build bridges differently until the bridge model DOESN’T fail: only model bridges.
I’m not sure whether Lars is in fact making an effort, or whether the RWER editor is simply pasting up such of his previous comments that are worthy of discussion. That would at least explain why we cannot get him to offer an informed opinion himself on our differing views.
The ‘John Locke’ position being advanced is the Critical Realist one (i.e. that of Tony Lawson, Lars and myself: appreciative of advances made by others even when – like Locke’s empiricism, democracy and ‘rights of ownership’ doctrines – they ultimately misfire). Now Locke’s misfire has become obvious the priority needs to shift from the ‘appreciation’ to the ‘advancement’. The new language of Shannon’s Information Science makes it possible to write new answers both to old problems and the new problems arising from their failure (like people brought up in towns only seeing street pictures rather than interacting with nature).
For fun, I tried the cinema cartoon on my wife. The picture she’d rather watch was “A Reassuring Life”, whereas mine was “An Inconvenient Truth”. Seems reasonable enough, since her job as a home-maker requires life to go on reassuringly around her, whereas mine as a problem fixer requires me to remove the real inconveniences from inconvenient truths. Would that on this job there were more model-builders like me: like Robert Locke’s apprenticed engineers.
I’m in general agreement with what Yoshinori says in his first comment, but this is worthy of discussion: “Kay and King are from the start wrong because they cannot distinguish theory and models. Models are only a form of theory”.
Are they? Or are models embodied theories?.
The scientific point of that being that – like say a model railway – they are working models with an ‘inside’ that can go wrong and be shown or studied to see how it works (the dynamic effects of causes sometimes being surprising). By comparison, in ‘black box’ theories the effects are static and built into (implied by) their deductive logic and observable (skin-deep, present-day knowledge) parameters. A computer program is then a theory; it only becomes a model when it is activated in a computer. As the computer includes error correction it always does what it is told, but the results being surprising implies that its logic is misconceived. Either we’ve told it to do the wrong thing or what we think we have to do to the data is mistaken, e.g. we are asking it to add and oranges or (more trickily) variables that can (if rarely) take that form. (I’m thinking of a dating algorithm which successfully coped with the scary millenium change but nevertheless managed to fail (just once): as I remember due to changes in the earth’s orbit causing its fixed leap year correction actually to change in length)!
Working hypotheses, anyway? Theories use static logic. Models use dynamic logic with static premises, which before anything evolved to be measured could only be measuring method conventions, e.g. the orthogonal coordinates of complex number, chosen for their symmetry.
I agree with Yoshinori, especially his first post. Distinguishing models from theories is key, and without that, neither economics nor economic methodology will make any significant progress.
In the natural sciences like biology, there are many models, but they are nested within broader theories. These theories are empirically based, and are centrally concerned with causation. It works extremely well, e.g. we now *know* how the human circulatory system works, and the causes of many diseases – and there are literally hundreds (or probably thousands) of similar examples. The iterative interplay between evidence and theorizing about the likely causal mechanism is the key; either can come first.
The models in natural sciences (apart from some branches of physics, which have their own problems by the way) are *embedded* in an existing empirically-based causal theory. Like the pressure relationships within the circulatory system, which are mathematical, but only became possible when the nature of the system was established. Or the modelling of the spread of infectious diseases, much in the news in recent months, which would be impossible without the germ theory of disease as applied to a specific type of infection.
One decisive advantage of models embedded in empirically-based causal theories over disembodied models is that when you do it this way, you already have a broad map of the various causal relationships and how they interact. Then by doing a model, which is necessarily a simplification, *you can see what is being omitted*, which as all good modelers know is one of the most important attributes of a model.
For these ideas in a more fleshed out fashion, with descriptions of examples from both biology and economics, see my 2017 paper “Causal theories, models and evidence in economics—some reflections from the natural sciences.” https://www.cogentoa.com/article/10.1080/23322039.2017.1280983.pdf
To come back to Yoshinori’s post, economic methodologies that *start* with the examination of the strengths and weaknesses of models are in reactive mode. And almost invariably, their restricted focus means that they miss the big picture, as briefly described above. This is unfortunately true of a large proportion of current economic methodology. Lars’s article is guilty of it too, but he is in “good” company, in the sense that this is what economic methodologists mainly talk about. Also economists, when they talk about methodology.
And anyway, the central problem of economic theory is that the dominant models are a priori, conjured out of the air on the basis of introspection about “what must be true”. It just isn’t a way of representing the real world that actually works. And starting from such a theory can lead to the most obvious explanations being completely missed! – e.g. in trying to explain the massive China-to-US capital flows starting from the Lucas puzzle (see my “Why does capital flow from poor to rich countries? – the real puzzle” at http://www.paecon.net/PAEReview/issue81/Joffe81.pdf).
PS: I don’t think it is useful to use examples from computers and their programs, or indeed from physics (and still less from philosophy of physics!).
PS to this PS: compare this with https://rwer.wordpress.com/2020/09/10/michael-woodford-on-models/#comment-172645
(About using computers and their programs – not their uses – as examples).
There would be general agreement among contributors to this blog that many vaunted economic models are not “useful”. Lars Syll has made that point squarely but sometimes he seems to go further. He criticises the practice itself of constructing a model and working out its implications. For example he took exception to some remarks of Michael Woodford about modelling in general. He comes close to arguing that no model can be useful.
Now I’ll concede that it is legitimate to criticise a given theory or model, indeed it may be an obligation, even if one does not have an alternative theory to propose. But I think if you criticise a whole way of proceeding, a whole methodology if you like, there is some requirement to indicate what people should do instead. Should we all just study economic history and refrain from trying to generalise about economics at all?
Lars stresses the unreality and empirical irrelevance of many economic models and, again, one has to agree. Yet he also disparages what techniques of empirical testing that we have. He was dismissive of pilot studies and critical of Du Flo and associates when they got a Nobel prize. And he doesn’t like econometric testing of theories. Of course the results of pilot studies can be overstated and there is plenty of sloppy or tendentious econometrics to criticise but Lars seldom criticises a particular study or piece of work. He delvers an anathema on the whole practice, of statistical analysis for example. Surely one is entitled to ask then: how do we decide on the empirical relevance or usefulness of a given proposition?
By taking potentially valid criticisms of common malpractices in economics and trying to elevate them to criticisms of methodology, Lars is like a man who locks all known doors and then complains you won’t leave the room.
On modelling in general in economics I would certainly argue that the kind of detailed historical analysis that (economic) historians conduct contributes much more to explanation and understanding than the kind of modelling mainstream economics builds on. The heuristic and explanatory value of axiomatic-deductive modelling in social sciences is next to nil, at least if what we want to achieve is explaining and/or understanding real-world phenomena.
Lars, that is your position. It is a possible way of thinking. But, what your are repeatedly emphasizing is that using mathematics inevitably lead to wrong economics. You have not proved it. It is a proposition that nobody can prove nor refute. And yet, you repeat it as if it is an established fact.
Please remind of what I have argued in my post on February 23, 2020 at 3:03 am in a comment to Lars Syll’s article Paul Romer explains what went wrong with economics. I have an impression that Lars Syll has been trained as econometrician and has no experience to have worked on theoretical questions in which he does not know whether a proposition is true or not in a given situation. Even if it is formulated in a mathematical form, it is not easy to determine whether the proposition is true or false. But in some cases (not all), to know if a conjecture is true or not is a necessary part of economic research.
(I do not claim it is all or a major part of research. Majority of economists never come to confront with such a situation. But, in some cases, even in a very rare cases, this is an important part of economic research. As a theoretical economist, I have several occasions to encounter such occasions. For example, Morioka’s theorem was what I wanted to solve but failed. See Marc Lavoie’s book review. The theorem was obtained by Morioka and it is explained in Chapter 4. I have given my personal research history in Section 2.7 in Chapter 2. Another example is the new theory of international values. I had a basic idea already in 1985, but could not build a good theory until 2014. Major difficulty lied in mathematics. See Section 2.6 in the book cite above.)
Models ‘beating’ models seems nonsensical to me. Models are not persons. They are not even institutional persons. I suggest a different route.
In their book ‘Practical Wisdom’ Barry Schwartz and Kenneth Sharpe point out “We are, more than ever, longing for certainty and an end to the maze of unanswerable questions.” And impossibility for us, say the authors. “Practical wisdom means knowing how to balance conflicting aims and principles. This kind of wisdom acknowledges that uncertain risk cannot be eliminated, but guides us in becoming wiser about how we manage it.” “Practical wisdom requires an appreciation that there is no perfect choice, and that each choice has benefits, drawbacks, and uncertainties.” “Wisdom depends on something besides cold, hard facts. It requires lived experience and knowledge of the people for and about whom we are making decisions.” In short, science is not the ultimate authority of human community and culture. Likewise, this authority is entrusted to what we do in daily life, outside and inside science. Here I suggest the following be our guide.
1. Start with the data (what we do know and what we can control)
2. Avoid black and white thinking
3. Start with the rules, and then consider wise modifications
4. Learn to accept uncertainty—it is a key ingredient for fostering wisdom
Radical Uncertainty
Decision-Making Beyond the Numbers
By: John Kay, Mervyn King
Great book. Listening to audio version.
i don’t seen any necessity to set up the historical specific approach as opposition to the effort to find valid generalisations. Both have a place. I agree with evidencebase’s description of how matters proceed in natural sciences and that economics is different. The reason for the difference, I believe, is not perversity for its own sake but because the phenomena under investigation in economics are not stable. In a natural science you can identify causal law that always holds within a given domain and you can then build models of a particular situation that embody that causal law as well as more particular features. Economics has no such general causal laws and some economists have tried to replace them with an axiom of self-interest. The assumption is useful in certain situations – auction design is a topical one at present – but as a general strategy it has been a failure. One reason for the failure is that the rational self-interest assumption has gone along with the desire to build theoretical models (i.e. ones based on stated axiomatic assumptions and not meant to fit a particular situation) that have analytic solutions. The desire for “solutions” has been counter-productive because it has meant the assumptions of the theory-model are set so as to enable a solution rather than being set so as to characterize real situations. Doing the latter would preclude analytic solutions and require computer simulation to arrive at results or understanding. Theorems to create a framework for thinking by establishing limiting cases have their place but one then has to move on, not suppose they help you to understand the real world. Reproducing those “results” should certainly not be a requirement of an empirical model, which has all too often been imposed. Empirical research should take over to build useful models, which go beyond stylized history but which will not usually be enormously general. They will illuminate a given economy or sub-economy for a time. Because no model will capture everything affecting a given situation they will inevitably be stochastic and their testing and validation will inevitably require the use of statistics. The evolutionary economics school, stemming from the work of Nelson and Winter does proceed in that way. So-called ABM models attempt to do the same.
But it is much easier to teach economics students the results of solvable models, without empirical content. That is “theory” after all. If they stay in the subject they can move on to the other stuff, the “applied” economics, ad hoc and lacking generality and so having lower status.
ghholtham referred to two fields that Lars Syll rarely argues.
Probably. Lars Syll does not know evolutionary economics after Nelson and Winter even if he knows Thorstein Veblen’s paper on evolutionary economics. If not, he must be intentionally ignoring it. It is now a big group of economics that is explicitly anti-neoclassical economics and does not use models too much. See Geoffrey Hodgson’s recent book Is there a future for heterodox economics? Although he knows the future is not very bright, he is proposing several concrete strategies to take.
As for ABS (Agent-Based Simulation), I have proposed in my paper A guided tour of the backside of agent-based simulation to consider it as a third mode of scientific research after theory (speculation) and experiments.