Home > Uncategorized > Viewed as an element of scientific method, are tests for predictive power best seen as tests for the ability of a theory to predict?

Viewed as an element of scientific method, are tests for predictive power best seen as tests for the ability of a theory to predict?

from Adam Fforde

Whilst it may superficially appear clear, an alleged ability of a theory to predict is easily shown to depend upon a host of tangled factors, so things are not clear at all.

At an extreme, to start with, a theory that is right 51% of the time could feasibly be described as predictive, but is not likely to be. Yet if the point is to win bets placed very many times, then it could be thought of as predictive. Theories from physics, such a Newton’s laws of motion, are widely felt to be predictive, but this is within certain bounds, about which quite a lot is known. On the one hand, for example, as velocities approach the speed of light, so mass, assumed constant, is thought to vary. Again, just as Newtonian space is conceptually made up of lines, with no presence outside one dimension, and points, with no presence at all, so mass is assumed to be something that can be situated at a single point, a centre of gravity. All of this can be understood to mean that the apparent clarity of Newtonian physics is not what makes it acceptable under some circumstance as a guide to action. The extent to which it matters that observables necessarily seen to flout the scientific metaphor involved – lines as measured have width, points in time have duration, forces cannot be directly observed – and are therefore associated with an ability to insure the resulting object (say, an aeroplane) depends on the local and social context. To develop this argument, if gun-laying was being done for “extremely inaccurate” riflemen in a war of accepted extreme levels of attrition (consider if the guns were aimed by cloned animals), then prediction that entailed a 51% accuracy rate could be, one can imagine, accepted, as it would arguably “win the war”. There is no escape from the social context in which beautiful theory like Newton’s might – or might not – be used.[1]  

Further, as Lakatos 1970 pointed out, to make sense of data requires observation theories, and the accuracy of observation – whatever that means – likely has some bearing on the way in which terms within theory map to observables. Thus, whilst predictive power may seem clear, it is not. One is tempted to conclude that predictive power exists when it is said to exist; this is done by some community, with reference to all the complex tangles human communities generally seem to be able to manage. They will therefore likely often argue about it. If this conclusion is reasonable then what can be said about predictive power?

What comes from my discussion of the contrast between the different criteria defining the acceptability of theory that we find in Crombie and Nisbet is that prediction is most important in that it requires two things, and neither are to do with prediction per se, as it is generally understood (e.g. “getting a rocket to the moon”).

First is the requirement for comparison between theories as a matter of procedure. If, however, this is not part of scientific procedure and a single truth is required, then this choice is logically done outside of scientific procedure.

Second is explicit management of the shift between suspension of scepticism in theorisation (Crombie’s inductive phase, when theory is empirically-founded) and its resumption when theory can be, if the empirics suggest, abandoned. Following such norms, theory has to be protected, but not for ever, and it has also to be killable.

This view of the nature of predictability seems to me to be novel, and also to allow us to get away from somewhat fruitless debates. Economics as a science is about providing insights and improved understandings, and this is shown by its method.

[1] As McCloskey 1985 puts it: “The numbers are necessary material. But they are not sufficient to bring the matter to a scientific conclusion. Only the scientists can do that, because “conclusion” is a human idea, not Nature’s. It is a property of human minds, not of the statistics.” (p. 112). And: “It is not true, as most economists think, that . . . statistical significance is a preliminary screen, a necessary condition, through which empirical estimates should be put. Economists will say, “Well, I want to know if the coefficient exists, don’t I?” Yes, but statistical significance can’t tell you. Only the magnitude of the coefficient, on the scale of what counts in practical, engineering terms as nonzero, tells you. It is not the case that statistically insignificant coefficients are in effect zero” (stress added p. 118). Quoted in Fforde, 2013.

Adam Fforde, “Economics as a science: understanding its procedures and the irrelevance of prediction”, real-world economics review, issue no. 81, 30 September 2017, pp. 91-109, http://www.paecon.net/PAEReview/issue81/Fforde81.pdf

  1. Rhonda Kovac
    December 3, 2017 at 6:32 pm

    Whenever I read discussions of statistical correlation and the inductive conclusion of predictability, I think of Bertrand Russell’s story. It’s about a turkey, fed every morning without fail, who, “being a good inductivist”, concludes this will continue. But then his throat is cut on Thanksgiving Day

  2. C-R D
    December 3, 2017 at 8:38 pm

    It is strange that the concept of linearity appears nowhere in the article.

  3. December 3, 2017 at 9:36 pm

    Insofar as rank ordering theories for their predictive power goes; Time series predictions i.e. predicting the next element in the time series, must always be judged against the base prediction that it will the same as the previous known value. You don’t get credit in other words for predicting that things will remain the same. On the other hand, many economic forecasts do much worse than this. Penn has a group out of Wharton that has formalized the ranking of forecasting models and this is their starting point.

  4. Frank Salter
    December 4, 2017 at 7:10 am

    Drawing from two frequently cited economics textbooks, in his paper, Adam Fforde presents alternate interpretations of how the scientific method might be applied to economic theorising. He argues that economic methodology is consistent with the interpretation introduced by the sociologist, Nisbet, and that this is demonstrated by the economic literature. While true, it is doubtful that any physical scientist would recognise such an interpretation as being the scientific method.

    However, the Wikipedia entry, ‘Scientific method’, is recognisable, by practising scientists, as being a reasonable description of the techniques whose application constitutes the scientific method and aligns closely to the method, Fforde attributes to Crombie.

    Science develops quantitative relationships which are intended to describe reality, so if their evaluated results are not consistent with the empirical evidence then these hypotheses are rejected. Wikipedia also has an entry entitled ‘Scientific theory’ which sets out what is required for a hypothesis to be accepted as theory — a much more stringent definition than the vernacular usage found in economics. A theory must be both falsifiable and also accepted as a theory by the scientific community. Nisbet appears only to require a theory to be accepted by peer review. While this is also a necessary condition to establish a scientific theory, it is not a sufficient condition. To be a scientific theory, it must be both falsifiable and accepted as valid by the scientific community — which is the sufficient and necessary condition.

    At present, a major problem is the acceptance by academic economists of a scientific theory which is so watered down that it ceases to be any form of scientific method. By simply applying the tool of dimensional analysis to production theory, the number of scientifically acceptable theories is reduced enormously. Every hypothesis based upon a production function flounders. They are all invalid — not one has dimensions which are small positive or negative integers! Not one can be accepted as scientific theory! However, this does not imply that production functions are not useful as interpolating relationships. Current economic thinking appears to be unable to recognise that they are totally different — correlation does not imply causation! At present economic analysis conflates the two — big big mistake!

    In summary, Adam Fforde describes the present economic assumptions. As yet, there are no scientifically accepted theories of production! Not one meet both parts of the sufficient and necessary condition described above. To the best of my knowledge, only a single analysis meets the first part of the requirement for being a theory. It passes the non-falsifiable test. It is ‘Transient development’ — RWER-81. The rest is up to all of you!

  5. December 4, 2017 at 9:59 am

    It is not alleged. Prediction from theory, or from the application of theory in research depends on a host of tangled, in part tacit, and often complexly intertwined factors. The entire enterprise of science is dependent the same way. Andrew Pickering calls this the “Mangle of Practice.” John Law refers to is as the “Mess of Scientific Research.” The only clear-cut thing about science is that it’s not clear cut.

  6. Risk Analyst
    December 4, 2017 at 6:25 pm

    I have a different interpretation of this article than Frank or Peter’s. My takeaway is the emphasis on the critical difference between statistical and economic significance as hinted at in the title. Many modelers beat the data until it confesses a statistical significance justifying a position. That does not create an economic significance in which the relationship is important enough for policy or analytical purposes. I recall that back in the salad days of real business cycle theory, while flailing around looking for explanations of why the markets and economy were doing what they were doing, that year’s failure of the Peruvian anchovy harvest was seriously included as a factor. So while anchovies were statistically significant, they were most likely not economically significant.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s