Home > Uncategorized > P-values are no substitute for thinking

P-values are no substitute for thinking

from Lars Syll

A non-trivial part of statistics education is made up of teaching students to perform significance testing. A problem I have noticed repeatedly over the years, however, is that no matter how careful you try to be in explicating what the probabilities generated by these statistical tests really are, still most students misinterpret them. 

This is not to blame on students’ ignorance, but rather on significance testing not being particularly transparent (conditional probability inference is difficult even to those of us who teach and practice it). A lot of researchers fall prey​ to the same mistakes.

If anything, the above video underlines how important it is not to equate science with statistical calculation. All science entail human judgement, and using statistical models doesn’t relieve us of that necessity. Working with misspecified models, the scientific value of significance testing is actually zero —  even though you’re making valid statistical inferences! Statistical models and concomitant significance tests are no substitutes for doing real science.

In its standard form, a significance test is not the kind of ‘severe test’ that we are looking for in our search for being able to confirm or disconfirm empirical scientific hypotheses. This is problematic for many reasons, one being that there is a strong tendency to accept the null hypothesis since they can’t be rejected at the standard 5% significance level. In their standard form, significance tests bias against new hypotheses by making it hard to disconfirm the null hypothesis.

And as shown over and over again when it is applied, people have a tendency to read “not disconfirmed” as “probably confirmed.” Standard scientific methodology tells us that when there is only say a 10 % probability that pure sampling error could account for the observed difference between the data and the null hypothesis, it would be more “reasonable” to conclude that we have a case of disconfirmation. Especially if we perform many independent tests of our hypothesis and they all give ​the same 10% result as our reported one, I guess most researchers would count the hypothesis as even more disconfirmed.

Most importantly — we should never forget that the underlying parameters we use when performing significance tests are model constructions. Our p-values mean next to nothing if the model is wrong. Statistical​ significance tests DO NOT validate models!

statistical-models-theory-and-practice-2-e-original-imaeahk3hfzrmxz9In journal articles a typical regression equation will have an intercept and several explanatory variables. The regression output will usually include an F-test, with p-1 degrees of freedom in the numerator and n-p in the denominator. The null hypothesis will not be stated. The missing null hypothesis is that all the coefficients vanish, except the intercept.

If F is significant, that is often thought to validate the model. Mistake. The F-test takes the model as given. Significance only means this: if the model is right and the coefficients are 0, it is very unlikely to get such a big F-statistic. Logically, there are three possibilities on the table:
i) An unlikely event occurred.
ii) Or the model is right and some of the coefficients differ from 0.
iii) Or the model is wrong.

  1. Frank Salter
    November 26, 2018 at 8:27 am

    Lars Syll continues to conflate models and theory as if they were the same.

    A function obtained by curve fitting is a “concrete” relationship. Whatever statistical tests may then be applied are irrelevant. They can only inform about implied noise. If a perfect fit is required then fit a polynomial of the same degree as the number of data points. The function will fit the data everywhere and be totally useless at every point away from the data points.

    For a relationship to represent valid theory. it must conform to the quantity calculus and it must not be invalidated by the empirical evidence. These are necessary and sufficient conditions. It would be better if Lars Syll were to start from this point. He would then be able to discuss the fact there are only a handful of economic papers which may be possibly be valid theory. This would be a far better use of his blogs rather than to continue to march on the same old worn out spot which has proved so ineffectual.

  2. November 27, 2018 at 4:35 pm

    Another way of putting it would be that the only true statements economists are able to make are tautological. Problem is, economics as currently conceived and practiced is based on premises that are demonstrably false and they use bad logic to build on them. For a hilarious example, wade through Friedman’s “Methodology in positive economics” sometime. Friedman argues – incorrectly – that since real science uses sketchy “as if” arguments, economists should too. Ha!

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: