RWER issue 57: Hollanders

Five methodological fallacies in applied econometrics

D.A. Hollanders   

Abstract

This paper discusses five methodological problems in applied econometrics. These are the problem of measurement, data mining, publication bias, the Duhem-Quine critique, and the non-repeatable nature of historical events. These problems form a third source of error next to two other more common sources of error in econometrics, sampling error and specification error. The paper argues that these problems aggravate the already difficult task of testing, but can often be dealt with. In some circumstances however testing itself is inappropriate, and econometrics is better understood as a means for description than for testing. 

JEL classification: B40, C18, C50. Key words: applied econometrics, methodology, philosophy of science.

1. Introduction

 

            Econometrics is a collection of probability statements. That is, estimated regression-coefficients come with a disclaimer that they may be wrong. The disclaimer takes the form of the probability that the estimated coefficient would have popped up if the true value of the coefficient equals zero. If this probability is below 5%, the coefficient is said to be significant. In a world of finite data the probability a true hypothesis is rejected – the type I error-never equals zero and it has to be accepted as a fact of life, even if the model is correctly specified.

            A second concern, besides this Type I error, relates to the if-statement that the model is correctly specified. In its basic from, the ordinary least squares-model assumes regressors exogenous, error terms homoskedastic and uncorrelated, and sometimes normally distributed. Violations of these kind of model-assumptions form a second source of error, called specification error. Testing model-assumptions and thereby enabling correct inference is the core business of econometrics. A large part of econometrics consists then, in the words of Hendry [1980] of ‘test, test, test’.

            To deal with misspecification error and being aware of sampling error distinguishes good econometrics from bad econometrics. Nonetheless exclusively focusing on these two sources of error suggests there are no other possible sources of error. There are. The paper concerns itself with five such sources of error in econometrics; these fall outside the two categories mentioned, and together constitute what may be called methodological errors. These are (1) measurement error, (2) data mining, (3) Duhem-Quine critique, (4) publication bias, (5) historical events being sui generis.

            These methodological concerns are not new and no claim to novelty is made. All the same, they are treated, if mentioned at all, non-systematically in many econometric textbooks, and are dealt with, if treated at all, ad hoc in applied work. It may therefore be useful to group and categorize them. The five concerns all circle around the question

if and how it is possible to test, or if one likes, they form some epistemological disclaimers that comes with testing.

 2.1. Measurement problem and the problem of conceptualizing

 You may download the whole paper at:/ http://www.paecon.net/PAEReview/issue57/Hollanders57.pdf