Pluralism in Economics and Research Assessment Systems
from Donald Gillies
For the last five years I have been conducting research into the effects of research assessment systems such as the research assessment exercise and the research excellence framework in the UK. There are now plans to introduce similar systems in other countries such as France and Italy. My first paper on this subject was published in the post-autistic economics review in 2006 (http://www.paecon.net/PAEReview/issue37/Gillies37.htm), and in December 2008, to coincide with the results of the last UK research assessment exercise, I published a book on this subject entitled: How Should Research be Organised? This is available here at amazon.co.uk and here at amazon.com.
The main result of my research is that research assessment systems have a systematic tendency to decrease the quality of the research output. One reason for this is that they evaluate mainstream research programmes on which the majority of researchers are working more highly than minority research programmes which attract only a few researchers. Yet the history of science shows that minority research programmes are often the ones which produce the best results.
To show why research assessment systems tend to discriminate against minority approaches, let us consider a simple example. Suppose that research is being carried out on some problem and that four different research programmes have been proposed to solve it. It may be almost impossible to say at the beginning which of the four programmes is going to lead to success. Suppose it turns out to be programme number 3. Let us suppose further (which indeed is often the case) that initially programme 3 attracts many fewer researchers than programmes 1,2 & 4. Now it is characteristic of most researchers that they think their own approach to the problem is the correct one, and that other approaches are misguided. If a peer review is conducted by a committee whose researchers are a random sample of those working on the problem, then the majority will be working on programmes 1,2 & 4, and are therefore very likely to give a negative judgement on programme 3. As the result of the recommendation of such a peer review, funding might be withdrawn from programme 3, and the solution of the problem might remain undiscovered for a long time.
Here I am assuming that peer review is used as the method of assessment, but the use of bibliometrics such as citation indices produces exactly the same result. To do well on a citation index, it is necessary for a paper to be cited by a large number of other papers. This is almost impossible for a paper in a minority research programme, since, even if the paper is well received by the few researchers working on the programme, this will not create a sufficiently large number of citations to produce a good result on a citation index.
A recent example of the success of a minority research programme is the discovery that a form of cervical cancer is caused by a preceding infection by the papilloma virus. In 2008, Zur Hausen was awarded the Nobel prize for this discovery. In the research which led to the discovery, however, the majority of researchers favoured the view that the causal agent for cervical cancer was a herpes virus and not a papilloma virus. Zur Hausen was one of the few who favoured the papilloma virus.
The dominance of the herpes virus approach is shown by the fact that, in December 1972, there was an international conference of researchers in the area at Key Biscayne in Florida, which had the title: Herpesvirus and Cervical Cancer. Zur Hasuen attended this conference and made some criticisms of the view that cervical cancer is caused by a herpes virus. It is reported that the audience listened to zur Hausen in stony silence (Mcintyre, 2005, p.35). The summary of the conference written by George Klein (Klein, 1973) does not mention zur Hausen. Clearly at that time, the assessment of zur Hausen’s research programme by peer reviews or citation indices would not have been very favourable, although in the long run zur Hausen proved to be right.
This example shows how research assessment systems can produce bad results in the bio-medical sciences, and, naturally, the situation is worse in economics. In economics, the majority, mainstream, approach can be described as mathematicised neo-classical economics. There are several other approaches such as Keynesian, Marxist, Institutionalist, and so on.. The workings of research assessment systems will give higher valuations to the mainstream approach than to a minority approach, thus encouraging politicians to adopt the advice of neo-classical economists rather than that of other schools. Yet the course of events following the great financial crash of 1929 strongly suggests that Keyneisan economists are more likely to produce a solution to the current crisis (cf. Paul Davidson’s blog). If governments had introduced a vaccine based on the herpes virus, it would have had no effect on the incidence of cervical cancer, whereas the current vaccine based on the papilloma virus is producing excellent results. If governments are going to produce successful interventions, it is necessary for them to disregard the results of research assessment systems which are systematically misleading. Instead, there is the need for the examination of a plurality of approaches whether in the bio-medical sciences or in economics. Often the results of a minority approach will be those which work best in practice.
The example of Zur Hausen and the cause of cervical cancer illustrates the general approach adopted in my book. I criticize research assessment systems by giving examples from the history of science, which are analysed using ideas from the philosophy of science, and which show that the methods of research assessment systems produce bad results.
Klein, G. (1973) Summary of Papers Delivered at the Conference on Herpesvirus and Cervical Cancer (Key Biscayne, Florida), Cancer Research, 33(June 1973), pp. 1557-1563.
McIntyre, P. (2005) Finding the viral link: the story of Harald zur Hausen, Cancer World, July-August, pp. 32-37.