Pluralism in Economics and Research Assessment Systems
from Donald Gillies
For the last five years I have been conducting research into the effects of research assessment systems such as the research assessment exercise and the research excellence framework in the UK. There are now plans to introduce similar systems in other countries such as France and Italy. My first paper on this subject was published in the post-autistic economics review in 2006 (http://www.paecon.net/PAEReview/issue37/Gillies37.htm), and in December 2008, to coincide with the results of the last UK research assessment exercise, I published a book on this subject entitled: How Should Research be Organised? This is available here at amazon.co.uk and here at amazon.com.
The main result of my research is that research assessment systems have a systematic tendency to decrease the quality of the research output. One reason for this is that they evaluate mainstream research programmes on which the majority of researchers are working more highly than minority research programmes which attract only a few researchers. Yet the history of science shows that minority research programmes are often the ones which produce the best results.
To show why research assessment systems tend to discriminate against minority approaches, let us consider a simple example. Suppose that research is being carried out on some problem and that four different research programmes have been proposed to solve it. It may be almost impossible to say at the beginning which of the four programmes is going to lead to success. Suppose it turns out to be programme number 3. Let us suppose further (which indeed is often the case) that initially programme 3 attracts many fewer researchers than programmes 1,2 & 4. Now it is characteristic of most researchers that they think their own approach to the problem is the correct one, and that other approaches are misguided. If a peer review is conducted by a committee whose researchers are a random sample of those working on the problem, then the majority will be working on programmes 1,2 & 4, and are therefore very likely to give a negative judgement on programme 3. As the result of the recommendation of such a peer review, funding might be withdrawn from programme 3, and the solution of the problem might remain undiscovered for a long time.
Here I am assuming that peer review is used as the method of assessment, but the use of bibliometrics such as citation indices produces exactly the same result. To do well on a citation index, it is necessary for a paper to be cited by a large number of other papers. This is almost impossible for a paper in a minority research programme, since, even if the paper is well received by the few researchers working on the programme, this will not create a sufficiently large number of citations to produce a good result on a citation index.
A recent example of the success of a minority research programme is the discovery that a form of cervical cancer is caused by a preceding infection by the papilloma virus. In 2008, Zur Hausen was awarded the Nobel prize for this discovery. In the research which led to the discovery, however, the majority of researchers favoured the view that the causal agent for cervical cancer was a herpes virus and not a papilloma virus. Zur Hausen was one of the few who favoured the papilloma virus.
The dominance of the herpes virus approach is shown by the fact that, in December 1972, there was an international conference of researchers in the area at Key Biscayne in Florida, which had the title: Herpesvirus and Cervical Cancer. Zur Hasuen attended this conference and made some criticisms of the view that cervical cancer is caused by a herpes virus. It is reported that the audience listened to zur Hausen in stony silence (Mcintyre, 2005, p.35). The summary of the conference written by George Klein (Klein, 1973) does not mention zur Hausen. Clearly at that time, the assessment of zur Hausen’s research programme by peer reviews or citation indices would not have been very favourable, although in the long run zur Hausen proved to be right.
This example shows how research assessment systems can produce bad results in the bio-medical sciences, and, naturally, the situation is worse in economics. In economics, the majority, mainstream, approach can be described as mathematicised neo-classical economics. There are several other approaches such as Keynesian, Marxist, Institutionalist, and so on.. The workings of research assessment systems will give higher valuations to the mainstream approach than to a minority approach, thus encouraging politicians to adopt the advice of neo-classical economists rather than that of other schools. Yet the course of events following the great financial crash of 1929 strongly suggests that Keyneisan economists are more likely to produce a solution to the current crisis (cf. Paul Davidson’s blog). If governments had introduced a vaccine based on the herpes virus, it would have had no effect on the incidence of cervical cancer, whereas the current vaccine based on the papilloma virus is producing excellent results. If governments are going to produce successful interventions, it is necessary for them to disregard the results of research assessment systems which are systematically misleading. Instead, there is the need for the examination of a plurality of approaches whether in the bio-medical sciences or in economics. Often the results of a minority approach will be those which work best in practice.
The example of Zur Hausen and the cause of cervical cancer illustrates the general approach adopted in my book. I criticize research assessment systems by giving examples from the history of science, which are analysed using ideas from the philosophy of science, and which show that the methods of research assessment systems produce bad results.
References
Klein, G. (1973) Summary of Papers Delivered at the Conference on Herpesvirus and Cervical Cancer (Key Biscayne, Florida), Cancer Research, 33(June 1973), pp. 1557-1563.
McIntyre, P. (2005) Finding the viral link: the story of Harald zur Hausen, Cancer World, July-August, pp. 32-37.
The problem is not simply the formal structures, criteria, factor weighting and procedures of typical research assesssment systems. Ultimately it comes down to the character, or lack of it, of those in academia who produce the inputs into the systems, those who do the scoring in their assessments, and those who utilize the scores. Nothing prevents mainstream academics from venturing outside of the usual parameters, angles, scopes and topics of neoclassical economics and the like of the supposed “mainstream” except cowardice, self-imposed censorship, careerism, opportunism and perhaps some cognitive dissonance. But the charge of cherry picking, ideologically-skewed scoring of articles, self-censorship in choice of topics and methodological approaches, ideological cloning etc does not apply only to the neoclassicals or right-wing: it applies to those who see themselves in careerist ways rather than as scientists or dedicated to a transcendent cause of whatever ideological stripe which demands methodological objectivity to realize the goals of the cause (one cannot transform, treat or eliminate what one does not concretely understand). I am still fond of the statement made by Marx in his letter to Arnold Ruge in 1843 when Marx was being urged to be more utopian and to focus on what future forms of socialism would look like:
“If the construction of the future and its completion for all time is not our task, all the more certain is what we must accomplish in the present; I mean, the ruthless criticism of everything that exists; that criticism being ruthless in the sense that it neither fears its own results nor fears conflict with the powers that be.”
let me give an exaple in economics. In 1979 I was approached by someone in the NSF economics granting program — who heard I was about to embark on a research topic about reforming the world’s international payments and financial system to avoid the kind of breakdown that occurred in the 1929-1930 and led to the Great Depression. He encouraged me to submit a grant proposal to NSF. And so I did.
Needless to say my proposal was turned down with a majority of the inside per reviewers giving it low marks. But the most revealing comment was by one peer insider who wrote: “Davidson has had suprising success in his research ouput publication — BUT he marches to a different drummer than most of us. Therefore let him find his own grant money and not try to use ours!”
These evaluations of my 1980 research proposal are filed in the archives of the Duke University library collection of correspondence of economists.
Paul Davidson
n
Sometime ago I nearly had a paper, or article published on Transfinancial Economics by Professor Prem Sikka. He was impressed by the evolving internet material on the subject, and realized the importance of getting it better known to a wider public. Anyway, the paper was sent, and I signed the copyright release forms. However, due to an “argument” I withdrew it from publication.
For a laugh, I sent another version to the LSE but ofcourse I knew it would be rejected…and I was right but I got a rather “strange” email! I wish I had a copy of it rather than delete it.
All is not lost though, George Soros, and his emerging Institute of New Economic Thinking is meant to take a far more advanced, and more open minded approach to radical ideas in economics. Obviously, TFE would fit the bill..hopefully…!
Dear Donald Gillies,
John Gelles pronounces his name Gellies or Gellees. So we are almost twins.
We need:
o . full employment budgeting
o . inflation protected savings (like TIPS accounts)
o . debt-free sovereign money supplements (like quantitative easing)
o . government seeding investment in the needed-now hydrogen economy
o . an end to income tax and all legalistic screwing of consumers and most workers
o . equity in place of all law where possible
o . brevity in all rules because a priori efforts are no more able to predict their effect than the models you challenge. very brief rules — followed by good justices who have the benefit of being timed to actual events — let them decide and kill off stare decisis.
o . the purpose we find in functional finance — without it we flounder around.
o . focus on zero poverty not on upper bounds.
o . cost accounting and cost plus pricing
o . the end of patents and copyrights and beginning of prize juries as needed to reward genius and competence.
o . going back to bed more often
my website is a mess at http://www.ustaxreform.us
it takes you into my head and on to email links to me on every page