Let users decide if research is relevant
Peer review, a necessary criterion of excellence, is no longer a sufficient criterion on its own. The relevance of the peer review group must also enter the equation. This piece from 2005 appeared as part of the discussion in Australia about assessment of the broader impact of research in the lead up to the proposed Research Quality Framework (RQF).
First published in The Australian, p. 39.
Access on QUT eprints.
The peer review process needs to be broadened and expanded to decide which theoretical work deserves public support, writes Arun Sharma.
The debate about the effect of research, the relationship between effect and excellence, and whether end users should be included in the peer review process is an interesting one.
It reminds me of an incident last year when a colleague from the US, Jeff Ullman, was visiting Sydney for the scientific advisory group meeting of National ICT Australia. The meeting was in the NSW government offices on the 44th floor of Sydney's Grosvenor Tower.
I was escorting my colleague to the venue in a crowded lift when I spotted someone I knew. I introduced my visitor as a professor of computer science at Stanford University. My Australian colleague appeared impressed, but there was not a flicker of interest from any of the other people in the lift.
I was escorting my colleague to the venue in a crowded lift when I spotted someone I knew. I introduced my visitor as a professor of computer science at Stanford University. My Australian colleague appeared impressed, but there was not a flicker of interest from any of the other people in the lift.
I then mentioned that Ullman was one of the world's leading database theoreticians. A couple of people in the lift—possibly IT consultants—took more notice and glanced at Ullman. I then added that Ullman was also the second most-cited computer scientist in the world. This attracted some more attention. I finally mentioned that one of Ullman's PhD students had dropped out and co-founded Google. Suddenly something registered, and everyone in the lift was looking at Ullman.
He is a rare example of a researcher who has made a foundational contribution, whose work has received exceptional peer recognition, and whose ideas have also had influence on the scale of Google. In most cases, the link between peer recognition and wider impact is far less direct.
As a hypothetical scenario, let us take the example of a group of theoretical physicists and a group of cultural theorists. Both groups do work that is abstract, elegant, intellectually satisfying, contributes to our understanding, and is in all likelihood as far removed from everyday reality as possible. But somehow the work of theoretical physicists, at least as perceived by the cultural theorists, has broader support from funding agencies and university administrators. If one digs deeper, one may find that the group of theoretical physicists is respected by one group, which is respected by another group, which is respected by people who make bombs and missiles. The cultural theorists are also respected by a peer group, but there is perhaps no link to a wider group that is the equivalent of the makers of bombs and missiles.
Recognition by peers has served the academic research community well, and it remains an essential component of the research evaluation process. But we are also living in changing times. The increasing cost of healthcare, and other competing demands on the public purse, mean that research, especially research funded by taxpayers, must be relevant. Hence peer review, a necessary criterion of excellence, is no longer a sufficient criterion on its own.
The relevance of the peer review group must also enter into the equation. Research groups have a responsibility to make that link to wider impact — the equivalent of the makers of bombs and missiles. It is no longer sufficient in today's climate—at least until the baby boomers have made their graceful exit through the system, and as long as no other pressing needs demand the attention of the public purse—to say that one's research has impact purely because it is influencing the research of other researchers. That is, of course, unless these other researchers have a link to end users with wider influence.
The traditional reluctance to include end users in the peer review process does have the potential to make us—the academic research community—appear to the wider community as yet another self-indulgent interest group. But there is a more compelling need to involve end-users in the evaluation process. Society is increasingly looking to the research community to create economic opportunities and to help solve its problems.
Solutions to society's problems and ideas that create economic opportunities seldom come packaged in a single discipline. More frequently, they have interdisciplinary angles. Much innovation occurs at the intersection of disciplines, as the context of one discipline provides the right stimulus for the development of another. Good interdisciplinary research is inherently difficult, and can be made even more difficult if suitable incentives are not put in place.
As researchers, we tend to build our credibility in a single discipline, and our track record ensures that our papers get a fair hearing from our peers. When we venture into interdisciplinary research, we are suddenly being evaluated by reviewers who do not have a pre-existing credibility assigned to us, and some of our papers may have a more critical reception. If at this time our research output (and volume can sometimes be easily confused for quality) is being evaluated, there is always the temptation to claw back into the comfort of our old disciplinary burrows.
The peer review process is very effective at evaluating research in disciplinary silos, but less so when it comes to interdisciplinary work. End users, however, tend to be outcome-focussed, and are less likely to display the biases of any particular discipline. They tend to be more concerned with and effective in evaluating the relevance, if not the elegance, of interdisciplinary research.
The Research Quality Framework is a necessary step in reinforcing society's support for publicly funded research. It will be successful if it leads to a dynamic, publicly funded research sector, where the best institutions have to continue to work hard to maintain their status, and where there is also opportunity for aspiring institutions to emerge as leaders if they are willing and able to do so. The relegation system of sports leagues is often better than an ossified caste system; the latter may be stable and predictable, but it seldom leads to global competitiveness.
