News Nota Bene

The San Francisco Declaration on Research Assessment is gaining ground in Canada

By proposing an update to traditional criteria for evaluating applications, the declaration asks the research community to rethink its approach.

By

Maxime Bilodeau

April 21, 2021

Posted in

Articles
Man making Research Assessment
The San Francisco Declaration on Research Assessment, signed by the three major research councils and many Canadian universities, calls for the need “to improve methods of assessing [academic] research output,” but its biggest impact may be in promoting greater diversity, inclusion and equity in science.
Read this article in French 1 Comments

The journal impact factor is a commonly used metric in the research community, maybe excessively so. At least that’s what 2,185 organizations acknowledged when they endorsed the San Francisco Declaration on Research Assessment (DORA), which the American Society for Cell Biology developed at its annual meeting in 2012.

In doing so, these organizations, which include funding agencies, academic institutions and other scientific literature publishers, agree there is a pressing need “to improve methods of assessing [academic] research output.” In addition, they are committed to recommendations built upon the concept of not using “journal-based metrics as a surrogate measure of the quality of individual research articles.”

DORA has been endorsed by the research ecosystem in Canada and Quebec. The Government of Canada’s major research funding agencies signed the agreement in the fall of 2019, followed by the Fonds de recherche du Québec (FRQ) in June 2020. “We have been working on these important guidelines ever since. We’ve begun to discuss them, but they are not yet operational,” says Emmanuelle Lévesque, FRQ’s research ethics advisor.

A similar situation exists in academic institutions across the country. The only universities that signed the DORA did so in the last few months. One of them is the École de technologie supérieure (ÉTS), a member of the Université du Québec network. “Our goal is to reinstate the importance of often-neglected aspects of research, such as social impacts. We are currently revising our evaluation grids for hiring, tenure and allocation of research chairs,” says Ghyslain Gagnon, ÉTS’s dean of research.

The San Francisco Declaration on Research Assessment, signed by the three major research councils and many Canadian universities, calls for the need “to improve methods of assessing [academic] research output,” but its biggest impact may be in promoting greater diversity, inclusion and equity in science.

Setting the standard

However, DORA is devoid of any binding mechanisms. Therefore, it is possible to support DORA while maintaining an overemphasis on an index like the Hirsch index, which measures a scientist’s productivity by the number of citations of their publications. For example, the University of Liverpool, a signatory to DORA, was recently in the news when it threatened to fire 47 researchers for not being cited enough.

“DORA’s values are an ethical ideal; you can’t be against ethics! However, many signatories still follow practices that contravene the very principles of this declaration,” says Vincent Larivière, professor at the École de bibliothéconomie et des sciences de l’information de l’Université de Montréal and Canada Research Chair in the Transformations of Scholarly Communication.

For example, Elsevier, a leading global publisher of scientific literature, may endorse DORA and pledged to “significantly reduce the importance of the impact factor as a promotional tool,” but its publications continue to promote it openly. “In many cases, supporting DORA is more of a public relations exercise than a genuine change in practice,” he says.

Dr. Larivière wonders about DORA’s relevance in Canada and Quebec, where impact factors are not as institutionalized as in other countries. “Unlike many other countries, such as China and the Netherlands, we continue to attach great importance to peer review. Overall, our evaluation policies are more qualitative than quantitative,” he says.

Diversity, inclusion and equity

The true impact of DORA may lie elsewhere, in promoting greater diversity, inclusion and equity in science.

As the declaration requires an overhaul of conventional criteria for evaluating applications to various research bodies, it would mitigate the biases that traditionally favour particular groups of researchers. It would also enhance the overall quality of scientific production, as suggested by a study published in The Lancet in 2019 and co-authored by Dr. Larivière.

Ms. Lévesque of the FRQ concurs with this analysis. “Impact factors promote a very Euro-centric view of science. DORA has the advantage of transforming the lens through which we analyze research,” she says. The same applies to ÉTS, which launched a new engineering research chair program last fall to promote resilience and resolve in the face of adversity — this criterion alone accounts for 40 percent of the applicants’ evaluation record, compared to 30 percent for excellence.

“If you always use the same recipe, it’s obvious that you’re always going to get the same cake,” says Dr. Gagnon of ÉTS. “The same goes for research: emphasizing certain evaluation criteria over others recognized by academia will open doors to a more diverse faculty.”

The first internal chairholders for this new program were selected last December. The new external chairholders will be announced in the near future.

Maxime Bilodeau

Most popular

Featured Jobs

  1. Electrical Engineering - Assistant Professor (Electromagnetic/Photonic Devices and Systems)
    Toronto Metropolitan University
  2. Economics - Associate/Full Professor of Teaching
    The University of British Columbia
  3. Electrical and Computer Engineering - Assistant/Associate Professor
    Western University
  4. Indigenous Studies - Assistant Professor, 1-year term
    First Nations University of Canada
View all jobs

More from News

More from Nota Bene

More from Articles

This site is registered on wpml.org as a development site. Switch to a production site key to remove this banner.