Skip navigation
The Black Hole

Final notch in the STIC: How should we measure knowledge development?


With the goal of assessing the state of science and technology in Canada, it seems quite reasonable that the STIC report has identified the development of new knowledge as a key metric for assessing Canada’s relative and absolute performance.  The difficulty with such a task though is understanding what is meant by knowledge development – should we focus on quantity, quality, impact, novelty or some other metric?  The STIC report attempts to break knowledge metrics into manageable chunks, but sadly stays far away from passing any sort of judgment on the relative importance.  The attributes they identify are:

  • number of publications
  • citations
  • relative impact
  • specialization
  • international co-publication

Number of publications
Perhaps the easiest to understand of all metrics is the number of publications produced by a country and as noted by the STIC report:

in 2008 Canada, with a share of only 0.5 percent of global population, accounted for 3.3 percent of scientific publications in the world. In absolute terms, this places (Canada) in 8th position after the United States, China, Japan, Germany, the United Kingdom, France and Italy.

Further notes, mostly derived from a far more comprehensive report from the University of Quebec include the dominance (82%) from the higher education sector and that nearly 70% of publications come from Ontario and Quebec.  Unfortunately, all that this tells us is that Canadian scientists, particularly in areas of the country with the most people ((Alberta, BC and Nova Scotia also punch well above their weight in terms of per capita publication output – they simply have fewer people)), write about their research more than others do.

This lack of quality control is where citations come in, though it’s a little unfair to attribute quality of a paper with the number of citations due to items to which we’ve alluded before.  Deficiencies aside, the good news is that Canada ranks 4th in the world in citations despite being 8th in publications, though I imagine much of this is due to countries like Japan and China relying on self-language citations rather than Canada which benefits from the English language dominated fields of scientific research ((If anybody knows where to find statistics on how many publications from Japan/China are in Japanese or Mandarin/Cantonese, it would be great to hear from you)).

Relative Impact
Here it seems that the STIC report makers were stretching for a bullet point as the relative impact index (again pinched from the UQAM report) is simply the ratio of the citations of a country to the number of publications – making the same point as above and subject to the same possible deficiencies.  What would have been really nice under this bullet point is some way of measuring the actual impact of research – how much of our research really advances the field?  Perhaps assessing the number of patents and their origin (or lack of origin) in Canadian research publications or documenting the number of healthcare decisions ((Questions like: Were particular drugs prescribed or not prescribed based on lab research? Did it save the public lots of money?)) taken based on studies performed by Canadian researchers?

One of the big reasons for different numbers of citations is the popularity of a field. Whereas biomedical science has good “value” with respect to citations per publication, fields such as astrophysics or comparative animal physiology are cited relatively little.  Something which impacts a country’s citation rate therefore is the type of research it embarks on, and one of the interesting things that the STIC report notes is Canada’s areas of specialization:

Canadian researchers account for 4.3 percent of world publications in applied biology and ecology, and 4.2 percent in astronomy, astrophysics and cosmology, as well as 3.9 percent in engineering science. Canadian researchers account for 4 percent of world publications in basic biology, but only for 2 percent and for 2.1 percent respectively in physics and chemistry.

With a small population, Canada almost certainly benefits from specialization in areas where it has established expertise.  This allows the universities/institutes to build a critical mass of researchers in particular areas that in turn creates the desire to do research in Canada with those world leaders.  My own field of stem cell biology is a great example where Canada has nurtured an enviable cadre of researchers from all across the globe by building on key discoveries ((Canadian researchers Jim Till and Ernest McCulloch were the first to formally prove the existence of a blood stem cell pool in mammals)) and carving out this area of biology as an important one for investment.  Through such programs as the Networks of Centres of Excellence and investments in physical infrastructure like the MaRS Discovery District, it appears that Canada has done well to identify and support some areas of research very well.  The last good summary on which areas Canada does and might in future benefit from was the Council of Canadian Academies first state of S&T report – an updated version of which should be coming out in 2012.

International co-publication
Working with international collaborators certainly benefits a country and being recognized as leaders in a field by having others working with Canadian research groups is another good indicator of success; however, I really fail to see the value in the “how many publications have a non-Canadian scientist on them” metric as this could as easily suggest that one’s national infrastructure is insufficient to get the job done and you are forced to look elsewhere to complete your research.  The STIC report implicitly presumes that it would be better for a Canadian cell biologist to get a genome sequenced with the Sanger Centre in the UK rather than at a similar facility in Toronto or Vancouver.  I worry even further when I see the countries that top the list of all star international collaborators: Switzerland, South Africa, Mexico, and Israel – it seems that scientists in the USA, Germany and the UK just go next door to get things done.

Overall, the STIC is quite correct to identify the need to assess “knowledge production” in both a qualitative and quantitative fashion, but therein lies the challenge that plagues the system – How do we best evaluate scientists (and their science)?   The shopping list of possibilities provided in this report merely scratches the surface and will require much more in depth and long term evaluation in order to assess the impact of Canadian research.  We’ll continue to keep our readers abreast of developments with articles like the Metrics for Evaluating Scientists ones from this year and last year.  If there are other ideas out there, please do drop us a line.

David Kent
Dr. David Kent is a principal investigator at the York Biomedical Research Institute at the University of York, York, UK. He trained at Western University and the University of British Columbia before spending 10 years at the University of Cambridge, UK where he ran his research group until 2019. His laboratory's research focuses on the fundamental biology of blood stem cells and how changes in their regulation lead to cancers. David has a long history of public engagement and outreach including the creation of The Black Hole in 2009.
Post a comment
University Affairs moderates all comments according to the following guidelines. If approved, comments generally appear within one business day. We may republish particularly insightful remarks in our print edition or elsewhere.

Your email address will not be published. Required fields are marked *

  1. Shannon Townsend / August 17, 2011 at 16:02

    Hi Dave. I read your article with interest. We work here in the area of performance measurement and evaluation for NRC and we’re always trying to push the limits on how we go about things. You may be aware that NRC is moving to a Research and Technology Organization (model). As a result, we’re likely oving away from clasical scientometrics towards exploring new measurement techniques. We’re exploring introducing a methodology whereby we measure technology maturity…based upon principles used under Technology Readiness Level (TRL). If you’d like to learn more, let us know. we think this might prove a novel way to examine how technologies are advancing, or maturing. Cheers.

    • Dave / August 19, 2011 at 09:19

      Hi Shannon,
      We would certainly be quite interested in hearing about this shift away from citations/publications as one of the recent focuses of the site has become “new metrics for evaluating scientists (and science) – please do send along any information you have on this and, with your permission, we would be happy to profile some of these new metrics on our site.
      David ([email protected])

  2. SubC / August 18, 2011 at 19:48

    Canada is a leader in science citations per capita as highlighted by this great little article in the Economist.
    We must be doing something (or, a lot) right to have earned this. Great article by the way !

    • Dave / August 19, 2011 at 08:05

      Hey SubC – thanks for this article, I hadn’t seen it and it certainly does look like Canada is doing well per capita. While the chart isn’t complete (i.e.: doesn’t expand the “other” box where better “per capita” countries may reside) I agree that 2004-08 looks pretty good for Canada amongst the heavy hitters.
      Completely unrelated to the post, I remember you mentioning you’d worked at UofA and UofC in the past – any chance you’re in the Edmonton area? I’ll be there in early Sept – would be cool to chat “science in Canada” with you. If keen, let me know at [email protected]

Click to fill out a quick survey