Skip navigation
The Black Hole

Lab challenge! Same protocol, different hands

We need to encourage the writing of more specific protocols and lobby researchers to publish them on open websites.


In May of this year, a fairly big paper emerged in one of our major research areas (trying to expand blood stem cells outside the body). We know the research group behind it and had a bit of a head start trying to transfer the protocol to our lab, but it turned out to not be as straightforward as we first hoped. The first few attempts showed some definite promise, but it didn’t consistently match the reported success of the original study. In the cases where it did work, it was the best result in blood stem cell expansion that I’d ever seen. This made it obvious that as a lab group we needed to find out where things were (sometimes) going wrong. Enter the lab challenge.

As some readers may know, I’m a big fan of putting my money where my mouth is, so I called the researchers in my lab together and offered a small financial carrot to see if any individual could consistently produce similar results to the published findings. It was a 28-day protocol and involved regular maintenance so we set the rules of the challenge: everyone got the protocol to read and was supplied with equal starting pots of blood stem cells and the same reagents to work with. Then they were left to their own devices to “interpret the protocol.”

This word “interpret” might come as a surprise to our non-science readers – it could easily be assumed that a scientific protocol is rigid, with little to no room for “interpretation.” However, the reality in very large and long protocols is there are a huge number of areas where scientists use non-exact language and this can make a huge amount of difference depending on the reader. I’ve listed some examples below that illustrate some of these less than specific items that biologists need to deal with on a daily basis. Terms like “gently flick”, “gradually remove”, “gently dribbling” and “aggressive mixing” all have room for interpretation. Combined with changing the reagents “every two to three days” in a procedure called a “complete” medium change with a further proviso to remove “ approximately 95 percent” of the medium leaves much to be interpreted by the individual reader. Whereas many protocols have sufficient wiggle room in them (i.e. they’ll work even if you miss the mark a little), it turns out that this particular one is quite finicky, leading to significantly different results (in our lab’s case this ranged from no stem cells retained to seeing incredible expansion from even single blood stem cells).

Some of the implications of this are quite far-reaching. For example, a number of articles/studies have pointed to a reproducibility crisis in biomedical science casting doubt on the quality of biomedical research in academic institutions. Depending on how you view the “experiment” we just performed, you could make this claim, although I would rather suggest that it is dependent on who is carrying out the protocol, or more accurately, how the protocol was executed.

So where do we go from here? Well the first obvious thing is to encourage more precise writing of protocols. Secondly, because research papers are almost always tight for space, we need to encourage the writing of more specific protocols chapters and lobby researchers to publish their protocols on open websites. Finally, (and it’s surprising this isn’t done more) researchers should be encouraged to create small videos for the particularly tricky aspects of a protocol – post them, share them, and amend practice based on feedback (there are also video journals now!). Sometimes it really feels like we are stuck in the dark ages when it comes to sharing best practice in science when the reality is that we should be doing everything possible to make sure that experiments across the world are done in as reproducible and rigorous a manner as possible.

David Kent
Dr. David Kent is a principal investigator at the York Biomedical Research Institute at the University of York, York, UK. He trained at Western University and the University of British Columbia before spending 10 years at the University of Cambridge, UK where he ran his research group until 2019. His laboratory's research focuses on the fundamental biology of blood stem cells and how changes in their regulation lead to cancers. David has a long history of public engagement and outreach including the creation of The Black Hole in 2009.
Post a comment
University Affairs moderates all comments according to the following guidelines. If approved, comments generally appear within one business day. We may republish particularly insightful remarks in our print edition or elsewhere.

Your email address will not be published. Required fields are marked *

  1. Corinne Hoesli / September 11, 2019 at 20:15

    I really love this idea. I do like the video journals but perhaps there is a better open platform (anyone have any suggestions?) without fees and preofessional editors which could be usd to share standard operating procedures and videos – especially now that most students have a high-quality camera in their pocket…

    We’d still need volunteer editors of these SOPs – similar to Wikipedia…

    Then, there is the issue of researchers “hanging on” to their internal SOPs and know-how to keep competitive advantage. Unless the platform were to become recognized (e.g. when applying for grants), many would be reluctant to give up their “trade secrets”.

Click to fill out a quick survey