What science should look like in practice

The following is a five-part series on the core issues facing academic science today that will comprise the foundation of our first book, targeted for publication later this year.

“Nothing is more sad than the death of an illusion.”

– Arthur Koestler

Science is our understanding of the natural world. We review, refine, and grow this understanding using an approach we call the scientific method – a logic system comprised of posing a question, making an educated prediction at the answer, testing that prediction, and reconciling our conclusions to correct our prediction (or sometimes change the way we ask the question) to help arrive at a component of the answer. It works, and easily qualifies among society’s greatest accomplishment to-date. It may even be its greatest! The prediction is called a hypothesis*, and an experiment is the exercise in testing a prediction.

Scientists are practitioners of science, and while many of us receive formal training to be good and eventually great at it, formal training in science does not make one a scientist. A scientist is anyone that takes the step beyond simply posing a question of the natural world and tries to answer it. Not all scientists employ the scientific method, but they should, since it was designed to avoid subjective bias from affecting one’s conclusions. Running counter to the scientific method is beginning with a predetermined answer, then adapting or interpreting the results of an experiment to fit this answer (as opposed to refining the prediction). It is very hard to practice science well, since most of our predictions of how the natural world works are based on (often) very strongly held beliefs that are the bedrock of how we define the world and our place within it. If our understanding of the world changes, so must our narrative of our role within it – and that has historically not been something people are very good at doing. Because the scientific method is so hard to execute without subjective bias, many of us go to school for many years to learn how get better at it. The more capable we become at using the scientific method, the more impactful or nuanced are the questions we can strive to answer.

This is important because scientists are ultimately judged by the research they publish, and pressure to publish can often push scientists to publish splashy results at the expense of quality or meaningful data. Exciting or novel studies are often more publishable than confirmatory or negative results, and this bias can creep into decisions scientists make when considering their scientific design, or even into their interpretation of results (eg. choosing whether or not to randomize participants, include a control group, or account for possible confounding factors). Perverse incentive structures such as dependency on grant funding, publication records being valued by number (vs. quality) of publications and authorship position, and threshold of demonstration of statistical significance (see: “p-hacking”) can push scientists to cut corners in how they analyze their data and tempt scientists to bend and sometimes even break the rules.

This hurts everyone.

We can dissuade this behaviour, and in so doing improve the quality and output of our scientists by identifying and understanding the incentive structures that reinforce undesired outcomes.

*Use of term ‘theory’ was corrected to ‘hypothesis’