I teach a gigantic first-year introductory course in cognitive science that traditionally has about 300 students in class, and another 1,000 or so online. My lectures are recorded, edited and released to the online students days later.
I have not seen much of a difference in how my in-class and online students have performed. In the fall of 2018, for example, the in-class students finished with an average final grade of 76.78 per cent while the online students averaged 74.21 per cent. That might be statistically significant, but to me it’s not that noteworthy since both are equivalent to a B letter grade. In the winter of 2019, the in-class students averaged 76.04 per cent and the online students 75.39 per cent. One possible explanation for the slight difference is self-selection. It could be that those who procrastinate end up in the online section, since registration for the in-class section fills up quickly.
When the COVID-19 pandemic hit, all students went online, so there was no more self-selection. The average grade fell from 69.21 per cent in the fall of 2020 to 68.5 per cent in the winter of 2021 – a drop of around six percentage points from pre-pandemic semesters! (I have heard that other classes experienced the opposite trend – students were doing better when they were all online. It could be that online learning is sometimes more effective, or that instructors are giving easier or fewer assignments or being more lenient toward students.)
When we went entirely online, I mostly used my previous videos, although I added one new lecture to replace an older lecture. So the pre-pandemic students in the winter of 2020 received the same course material, tests, and assignments as the pandemic students in fall 2020 and winter 2021. (There are four multiple-choice tests and one essay.) This provided a nice control.
I was interested in whether there would be a difference between releasing all of the lectures at once (“the Netflix model”) or releasing the videos week by week (“the Disney+ model”). Would it affect student grades? For the fall 2020 semester, I released all the video lectures on the first day; for winter 2021, I released them week by week.
I tried to keep as much about the two semesters the same as I could. The tests had to be taken during a specified time slot and were always conducted online, both before and during the pandemic. They were open-book and open-web, with three options per question. The multiple-choice questions are from a question bank that does not significantly change from semester to semester. I also typically have about eight teaching assistants to grade the essays. And while there was some overlap in TAs from fall to winter, there was some turnover. However, all TAs were instructed by me on how to grade, and we practiced grading together to increase inter-grader reliability.
My results: the average grade was less than one percentage point different – 69.21 per cent in fall 2020, versus 68.5 per cent in winter 2021.
It’s worth remembering that the winter students were in their second semester and the fall students were fresh out of high school. So you might think that the winter students would do better because they have more experience at university. But there was no difference in grades between fall and winter semesters in the past and, in this case, my pandemic winter student average was lower than my pandemic fall student average.In any event, with such a small difference between the fall and winter semesters, I concluded that releasing all lectures at the beginning or week-by-week has little to no effect. Since it’s less work to release them all at the beginning, I recommend the “Netflix” model. That’s what I’m doing now.
Of course, this analysis doesn’t apply to many courses. Some instructors do not have all their lectures pre-recorded, for example, or they may have smaller classes that incorporate discussion. But for those in a similar situation as myself, I hope this proves helpful.
Jim Davies is a professor at Carleton University’s Institute of Cognitive Science, and director of the Science of Imagination Laboratory.