Skip navigation

Large, linked data sets are transforming historical research

New book describes how historians are using longitudinal analysis to shed light on our collective past.


In the last few decades, societal evolution has been thrown into a new light as hundreds of thousands of old records – from censuses to hand-written church records – have been digitized and aggregated. These massive databases allow researchers to track long-term change in populations through generations all over the world, revealing patterns of birth, employment, marriage, parenthood, employment, even war and crime.

In turn, this has recast how some researchers in the field of history envision data. They have moved from looking at information as cross-
sectional – data points like birds perching on a wire – to the longitudinal – a flock flying wing-tip to wing-tip, undulating like a wave through time. Economists, health researchers and some social scientists have used longitudinal analysis for some time, but researchers in humanities disciplines including history are more recent adopters of this method.

Some of the recent findings emerging from longitudinal historical research in Europe, North America and Australia are described in a new book, Lives in Transition: Longitudinal Analysis from Historical Sources (McGill-Queen’s University Press, 2015), edited by Peter Baskerville, who holds a research chair in modern Western Canadian history and leads the Transforming Historical Data Collaboratory at the University of Alberta, and Kris Inwood, co-director of the Historical Data Research Centre at the University of Guelph. The book was the centrepiece of a seminar at this year’s Congress of the Humanities and Social Sciences at the University of Ottawa in June.

The book collects 12 papers in this emerging field on topics ranging from Scottish migration to how the influenza pandemic of 1918 was spread by mobilizing armies. Several are co-authored by Canadian authors, including a paper investigating which Canadian Aboriginal groups fought in the First World War and another positing that during that same war some early cases of the 1918 influenza pandemic were brought to Canada by Polish soldiers who had been recruited in the U.S. and sent to Niagara-on-the-Lake for training.

In their introduction, the editors write that it is the first book “that provides a discussion of the possibilities for new knowledge that can be realized through linking individual-level data for large numbers of people between two or more points in time and also demonstrates through actual research how such a procedure can lead to a new and enhanced understandings of important aspects of our collective past.”

But they and other Canadian historians doing longitudinal research worry that their efforts are not well understood in their own discipline, and that this disinterest is hurting their efforts to win grants from research funding agencies like the Social Sciences and Humanities Research Council. Eric Sager, history professor at the University of Victoria, asserts that Canada was once a leader in this field but now lags so far behind other countries it may never catch up. This became a lively topic of discussion at the Congress seminar.

Of course, researchers in every field are keenly disappointed when they don’t get funding. But Herbert Emery, an economic historian at the University of Calgary, says this issue is one that the larger academy, and the general public, should care about. If we are unable to analyze data that reaches back generations, he says, then Canada will lose crucial information to answer today’s big questions. For example, if we hear that young people with university degrees are not advancing in their jobs as they did in the past, is that true, and if so, why? Is it the economy, the digital age, or the fact that so many more people now have degrees than in their parents’ era? “Those kind of questions [are ones] we can’t answer without the historical perspective,” says Dr. Emery.

Chad Gaffield, history professor at the University of Ottawa, is in a unique position to place the issue in a larger context. As president of the Social Sciences and Humanities Research Council for eight years until August 2014, he is also principal investigator for the Canadian Century Research Infrastructure (CCRI) project. Funded by the Canada Foundation for Innovation and many other partners, this $15-million project spanned several institutions and disciplines to develop a set of interrelated databases from the five Canadian censuses between 1911 and 1951. These, in turn, are being linked to databases from 1871 to 1901, and eventually will be connected with those from 1971 to 2001. (Unfortunately, future projects may not include the 1961 census data, which cannot be recovered from the computer tapes used at the time.)

The result of all this linked data is a new foundation for the study of social, economic, cultural and political change – an infrastructure not of roads or particle accelerators but of data. Researchers from many fields are now analyzing these data to study Canada’s transformations as it moved from being a marginal, small society to one of the most successful societies in the world.

Understanding long-term change is fundamental to our future, says Dr. Gaffield. “We have made a huge investment in understanding colliding particles. We need to take seriously the challenge of human thought and behavior – colliding people.”

Dr. Gaffield says that scholars using new techniques like longitudinal data analysis should learn to think big and to truly cooperate. “The education system individualizes us,” he says. “It never teaches us to collaborate, or especially to work across those using different vocabularies, and different ways of thinking.” He would like to see more coaching, support and graduate programs to help people come forward with a larger number of truly compelling proposals.

Dr. Emery suggests a return to sweat equity. Before data came in such torrents, professors and their graduate students did the work of entering it all into a computer. Once that became unwieldy, “we started to demand that someone else do the work. And once we started doing that … we were no longer making our own personal investments in it. I’ll be criticized for suggesting people should have to do this but I think that’s one solution to get it moving.”

Dr. Gaffield remains an evangelist for this field. “For the past two centuries, we have believed that science and technology would solve all our problems with silver-bullet solutions: just the right widget, or labor saving device, or a pill that would solve all your health problems. As appealing as these promises were, they just weren’t true.

“Now I think we recognize that building a better world has to start with culture, has to start with decisions people make, how we think, how we imagine ourselves in the world, and the only way we’re going to get there is how we do justice to the complexity of human beings.”

Post a comment
University Affairs moderates all comments according to the following guidelines. If approved, comments generally appear within one business day. We may republish particularly insightful remarks in our print edition or elsewhere.

Your email address will not be published. Required fields are marked *