Smart Ideas: Q&A
Miao Song and computational media

This series, sponsored by the Federation for the Humanities and Social Sciences, features notable humanities and social sciences researchers with smart ideas for a better tomorrow. This month, we speak with Miao Song, an affiliate assistant professor in Concordia University’s faculty of engineering and computer science.

Miao Song has carved out what can only be described as a unique niche for herself in the world of live performance. Originally from China, where she was an award-winning stage artist and later a journalist on national television, she came to Canada to broaden her experience in how progress in new media and computer graphics technology is shaping the nature of film, dance and theatre presentation. Although she is currently an affiliate assistant professor in Concordia University’s faculty of engineering and computer science, her work continues to draw from her roots in the arts, as she brings together engineers and computer programmers with dancers and actors to establish a rapidly growing new field of technologically mediated performance that cuts across disciplines.

You were well established in China. Why come to Canada?

Dr. Song: A lot of people who work in the arts in China never leave, often because they don’t have the language skills to work somewhere else. I could easily see the rest of my life unfolding there, but I was eager to see what was happening in the rest of the world, so I started studying English so I could pass the proficiency exams and go abroad. Maybe I just like a challenge.

What did you do once you arrived here?

Dr. Song: I started my academic career over again. I was interested in computer animation and found some important mentors at Concordia University who introduced me to the emerging area of computational media. That included Peter Grogono in the Department of Computer Science and Engineering, and Jason Lewis, who holds the University Research Chair in Computational Media and the Indigenous Future Imaginary. They helped me create the Illimitable Space System, which became the subject of my PhD thesis in 2012. This technique uses a Kinect camera from an Xbox game console to collect data about how actors or dancers are moving on a stage. This information is then processed by a computer to create images that can be presented while the performance is happening.

Has it been challenging to bring engineers and computer programmers together with people who make their living in the arts?

Dr. Song: I am helping researchers from very different fields come together to develop an entirely new field. There can be misunderstanding between these groups because they are unfamiliar with each other’s backgrounds. Those who develop technology for the theatre do not always have a feeling for the whole process of what happens on a stage. As an actor, I know that I can do things that will bring an audience to tears, but experts in computer science usually have no idea about how that happens. And most artists know little about programming languages and how they are structured; they also have little idea how the machinery works and what it could do for them if properly used. Many of them even reject this new technology and want none of these interactive elements, so that only their behaviour affects the audience. All of these perspectives reflect individual experience and that’s just the way human beings are. However, I am trying to change that.

What have you done to bring these two groups together?

Dr. Song: The technology we are developing makes it easier for directors and performers to express themselves, without having to accommodate the equipment on stage. When we were starting out, a lot of them thought it was just funny. The first time we were on stage at Concordia’s D.B. Clarke Theatre, we had the Kinect on the floor in a place where the dancers accidentally kicked it. And the camera can only capture a certain range of angles, so we had problems like an image projected on the screen of the dancer with no legs! The technology has constraints, but it is also progressing. A lot of people expect the graphics to look fake, but once they see it, they realize that these projections do not just capture the movements of the performers, they are a separate performance that is happening at the same time. In my thesis, I described how this technology was changing the attitude toward what is called “poor theatre,” where the stage has almost no props and actors wear only plain outfits. The multimedia technology I call the Illimitable Space System looks very simple and takes up very little room, but it has the potential to turn “poor theatre” into “rich theatre,” adding extra dimensions that have never been incorporated into traditional theatre forms. I believe that this technology will become a very big part of these art forms.

Where have you applied this technology?

Dr. Song: I initially used it to enhance a film documentary I had made, so that viewers in a theatre could create virtual reality images like those you would see on an interactive website. It subsequently appeared at a Chinese New Year gala hosted by the Montreal Centre of Chinese Culture and Arts. I later incorporated the software from the Illimitable Space System into several scenes of a play called “Like Shadows,” which was a collaboration between Concordia University and the Central Academy of Drama in Beijing. The play was performed for the public in 2014, and it might have been the first time ever that real-time interactive media was used for a traditional theatre production in China. Each of these performances has provided experience and data that have given us the confidence to keep working on this system. It also gave me the confidence to become founder of the Canada China International Film Festival, which began just this past September. The festival included an Entertainment Technology Summit and Exhibition where people from industry and academia could meet to exchange ideas and technology, and form partnerships. This is where the festival brought artists and engineers together and where we exhibited our own Illimitable Space System work, as well as the work of other researchers from Concordia University, McGill University, the National Film Board, Disney, GoogleVR, Cirque du Soleil, the Chinese Academy of Sciences, and others.

Just as you brought artists and engineers together, do you also see yourself as a mediator between Canadian and Chinese culture, helping audiences in each country understand one another better?

Dr. Song: At first I didn’t think this aspect of the work was all that important, but as these activities have grown I am starting to think that this role could be very significant. I still can’t believe that I have a team of 60 people who work with me on the film festival, and that leads me to believe that we can bring even more people to take part in this cultural exchange between Canada and China. We have also been able to build partnerships with universities in the U.S. that are exploring this technology in their own exciting ways.

As more and more people become enthusiastic about these new modes of presentation, do you ever worry that the original beauty of traditional art forms might be compromised or lost?

Dr. Song: Traditional art is beautiful; it’s eternal. But I do believe that this new technology can give you a lot more possibilities for expressing yourself through art. From the bottom of my heart, I believe there is no conflict. This is a way of bringing traditional art into the wider entertainment industry, so more people can experience it. That includes the artists themselves, who may only think about showing audiences how they see the world; this interactive technology makes artists learn how audiences see the world.