by JEFFREY M. ZACKS
People watching a 3D film at a cinema in Beijing. These days visual effects companies’ revenues largely depend on orders from films that have a high-budget for post-production. PHOTO/Ecns
Filmmakers have tapped laws of perception still unexplored by neuroscience to create a visual feast in the brain
About 25 minutes into the action film Iron Man 2 (2010), there is an explosive sequence in the middle of an auto race through the streets of Monaco. The scene is a technical tour de force, with explosions, cars flipping and fire everywhere, all in front of thousands of panicked race spectators. At a 2014 event at the Academy of Motion Picture Arts and Sciences, the film’s director Jon Favreau got to see the eye movements of audience members who watched the clip. He told us he was thrilled – and relieved – to see that everyone was watching the actors Robert Downey Jr and Mickey Rourke, particularly their faces and hands, and that nobody was looking at the crowd – because the crowd was all computer-generated, and if you look closely they don’t look all that real. As long as you don’t look closely, Favreau (who was also an executive producer) could go a little cheap on these effects and save the money for where it would really count.
This phenomenon – the audience’s eyes moving in unison – is characteristic of film viewing. It is not typical of real-world vision. Rather, filmmakers use editing, framing and other techniques to tightly control where we look. Over 125 years, the global filmmaking community has been engaged in an informal science of vision, conducting a large number of trial-and-error experiments on human perception. The results are not to be found in any neuroscience or psychology textbook, though you can find some in books on cinematography and film editing, and in academic papers analysing individual films. Other insights are there in the films themselves, waiting to be described. In recent years, professional scientists have started to mine this rich, informal database, and some of what we have learned is startling.
To understand how the eyes are affected by movies, you need to know a bit about how they work outside the theatre. When we are just living our lives, our eyes jump from one location to another two or three times per second, taking in some things and skipping over others. Those jumps are called saccades. (Our eyes also make smooth tracking movements, say when we are following a bird in the sky or a car on the road, but those are somewhat rare.) Why do we do this? Because our brains are trying to build a reasonably complete representation of what is happening using a camera – the eye – that has high resolution only in a narrow window. If any visual detail is important for our understanding of the scene, we need to point our eyes at it to encode it.
The way people use eye movements to explore a scene has a consistent rhythm that involves switching between a rapid exploratory mode and a slower information-extraction mode. Suppose you check into a resort, open a window, and look out on a gorgeous beach. First, your eyes will rapidly scan the scene, making large movements to fix on objects throughout the field of what you can see. Your brain is building up a representation of what is there in the scene – establishing the major axes of the environment, localising landmarks within that space, categorising the objects. Then, you will transition to a slower, more deliberate mode of seeing. In this mode, your eyes will linger on each object for longer, and your eye movements will be smaller and more deliberate. Now, your brain is filling in details about each object. Given enough time, this phase will peter out. At this point, you might turn to another window and start all over again, or engage in a completely different activity – writing a postcard or unpacking.
Aeon for more