August 2015 – We are all fascinated with faces. We stare at faces from a very early time in our lives. We imitate facial movements already hours after birth. We surround ourselves with reproductions of faces on our desks, our phones, our computers, our walls. We are obsessed with selfies – well, at least many of us are. There are museums dedicated to portraits, and mostly these show faces and not feet or other body parts. Profile pictures show heads and not bellies. Passports and ID cards focus equally on that little ball sized thing that houses our brain, and features some important sensory organs. However, it is not just any face we like to freeze in time with our recording devices – we like faces that smile. So much so that some camera manufacturers are now installing smile detecting software that can delay the trigger until a smile has been shown! So what’s the deal with the smile then? This issue of the Emotion Researcher focuses largely on research on facial “expressions”. The fascination with emotional faces has played a major role in the development of emotion research at several historical junctures, as I proceed to show. In that sense too, facial expressions are special!
Charles Darwin and Facial Expressions
Clearly, Charles Darwin’s masterpiece, The Expression of the Emotions in Man and Animals (1872), covers much more ground than facial movements alone, but most people remember vividly the powerful facial illustrations contained in the book. Darwin presented a number of photos of faces and bodies taken earlier in the 19th century by Duchenne de Boulogne, as well as several drawings, also based on Duchenne’s work. In fact, the pictures in Darwin’s book were at the time a powerful new feature for a science book and they are likely one of the reasons why the book sold very well.
The book Darwin’s Camera by Philipp Prodger covers much info on the visuals in The Expressions and also includes unpublished photographic material. If you are interested in this angle, I do recommend the book, which also includes many interesting stories about Duchenne, about the fate of the photos etc. But the point I really want to make in this column is that The Expressions basically bootstrapped experimental emotion research and moved the scholarly interest in emotions from a largely philosophical context to that of a behavioral science.
The intangible feelings with which emotion science has occupied itself until then were replaced by movements that could not only be observed in the moment and verbally described, but that could, via photography, be frozen in time and studied at leisure. In this way, Darwin shifted the focus by bringing the internal world of emotions to the outside. On the negative side, Darwin’s focus on observable expressive behaviors paved the way for the behaviorist revolution that was to haunt psychology a few decades later.
Of course, there are many myths and rumors surrounding The Expressions among scholars in the 21st century. Some are disseminated by folks who have quite clearly never read the book, and only rely on secondary or tertiary sources. I highly recommend having a look at the original text, which continues to be seminal for our field. Here is an example among many of Darwin’s contemporary relevance. The voice is becoming a real draw for students of expressions, and it was brought into the debate on emotional expressions by none other than Darwin. Consider the following quote from page 93 of the 1872 edition of The Expressions: “As fear causes all the muscles of the body to tremble, the voice naturally becomes tremulous, and at the same time husky from the dryness of the mouth, owing to the salivary glands failing to act.” – a rather modern view integrating, expression, physiology, and acoustics!
Darwin also manifested a profound understanding of the complex and multi-modal nature of expressions. Dacher Keltner counted more than 50 emotional expressions discussed in the book, despite the widely held belief that Darwin had focused only on 6 or 7 emotions. The Expressions is mostly about expressive behavior writ large (and the its evolutionary functions and origin), but it also touches on the perception side in the last few pages. It even clearly states a version of the facial feedback hypothesis 100 years before James! This being said, without a pre-existing fascination with facial expressions in the scientific community, Darwin’s book might not have triggered the empirical study of emotions.
Paul Ekman and Facial Expressions
The combined left-right-punch of scientists’ failure to find clear evidence for Darwin’s prediction of universal emotional expressions in the first decades of the 20th century, and behaviorism’s assault on emotion as a mental construct had left emotion research pretty down few decades after the publication of The Expressions.
While “appraisal” was a new kid in town with Magda Arnold’s double volume Emotion and Personality in 1960, followed by Richard Lazarus’ work on cognitive appraisal, it was arguably Ekman’s work on the universality of facial expressions, inspired by Darwin and Tomkins, that created new interest in facial expressions, nonverbal behavior in general, and finally basic emotions.
Ekman’s work was catapulted to the forefront of emotion research by his classic 1969 Science paper “Pan-cultural elements in facial displays of emotions”, co-authored with Sorenson and Friesen. Even though his work would fit into a natural science department, Paul Ekman likes to remind us that his Human Interaction Lab was hosted in a psychiatry department at the Langley Porter Institute, as psychology departments at the time tended to be deeply suspicious of work on emotional expressions.
However, the combination of the new empirical data presented by Ekman and his colleagues, and the methodological innovations made possible by the Facial Action Coding System Ekman developed together with his colleague Wally Friesen, really provided motive and method for the renewed interest in facial activity and emotions. It is hard to find a member of the founding generation of ISRE scientists who was not directly or indirectly influenced by this research.
This is not to say that there was no important work on facial behavior and its perception before. For example, ISRE founding member Nico Frijda, who passed away recently, was very much interested in facial expressions, and had a line of research in the 1950s that involved even dynamic expressions – going further than Darwin’s illustrations before or Ekman and Friesen’s well known static sets decades later. But Frijda’s work on faces arguably did not result in as much public attention compared to the mass media appeal of Paul Ekman’s work.
Since the 1970s, there has been fierce debate regarding how many emotions there are, how they are linked to facial or bodily activity, and whether expressions are universal. As the fascinating discussion between Keltner and Cordaro, Fridlund and Russell in this issue testifies, the dust is far from having settled on these topics. Let me lay my cards on the table. My own work on facial expressions has convinced me that no theory can predict yet who will show what expressions to whom in which condition. For example, the results of the study by Hess, Banse, and myself 20 years ago suggested that neither Paul Ekman’s Neurocultural Theory, nor Fridlund’s Behavioral Ecology view can do so at present. But this debate is for another issue, for another conference, for another time. Instead, I want to focus on the third pivoting point in this brief history of how the study of expressions has impacted emotion research. This one is in the here and now.
Affective Computing and Facial Expressions
I am currently involved in a project funded by the European Union on developing empathic robots for use as tutors in a school setting with children (EMOTE). It is one of many projects in the last decade that use facial activity as a clue for an artificial system as to the emotional state of a human. In my view there lies much promise here, but also, I am afraid, much headache and worries.
I tend to describe the challenge, somewhat cartoonesquely, as follows. The engineers ask emotion specialists to describe how many emotions there are and what they look like. The emotion scientist might say: “Well, for example when people are happy they smile. A smile involves action units 12 and 6 according to Ekman and Friesen’s FACS.” “Great” the engineers say, “we can do that.” They go away and come back with some piece of software that can identify AU12 and maybe even the wrinkles around the eyes, as in AU6, and a label pops up that says HAPPY. The emotion scientist says – “ah well, I forgot one thing. Actually, you cannot be sure the person is really happy if they smile. See, they could just be polite … or embarred …or”. This is where the engineer generally stops listening. Engineers are allergic to the branching out of theoretical possibilities.
Of course, this is a greatly exaggerated scenario, full of stereotypes. I apologize to my engineer friends if they feel portrayed as simpletons. They are not. The point is that if you had to build an actual robot, you would also prefer straightforward answers to endless disquisitions that make implementation hard to achieve because of all the complexities they raise. The point I want to make here is that the rapidly increasing interest in socially intelligent information and communication technology brings emotion science again to the forefront, but it also raises the bar for it.
Suddenly, there is a new application for emotion science. An application where there is, potentially, a lot of money to be made and a lot of interest in the media to be found: Affective Computing. So, the challenge to the emotion scientists is, and again apologies for oversimplifying, to leave the ivory tower and learn to convey in simple terms what we actually know. What engineers ask demands straightforward responses. How do emotions look from the point of view of an observer? What causes them? What functions do they have? We should think of this is an opportunity to re-focus our discussions. Maybe we can get clearer on what exactly we still need to know. Maybe being asked how to implement a concept is a chance to operationalize it more precisely. This is also a way to rekindle the scholarly fascination with emotions and make our work relevant in the real world. But there are of course additional risks when our theories are not just discussed in the abstract but put to work in automata.
Concerning facial expressions specifically, I think that we have at this point pretty clear empirical evidence that there is no one-to-one mapping between any specific emotional state and any specific display. Neither posed nor spontaneous expressions are clearly and uniquely distinguishable. A study might show that a particular cue is more often observable in condition A than in condition B, but this is not sufficient to make absolute statements about what faces express independently of the context.
The real world is fuzzy. We need to better understand this fuzziness, and artificial systems need to come to terms with that. This is a communication issue that requires consistency, precision, and the will to take “implementers” seriously. The alternative is that we are no longer taken seriously. A development team might say: “well, right, interesting stuff you are guys are arguing about…” – and they turn away and build a robot based on Ekman’s Science paper from 1969. Why? Because it has the virtue of being simple, straightforward, and easy to implement mechanically.
The reality is that we have 50 more years of emotion science that shows just how complicated the encoding and decoding of nonverbal behavior actually is. We are starting to understand how dynamic interactions shape discourse and emotional communication. Much of this complexity CAN be implemented in artificial systems, but we must take the challenge of translating it in terms that an engineer can understand and benefit from very seriously. Our science will get better for it. And once more, it will be the fascination with faces and facial expressions that will have provided a major push to emotion science at large. And that makes me smile.