Taft logo
Taft logo
Taft logo
Summer Journeys: Esther Lee ’28, Expression Intelligence

Esther Lee ’28 studied facial expressions through a project-based, directed research program at the University of California, Santa Barbara.

For four weeks last summer, Esther Lee ’28 studied faces—formally, and with intention. Esther participated in a project-based, directed research program at the University of California, Santa Barbara entitled Expression Intelligence: Leveraging Computational Methodologies to Decode Human Facial Display.

PyFeat facial feature data

An image sample of the way Py-Feat organizes facial feature data

“I’ve been interested in cultural differences in nonverbal communication and areas such as medicine or human society that intersect with it,” Esther says. “We studied facial muscle movements, called ‘action units,’ and how software like Py-Feat and OpenFace can detect them automatically from a video or image. I liked that the program combined communication, psychology, and computer science, and was built on active learning.”

The human face contains more than 40 muscles. Muscle movement—even subtle muscular shifts—creates changing expressions that can convey emotion, well-being, intent, and an array of reflective reactions, making the face is a powerful communicator that can often transcend cultures and contexts. Through a combination of behavioral science, cultural studies, and computational methods, Esther studied the face as a multi-signal, multi-message system, that informs perception, interaction, and social outcomes in a range of social arenas, from personal to professional.

“Every day we had lectures on the science of the face, such as how expressions evolved, how researchers code them with the Facial Action Coding System (FACS), and how automated tools work and where they fail,” Esther explains. “It truly surprised me how many fields this research reaches: health, biometrics, biomechanics, psychology, communication, and even veterinary medicine, where researchers have adapted FACS into systems like DogFACS and HorseFACS to help tell how animals are feeling.”

Esther and her research group presenting their capstone project. Team member images pixelated for privacy.

Esther’s studies culminated in a group capstone project, designed to synthesize their learning into a fully developed and executed research project. 

“With my team, I studied how emotion shows up in food advertising. We used Py-Feat to analyze facial expressions frame by frame in YouTube ads from 10 commercial brands, including Coca-Cola and Chipotle, and 10 non-profits, including Feeding America and World Central Kitchen,” says Esther. “We expected commercial ads to lean positive and non-profit ads to lean negative. Instead, we had very interesting results where both had very similar emotional valence profiles, mostly neutral and positive. We wrote up our findings as a research paper and presented them at a capstone seminar.”

For Esther, the breadth of the learning experience was wide, and the impact deep and lasting. 

“It was fascinating to see how much scientists still debate about the face. Paul Ekman had argued that six basic emotions show up the same way across cultures, and Alan Fridlund argued that expressions are social signals rather than readouts of what we feel,” notes Esther. “I learned that two researchers can look at the same evidence and both be partly right, depending on how they measure. 

“Some of the research showed how early faces matter. In the Still Face experiment, babies became visibly upset when their mother's face stopped moving. In another study, 4D ultrasounds showed fetuses reacting differently to carrot and kale.

“It was also just a great opportunity to get involved in producing original research work as a novice and get up-to-date lectures on that specific field. Working with passionate researchers and mentors was a great feeling and made me feel like I was contributing to the scientific community and the bigger academia.”

_______________________

Esther’s study was made possible in part by a Meg Page ’74 Fellowship grant.

To honor Meg’s commitment to compassionate healthcare, Meg Page ’74  fellowship grants are awarded annually to students who wish to explore an experience or course of study devoted to the provision of better health care in areas such as public health, family planning, medical research, mental health, and non-Western practices of healing.

There are no news posts to display