How EchoDepth measures response to an ad

FACS and Action Units

The Facial Action Coding System (FACS) was published by Paul Ekman and Wallace Friesen in 1978 and revised in 2002. It describes every visible movement of the face as a numbered Action Unit (AU). AU12 is the lip corner puller that forms a smile. AU6 is the cheek raiser that narrows the eyes. AU4 is the brow lowerer that appears with concentration, confusion or displeasure.

FACS describes movements, not emotions. That matters, because it means the raw output of a test is observable and checkable: a reviewer can look at a moment and see the movement that was scored. Interpretation comes in the next step, and we publish how we do it.

Some combinations carry well-documented meaning. AU6 with AU12 is the Duchenne smile, which research associates with felt enjoyment more than AU12 alone, the polite or social smile. AU1 with AU2, the inner and outer brow raise, often marks surprise or a jump in attention.

From webcam to Action Units

Each viewer takes part from home in a standard browser. There is no app to install and no lab.

On every frame, the face is located and tracked, and each Action Unit is given an intensity between 0 (absent) and 1 (maximum). Scores are taken relative to the viewer's own baseline, so someone whose resting face has a slight frown is not counted as frowning throughout. Frame-level scores are then averaged into one value per Action Unit for each second of the ad.

The demonstration on our homepage uses an AI-generated viewer and an open-source face model (MediaPipe Face Landmarker) so that the Action Unit values shown are genuine measurements of that clip. It illustrates the output, not our production model.

From Action Units to emotion

We describe emotional response on three dimensions from Russell and Mehrabian's model of emotion, often called PAD or VAD:

• Valence, from −1 (negative) to +1 (positive).
• Arousal, from 0 (calm) to 1 (highly activated).
• Dominance, from −1 (overwhelmed) to +1 (in control).

Each dimension is a weighted sum of Action Unit intensities, clipped to its range. The table shows which Action Units push each dimension up or down. The full weight table is versioned and included in the appendix of every study report, so any result can be recalculated from its Action Units.

The Emotional Response Curve

The Emotional Response Curve shows, second by second, how viewers' faces responded to the ad. It plots valence, arousal and dominance, each averaged across all viewers, and a single response line that combines valence and arousal so that strong positive moments stand out from calm ones.

The response line

For each viewer i and each second t, valence V (−1 to +1) is weighted by arousal A (0 to 1), then averaged across viewers:

R(t) = mean_i [ V_i(t) × (0.5 + 0.5 × A_i(t)) ]

R runs from −1 to +1, where 0 is neutral. Positive moments that also raise arousal score higher than calm positive moments, so a laugh counts for more than a polite smile. Every report shows the curve with a 95% confidence band, worked out by resampling viewers, so you can see where the audience agrees and where it splits.

Dominance is reported alongside the response line but is not part of it. A drop in dominance often marks a busy, fast or confusing scene, which is useful to know whether the moment is positive or negative.

A worked example

The viewer on our homepage is smiling three seconds into the clip. The measured Action Units are AU12 0.94, AU6 0.29, AU4 0.08, AU1 0.01 and AU2 0.04. The homepage demonstration uses a simplified five-unit version of the mapping:

V = 0.7 × AU12 + 0.5 × AU6 − 0.6 × AU4 = 0.76
A = 0.5 × max(AU1, AU2) + 0.4 × AU12 + 0.3 × AU6 = 0.48

So for this single viewer at 0:03, R = 0.76 × 0.74 = 0.56. A real study averages it across every viewer in the study, 15 or more per audience segment.

What the curve is not

The curve describes response, not results. We have not yet published a validation against in-market sales or brand tracking. Until we do, read it as a diagnostic that tells you where an ad works and where it doesn't, not as a forecast of sales.

The Cavefish Emotional Index (CEI)

The Cavefish Emotional Index (CEI) is a 0 to 100 score of how people feel, read directly from what they write or say. It is the same index used across every EchoDepth module. The CEI is the average of five readings, each scored from 0 to 100, where higher is healthier: Advocacy, whether they have anything good to pass on; Goodwill, how much hostility is in what they said; Engagement, whether they are still invested or have resigned themselves to it; Drive, whether they are still trying to fix it themselves; and Assurance, whether they know where they stand. Scores fall into four bands: Strong (80 and above), Good (60 to 79), Watch (40 to 59) and Act (below 40). In an ad test, the CEI is read from viewers' answers after they watch, so it captures what people say about the ad, separately from what their faces did.

How it is read in an ad test

After the ad, viewers answer open questions about it, in writing or out loud. EchoDepth scores what they said on the five readings and averages them into the CEI for the ad, and for any audience group you report on. The definition and bands are the same as in every other EchoDepth module; the canonical definition is published on cavefish.ai.

Stated vs felt

The CEI tells you what viewers said about the ad. The Emotional Response Curve tells you what their faces did while they watched. Setting one against the other shows where an ad is liked but forgettable, or remembered but resented.

Sample and quality

How many viewers

We recommend 15 or more viewers per audience segment for each ad or edit. Every group you want to report on separately, such as under-35s and over-35s, is its own segment and needs its own 15. When you compare edits, each viewer usually sees only one (a monadic design), so one edit can't colour the response to the next. If viewers see several, the order is rotated.

Who the viewers are

Viewers come from the EchoDepth tester panel, matched to your audience on age, gender, region, job and interests. Every tester has opted in to webcam studies in advance. For audiences the panel can't yet reach, we recruit through established partner panels under the same consent and quality rules. We screen out people who work in advertising, marketing, PR or market research, or who live with someone who does.

The day-after reading

Many studies add a short follow-up the next day: two questions on how viewers feel about the ad and the brand now, and what they remember. Set against the in-the-moment curve, it shows which reactions lasted and which faded.

Quality checks

• A setup check before the ad plays confirms the face is visible and lit. Viewers who fail the camera check don't continue.
• Duplicate sign-ups and repeat attempts at the same study are detected and removed.
• Viewers whose face is tracked for less than 80% of the ad are excluded from the analysis.
• An attention question after viewing removes people who weren't watching.
• Each report states the number of viewers who started, the number excluded and why.

What this can't tell you

Facial coding is a useful measure with real limits. We would rather state them than have you find them later.

• A face is not a feeling. Expressions and inner states are linked, but not one to one. A 2019 review led by Lisa Feldman Barrett found that the same expression can accompany different emotions depending on the person and context. That is why we report across many viewers and pair the curve with a survey.
• People watching alone are not very expressive. Most viewers show little visible expression for most of an ad. The signal is in the moments that do move faces, which is why results are aggregated across many viewers.
• Some Action Units are harder to detect than others. Smiles (AU12) and cheek raises (AU6) are detected reliably. Subtle movements such as AU14 and AU17 are harder, so we weight them cautiously.
• Conditions vary. Lighting, glasses, beards, head angle and webcam quality all affect tracking. Webcams at 15 to 30 frames a second miss very brief micro-expressions.
• Culture shapes display. Norms for showing emotion differ between cultures, so compare results within a market rather than across markets.

Independent research supports the core approach. Höfling and Alpers (2023) found that automatic facial coding of people watching video commercials predicted their own ratings of the ads and the brands. Earlier large-scale work by McDuff and colleagues linked facial responses to ad liking and purchase intent. In 2022, the UK Information Commissioner's Office warned that emotion analysis technologies can be immature. We take that seriously, which is why we publish our method.

Data, consent and the AI Act

What we keep

The webcam feed is processed as it streams and is not written to disk. We keep Action Unit intensities, valence, arousal and dominance scores and survey answers, linked to a panel ID rather than a name. We do not create face templates and we do not identify people from their faces. We keep readings, not video. How long they are kept is set out in each study's consent notice, and testers can ask for their readings to be deleted at any time.

Consent

Each viewer gives specific, informed consent to each study before the webcam turns on, and can stop at any time. Panel members can leave the panel and ask for their data to be deleted.

The EU AI Act

The EU Artificial Intelligence Act (Regulation (EU) 2024/1689) treats systems that infer emotion from facial data as emotion recognition systems. Article 5(1)(f) bans them in workplaces and education. Annex III lists emotion recognition among high-risk uses, and Article 50(3) requires people exposed to it to be told. EchoDepth only codes the faces of consenting research viewers who have been told exactly what is measured. We do not offer facial coding for employees or students. Our workplace products analyse written language only.

This is a summary of our approach, not legal advice. Clients with specific regulatory needs can ask for our data protection documentation.

References

1. Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System. Consulting Psychologists Press.
2. Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial Action Coding System: The Manual. Research Nexus.
3. Russell, J. A., & Mehrabian, A. (1977). Evidence for a three-factor theory of emotions. Journal of Research in Personality, 11(3), 273–294.
4. Ekman, P., Davidson, R. J., & Friesen, W. V. (1990). The Duchenne smile: Emotional expression and brain physiology II. Journal of Personality and Social Psychology, 58(2), 342–353.
5. Teixeira, T., Wedel, M., & Pieters, R. (2012). Emotion-induced engagement in internet video advertisements. Journal of Marketing Research, 49(2), 144–159.
6. McDuff, D., el Kaliouby, R., Cohn, J. F., & Picard, R. W. (2015). Predicting ad liking and purchase intent: Large-scale analysis of facial responses to ads. IEEE Transactions on Affective Computing, 6(3), 223–235.
7. Höfling, T. T. A., & Alpers, G. W. (2023). Automatic facial coding predicts self-report of emotion, advertisement and brand effects elicited by video commercials. Frontiers in Neuroscience, 17, 1125983.
8. Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered. Psychological Science in the Public Interest, 20(1), 1–68.
9. Information Commissioner's Office (2022, 26 October). Immature biometric technologies could be discriminating against people, says ICO in warning to organisations.
10. Regulation (EU) 2024/1689 of the European Parliament and of the Council (Artificial Intelligence Act), Articles 3(39), 5(1)(f) and 50(3), and Annex III.
11. Lugaresi, C., et al. (2019). MediaPipe: A framework for building perception pipelines. arXiv:1906.08172.