Carina I. Hausladen, Manuel Knott, Colin F. Camerer, Pietro Perona

Social perception of faces in a vision-language model

by @kevinschawinski

Leon Yin et al. (2024)

• Several studies have documented social biases in AI.
• Input-output bias is measured by prompting the model (e.g., “show me a CEO”).
• Analysis focuses on the demographics of generated or retrieved image outputs.

Easily Accessible Text-to-Image Generation
Amplifies Demographic Stereotypes at Large Scale
Bianchi et al. (2023)

The problem with this approach is that the bias categories are
hard to generalise.

Are the generalizable ways in which people categorize each other?

Are the generalizable ways in which people categorize each other?

Can these fundamental dimensions of human social perception
be used to assess
social perception in
vision-language models?

We measure social perception

• CLIP is a state-of-the-art vision-language model that connects images and text.
• It is used for tasks like image recognition, captioning,
• and powering applications such as DALL·E.

A photo of a

person

 A photo of a
person

Measuring social perception via

Cosine Similarity

age

CausalFace

CausalFace

female

age

CausalFace

female

age

Asian

Black

White

Legally Protected

 A photo of a
person

 A photo of a

person

 A photo of a

person

friendly

• A photo of a <attribute> person.
• A <attribute> person.
• This is a <attribute> person.
• Cropped face photo of a <attribute> person.

FairFace

UTKFace

CausalFace

Do the statistical properties of CausalFace embeddings systematically differ from real-world photographs?

Commonly used bias-metrics

Markedness 

• Markedness (Wolfe and Caliskan, 2022)
• WEAT (Caliskan et al., 2017)
• Skew@k, NDKL (Geyik et al., 2019)
• Mean cosine similarities

✓

✓

✓

✓

Commonly used bias-metrics

Protected attributes

female

age

Asian

Black

White

Non-protected attributes

smiling

lighting

pose

How do

protected and

non-protected

attributes affect social perception?

smiling

Bootstrapping differences

Bootstrapping differences

smiling

Bootstrapping differences

smiling

\(p<0.001\)

• Non-protected attributes cause as much social perception variation as protected ones.
• Considering a wide spectrum of protected and non-protected variables is necessary to understand and measure biases comprehensively.
• CausalFace is unique in providing the necessary attribute annotations.

Do age-related social perceptions vary across different social groups?

age

CausalFace

CausalFace

female

age

CausalFace

female

age

Asian

Black

White

Warmth

Competence

Communion

–

+

Agency

–

+

Agency

Agency

Agency

Agency

• CausalFace representation keeps facial expression, lighting, and pose constant, leading to distinct clusters.
• FairFace and UTKFace lack this level of control.

–

+

Agency

+

Positive Agency

youngest

oldest

• Perceived Warmth declines with age for Black women, but not for White women, contrasting Chatman (2022).
• Unlike Chatman (2022) showing no change for aging White men, we observe increased Warmth for older men across all three races.

Comparison to human subject research

How does
age-related
social perception
differ across datasets?

Uncontrolled attributes in FairFace and UTKFace make for noisy measurements and hide interesting phenomena.

age

smiling

Smiling

Smiling

a photo of a person

Smiling

a photo of a person

Smiling

a photo of a person

a photo of a person

Smiling

Smiling

Smiling

Smiling

Opposing valences are negatively correlated \( r_{smiling}=-0.21 \).

CLIP demonstrates human-like social perception

• ​ability to make broad associations, distinguishing race and gender
• exhibits fine-grained social judgments

How does the impact of facial expression on social perception vary across intersectional groups?

Warmth

most frowning

most smiling

most frowning

most smiling

Conservative Belief

Facial expressions influence social perception differently across groups.

Limitations

• Attribute manipulation effectiveness
  • ​Manipulations such as lighting or facial expressions might have differing levels of effectiveness across demographic groups. Human annotators validated this, but such validation is, of course, never perfect.
• Potential residual confounds
  • Some color confounds might still be present despite controls for background, clothing, and hair color.
• Dataset vs. model bias
  • ​We only investigate one CLIP model.

• Facial expression impacts social perception more than age, while lighting affects it as much as age.  
• Ignoring unprotected attributes may lead to incorrect conclusions.

Conclusion

• Varying facial expressions of Black women elicit the most extreme changes in social perception in CLIP, a previously unobserved bias pattern.  
• Bias patterns in wild-collected datasets remain hidden due to noise from uncontrolled confounding variables.
• Our method enables the discovery of new phenomena.
• Independent control of variables ensures causal conclusions rather than correlations.  

Conclusion

Outlook

0

0

1

3

5

LLM voting: Human choices and AI collective decision making

JC Yang, [...], CI Hausladen, D Helbing,

7th AAAI Conference on AI, Ethics, and Society, 2025.

Identifying latent intentions
via Inverse Reinforcement Learning in
repeated Public Good Games

carinah@ethz.ch

slides.com/carinah

https://github.com/carinahausladen

Appendix

Smiling

Smiling

Caliskan et al. (2017)

asian                                     black

—

WEAT

protected and non-protected attributes

–

+

Bootstrapping Variations 

• We randomly choose two distinct values, \(x_1,x_2 \sim X\), for the chosen dimension (e.g., white and black).
• For each pair of values, we select the respective image embeddings, \(i_1(x=x_1), i_2(x=x_2)\) that are equal in all other dimensions (in this example: gender, age, smiling, lighting, and pose).
• We then compute the difference in cosine similarities between each image embedding and a text embedding \(t\), defined as \(\Delta(t, i_1, i_2) = \lvert \cos(i_1, t) - \cos(i_2, t) \rvert\).
• This process is repeated 1,000 times, generating a bootstrap distribution of \( \Delta \) values.
• This distribution describes the impact of the specific dimension on the cosine similarity of image embeddings and text embedding.
   

Heatmap of Pearson correlation coefficients of positive and negative valence dimensions of the ABC model.

Theoretical Models of Labor Market Discrimination

Statistical discrimination (Arrow, 1998)

Unfair treatment of ethnic minorities can result from rational actions executed by profit-maximizing actors who are confronted with the uncertainties accompanying selection decisions.

Taste-based discrimination (Becker, 2010)

Discriminatory behavior is the result of people’s unfavorable attitudes toward ethnic minorities.

How does Facial Expression impact Social Perception?

Gender

Females

Males

Race

Asian 

Black

White

a photo of a person

Smiling

a photo of a
liberal
person

Smiling

Belief (progressive)

How does CausalFace compare to wild-collected datasets w.r.t. gender and race?

Commonly used bias-metrics

Markedness 

• Markedness (Wolfe and Caliskan, 2022)
• adapted WEAT (Caliskan et al., 2017)
• Skew@k, NDKL (Geyik et al., 2019)

✓

✓

✓

Commonly used bias-metrics