Carina I. Hausladen, Manuel Knott, Colin F. Camerer, Pietro Perona

Social perception of faces in a vision-language model

by @kevinschawinski

Leon Yin et al. (2024)

• Input-output bias is measured by prompting the model
  (e.g., “show me a CEO”).

• Analysis focuses on the demographics of generated or retrieved image outputs.

Easily Accessible Text-to-Image Generation
Amplifies Demographic Stereotypes at Large Scale
Bianchi et al. (2023)

The problem with this approach is that the bias categories are
hard to generalise.

Are the generalizable ways in which people categorize each other?

Are the generalizable ways in which people categorize each other?

Can these fundamental dimensions of human social perception
be used to assess
social perception in
vision-language models?

Carina I. Hausladen, Manuel Knott, Colin F. Camerer, Pietro Perona

Social perception of faces in a vision-language model

We measure social perception

A photo of a

person

 A photo of a
person

Measuring social perception via

Cosine Similarity

age

CausalFace

CausalFace

female

age

CausalFace

female

age

Asian

Black

White

Legally protected

 A photo of a
person

 A photo of a

person

 A photo of a

person

friendly

• A photo of a <attribute> person.
• A <attribute> person.
• This is a <attribute> person.
• Cropped face photo of a <attribute> person.

Carina I. Hausladen, Manuel Knott, Colin F. Camerer, Pietro Perona

Social perception of faces in a vision-language model

FairFace

UTKFace

CausalFace

Do the statistical properties of CausalFace embeddings systematically differ from real-world photographs?

• CLIP is a state-of-the-art vision-language model that connects images and text.
• It is used for tasks like image recognition, captioning,
• and powering applications such as DALL·E.

Commonly used bias-metrics

Markedness 

• Markedness (Wolfe and Caliskan, 2022)
• WEAT (Caliskan et al., 2017)
• Skew@k, NDKL (Geyik et al., 2019)
• Mean cosine similarities

✓

✓

✓

✓

Commonly used bias-metrics

Protected attributes

female

age

Asian

Black

White

Non-protected attributes

smiling

lighting

pose

How do

protected and

non-protected

attributes affect social perception?

smiling

Bootstrapping differences

Bootstrapping differences

smiling

Bootstrapping differences

smiling

\(p<0.001\)

• Non-protected attributes cause as much social perception variation as protected ones.
• Considering a wide spectrum of protected and non-protected variables is necessary to understand and measure biases comprehensively.
• CausalFace is unique in providing the necessary attribute annotations.

Do age-related social perceptions vary across different social groups?

age

CausalFace

CausalFace

female

age

CausalFace

female

age

Asian

Black

White

Warmth

Competence

Communion

–

+

Agency

–

+

Agency

Agency

Agency

Agency

Distinct Clusters

• CausalFace representation keeps facial expression, lighting, and pose constant.
• FairFace and UTKFace lack this level of control.

–

+

Agency

+

Positive Agency

youngest

oldest

• In line with Chatman (2022), we also find that perceived Warmth drops for middle-aged White women .
• We observe increased Warmth for older men across all three racial groups.
  • Chatman (2022) find that men's perceived warmth increases from young adulthood to middle age, but not beyond.

Comparison to human subject research

How does
age-related
social perception
differ across datasets?

Uncontrolled attributes in FairFace and UTKFace make for noisy measurements and hide interesting phenomena.

age

smiling

Smiling

Smiling

a photo of a person

Smiling

a photo of a person

Smiling

a photo of a person

a photo of a person

Smiling

Smiling

Smiling

Smiling

Opposing valences are negatively correlated \( r_{smiling}=-0.21 \).

CLIP demonstrates human-like social perception

• ​ability to make broad associations, distinguishing race and gender
• exhibits fine-grained social judgments

How does the impact of facial expression on social perception vary across intersectional groups?

Warmth

most frowning

most smiling

most frowning

most smiling

Conservative Belief

Facial expressions influence social perception differently across groups.

Limitations

• Attribute manipulation effectiveness
  • ​Manipulations such as lighting or facial expressions might have differing levels of effectiveness across demographic groups.
  • Human annotators validated this, but such validation is, of course, never perfect.
• Potential residual confounds
  • Some color confounds might still be present despite controls for background, clothing, and hair color.
• Dataset vs. model bias
  • ​We only investigate one CLIP model.

Conclusion

• Facial expression impacts social perception more than age, while lighting affects it as much as age.  
• Ignoring unprotected attributes may lead to incorrect conclusions.

Conclusion

• Varying facial expressions of Black women elicit the most extreme changes in social perception in CLIP, a previously unobserved bias pattern.  
• Bias patterns in wild-collected datasets remain hidden due to noise from uncontrolled confounding variables.
• Our method enables the discovery of new phenomena.
• Independent control of variables ensures causal conclusions rather than correlations.  

Conclusion

carinah@ethz.ch

slides.com/carinah

github.com/carinahausladen

Caliskan et al. (2017)

asian                                     black

—

WEAT

protected and non-protected attributes

–

+

Bootstrapping Variations 

• We randomly choose two distinct values, \(x_1,x_2 \sim X\), for the chosen dimension (e.g., white and black).
• For each pair of values, we select the respective image embeddings, \(i_1(x=x_1), i_2(x=x_2)\) that are equal in all other dimensions (in this example: gender, age, smiling, lighting, and pose).
• We then compute the difference in cosine similarities between each image embedding and a text embedding \(t\), defined as \(\Delta(t, i_1, i_2) = \lvert \cos(i_1, t) - \cos(i_2, t) \rvert\).
• This process is repeated 1,000 times, generating a bootstrap distribution of \( \Delta \) values.
• This distribution describes the impact of the specific dimension on the cosine similarity of image embeddings and text embedding.
   

Heatmap of Pearson correlation coefficients of positive and negative valence dimensions of the ABC model.