Fair Image Generation of Minority Groups

Give a vision model a simple job, decide whether a face is smiling, and it will happily cheat. On CelebA, women smile far more often than men in the data, so the cheapest way to be right most of the time is to glance at gender and guess from there. The model never learns smiling. It learns a correlation, and it fails on exactly the people who break it: smiling men and unsmiling women.

To study this on purpose, I took the smiling-vs-not task and injected a known bias by downsampling the two minority groups, smiling men and non-smiling women, to a tenth of their original size. Now I had ground truth: I knew precisely which groups the model should struggle with, and by how much. The question I set out to test:

Can a structural causal model, sitting in the latent space of a generative model, let me manufacture the missing people, smiling men and unsmiling women, well enough to undo the bias downstream?

The usual fixes are blunt. You can go gather more photos of smiling men, which is slow and expensive, or you can reweight the loss so rare groups count for more, which I tried as a baseline. I wanted something closer to the actual question a fair model should be able to answer: this same person, but smiling. That is a counterfactual, and counterfactuals are the natural language of causal models.

The catch is that you can only ask a counterfactual if the latent space is disentangled and causal: one knob for smiling, one for gender, and an honest account of how they push on everything else in the face.

I built on DEAR (weakly-supervised disentangled generative causal representation learning) and assembled three parts in a row. A ResNet18 encoder maps a face to a 100-dimensional latent code. A linear structural causal model sits on the first few dimensions, the ones tied to real attributes, and reshapes them so they respect a causal graph. A Self-Attention GAN generator turns the code back into a face.

The whole thing trains as a bidirectional GAN, so I can also run real images backwards into the latent space, with an extra supervised term that pins latent factors to actual labelled attributes. The objective is just the adversarial loss plus that supervised term, L = L_gen + lambda * L_sup.

The SCM encodes a causal graph over the attributes. In my setup, smiling points at cheekbone, mouth-open, chubbiness and narrow eyes, while gender points at narrow eyes. That structure is what separates two kinds of edit:

• A direct intervention flips the smile bit and nothing else. You get a mouth that smiles on a face that forgot to open it. The sample looks wrong.
• A causal intervention flips smile and propagates the change to its children through the graph, so a newly smiling face also opens its mouth and lifts its cheeks. The sample looks real.

The nice surprise: the graph weights it learned were sensible on their own. Smiling came out strongly linked to mouth-open, while gender-to-narrow-eyes settled near zero, which is roughly how faces actually work.

To make a smiling man: sample noise, push it through the SCM, set the smile and gender bits to the values I want, invert the SCM so the rest of the face stays coherent, and decode. I generated 5,000 faces each for the two minority groups, rebalanced the training set with them, and retrained the same ResNet18 classifier from scratch.

They are 64-pixel GAN faces, not portraits, but you can read the intervention clearly: the same identity rendered smiling and not, male and female, with the rest of the face moving along sensibly.

I measured delta-bias: the gap in accuracy between groups that share a label but differ in gender. Lower is better; zero means the model treats the groups identically.

The results split cleanly. Reweighting the loss, the popular quick fix, actually made the smiling gap worse, over-correcting until it hurt. Training on the causally generated faces cut the smiling delta-bias from about 25 down to 8, pulled the not-smiling gap from 25 to under 5, and lifted accuracy on smiling men from 73% to 84%.

This was my deepest dive into the seam between causal inference and deep generative modelling, and it changed how I evaluate models: pretty samples mean nothing if the latent factors are not doing what you claim. Reading DEAR closely and then bending it to a fairness setting was a different muscle from my from-scratch rebuilds, here the contribution was the experimental design and the causal framing, not the architecture.

The honest limits matter too. CelebA bias is a clean, almost toy setting, and real protected attributes are messier and rarely binary. Disentanglement is famously hard to measure, so I read these numbers as directional evidence under a controlled bias, not a deployable de-biasing recipe. The full derivations, metrics and figures are in the report below.