{"id":63,"date":"2022-12-20T00:49:20","date_gmt":"2022-12-20T00:49:20","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2023team2\/?page_id=63"},"modified":"2022-12-21T02:08:23","modified_gmt":"2022-12-21T02:08:23","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2023team2\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n

Related work<\/strong><\/h2>\n\n\n\n

Debiasing using a small amout of fair data and a fairness classifier is a well establieshed approach to improve fairness in generative models [1]<\/sup>. Specifically, in the first step, a classifier is trained to distinguish if a sample is containing more or less bias. <\/p>\n\n\n\n

\"\"
fairness classifier trained with fair and biased data<\/figcaption><\/figure><\/div>\n\n\n\n

Second, the generative model is finetuned on a set of data, in which each data’s weight is assigned by the bias probability assigned by the classifier.<\/p>\n\n\n\n

\"\"
assign sample weight using fairness classifier<\/figcaption><\/figure><\/div>\n\n\n\n

While this method works well for the many GAN models, it is not very practical for the DDIM models since the diffusion process takes much more VRAM and the forward passing takes long. <\/p>\n\n\n\n

Shift latent distribution<\/strong><\/h2>\n\n\n\n

Another related method achieves the goal by learning and shifting GAN model’s latent sampling space [2]<\/sup>. It first collects sample latent codes generating images for each subclass. For instance, the subclasses for the gender attribute is ‘Male’ and ‘Female’. Then, for each set of code, a GMM model is learned. Finally, fair sampling is done by uniformally sample from each class’s GMM model. <\/p>\n\n\n\n

Although this method do not require finetuning the generative model, we found that it is very difficult to learn the GMMs for the DDIM model since its latent space’s dimsion is much higher. <\/p>\n\n\n\n

Debiasing single attribute<\/strong><\/h2>\n\n\n\n
\"\"
Figure 3<\/figcaption><\/figure><\/div>\n\n\n\n

We approach this task via learning fair text embedding. In the DDIM model, a text prompt is is first turned into a text embedding through the CLIP text encoder. For example in Figure 3, the prompt, ‘face photo of a person’ is mapped to text embedding E<\/em><\/strong>, which is a 77×768 shaped tensor.<\/p>\n\n\n\n

\"\"
Figure 4<\/figcaption><\/figure><\/div>\n\n\n\n

We want to learn a new text embedding, E’, that after the DDIM process, gives us a more fair set of images on the attribute of interest. Specifically, we want to push the E to E’ in the CLIP image and text joint embedding space (Figure 4), where it initially bias to ‘Male’ and then befome fair.<\/p>\n\n\n\n

<\/td>P(E’ | image batch)<\/strong><\/td>P(E | image batch)<\/strong><\/td><\/tr>
<\/td>1<\/td>0<\/td><\/tr>
<\/td>1<\/td>0<\/td><\/tr>
<\/td>1<\/td>0<\/td><\/tr>
<\/td>1<\/td>0<\/td><\/tr><\/tbody><\/table>
Table 4<\/figcaption><\/figure>\n\n\n\n

While learning E’, we first initialize it to the same as E, then we sample batch of images from a small fair data set. For each batch, we extract image features using the CLIP image encoder, and enforce E’s probability given the image features to be larger than E<\/em><\/strong> as shown in Table 4. Using binary cross entropy loss, E’ will eventually become a fair text embedding via gradient descent.<\/p>\n\n\n\n

Debiasing mutiple attributes <\/strong><\/h2>\n\n\n\n

The above method, learning a single fair text embedding E’, is great in 3 aspects:<\/p>\n\n\n\n