Trends in Microbiome and Digital One Health

Abstract:

Diffusion models have proven to be highly effective, achieving remarkable performance levels in various applications, as evidenced by their successful implementation in fields such as image synthesis, video generation, and molecule design. However, existing text-to-image diffusion models suffer from generating demographically fair results when given facial-related descriptions. These models often struggle to disentangle the target language context from sociocultural biases, resulting in biased image generation. To overcome this challenge, we propose Fair Mapping, a model-agnostic and lightweight approach that modifies a pre-trained text-to-image model by controlling the prompt to achieve fair face generation. One key advantage of our approach is its high efficiency. The training process only requires updating a small number of parameters in an additional linear stacking structure. This not only reduces the computational requirements but also accelerates the optimization process.  We show that our method significantly improves image generation performance when prompted with descriptions related to human faces. By effectively addressing the issue of bias, we produce more fair and diverse image outputs. Finally, we discuss some potential applications to healthcare and biomedical image analysis.