Are Horses Always Strong and Donkeys Dumb? Animal Bias in Vision Language Models

Vision Language Models (VLMs), such as CLIP, are widely used for various multimodal tasks and offer significant advancements in image-text understanding. However, existing studies have revealed that VLMs inherit biases from their training data which lead to the reinforcement of harmful stereotypes and cultural misrepresentations. In the proposed work, we analyze the presence of biases associated with animals in the CLIP model. We introduce a novel taxonomy, called Animal Bias Taxonomy (ABT), which categorizes stereotyped associations of animals in three categories. We also curated an animal dataset from existing datasets and applied data-cleaning process on it to remove unwanted images. Using ABT, we evaluated the outputs of VLMs on animal dataset when prompted with animal-related stereotyped terms to assess whether CLIP propagates biased associations that align with cultural stereotypes. Our findings reveal that CLIP frequently exhibits skewed cultural interpretations, such as associating owls with wisdom. Our study underscores the necessity of bias evaluation in VLMs and calls for greater transparency and culturally diverse data curation to ensure fair and inclusive AI systems. The code is available at https://github.com/MohammadAnas5/Clip-sAnimalStereotyping.