{"id":133,"date":"2023-04-30T04:15:31","date_gmt":"2023-04-30T04:15:31","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/?page_id=133"},"modified":"2023-04-30T05:08:28","modified_gmt":"2023-04-30T05:08:28","slug":"data_uncertainty","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/data_uncertainty\/","title":{"rendered":"Data Uncertainty"},"content":{"rendered":"\n

Title<\/strong><\/p>\n\n\n\n

Visual Features Disentanglement into Interpretable Attributes Features using Prompts<\/p>\n\n\n\n

Research Question: <\/strong><\/p>\n\n\n\n

How do we describe the distribution of image data?<\/p>\n\n\n\n

Abstract<\/strong><\/p>\n\n\n\n

Deep Neural Networks are considered black-box mod- els as the feature embedding are often entangled and not human-interpretable. Though the feature embedding en- codes the information regarding understandable attributes (e.g., tail, head, wheel), it is distributed in the embedding and thus entangled. In this paper, we propose a simple method to disentangle dense image representations into a weighted combination of interpretable visual attributes by learning a simple linear projection in the joint space of a vision language model (VLM) in a data-driven fashion . By using images to weight attributes, our attributes are more semantically meaningful to the underlying data distribution than the generic curated attribute methods proposed in the literature. We demonstrate that our interpretable embed- ding performs competitively as compared to the dense em- bedding. Our discovered attributes act as better prompts under few shot settings, outperforming prompt-tuning methods (CoOp ) by 3% on 1 and 2 shot settings. Moreover, we also show a variety of potential downstream applications enabled by data-driven attributes, such as measuring domain shift, open vocabulary classifications and attribute- guided zero shot detection.<\/p>\n\n\n\n

Contribution<\/strong><\/p>\n\n\n\n

    \n
  1. We propose a simple yet efficient method to disentangle dense visual features to a weighted combination of interpretable attribute feature in language domain. <\/li>\n\n\n\n
  2. We shows that our interpretable feature is a semantically meaningful yet a competitive feature in adapting visual concept by showing our superior performance on few shot evaluation. <\/li>\n\n\n\n
  3. We also shows the potential of the attribute feature by showing a skew of downstream application like measuring distribution shrift, solving open-vocabulary classification and detection.<\/li>\n<\/ol>\n\n\n\n

    Motivation<\/strong><\/p>\n\n\n\n

      \n
    1. Feature embedding encodes the information regarding understandable attributes (e.g., tail, head, wheel) over data distribution, but it is distributed in the embedding and thus entangled.<\/li>\n\n\n\n
    2. Semantic based disentangled feature can be interpretable data distribution measurement. <\/li>\n<\/ol>\n\n\n\n
      \"\"
      Visual Feature disentanglement<\/figcaption><\/figure>\n\n\n\n

      Pipeline<\/strong><\/p>\n\n\n\n

        \n
      1. Visual feature encoding and attribute text feature encoding with CLIP.<\/li>\n\n\n\n
      2. Calculate pair-wise cosine similarity score between each attribute and class feature.<\/li>\n\n\n\n
      3. Dataset adoption by performing linear probing over cosine similarity. <\/li>\n\n\n\n
      4. Class-wise visual feature become a weighted combination of interpretable attribute feature.<\/li>\n<\/ol>\n\n\n\n
        \"\"
        Pipeline<\/figcaption><\/figure>\n\n\n\n

        Application<\/strong><\/p>\n\n\n\n

          \n
        1. Few shot generalization\n\n\n\n\n