Augmenting Clip-based few shot recognition using shapes and concepts

Date

2025-8-25

Author

Güder, Erce

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

934
views

0
downloads

Cite This

Contemporary deep learning models, such as CLIP, exhibit strong zero-shot recognition performance on a broad range of tasks. However, they still substantially benefit from limited supervision in few-shot regimes. This thesis investigates ways to inject external knowledge to make few-shot adaptation models more data-efficient and/or interpretable. We start with studying the role of shape-only representations in object recognition and show that they are more data-efficient than raw RGB representations. Motivated by these findings, and texture versus shape bias literature, we propose \emph{v1-shape}, augmenting CLIP-based few shot recognition with an additional shape-conditioned branch, yielding modest gains. We also introduce \emph{v1-concept}, a CLIP-based concept bottleneck model encouraged to base its decisions on more general semantic concepts, improving few-shot accuracy under many settings. Finally, we explore a CLIP adaptation approach blending zero-shot and linear probe logits adaptively during inference.

Subject Keywords

object recognition, image classification, few-shot learning, CLIP adaptation, deep learning

URI

https://hdl.handle.net/11511/115645

Collections

Graduate School of Natural and Applied Sciences, Thesis