VOCABULARY-AWARE DISTILLATION OF VISION-LANGUAGE MODELS FOR GROUNDED, EXPLAINABLE SCENE UNDERSTANDING

Date

2026-9-1

Author

Akyön, Fatih Çağatay

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

23
views

0
downloads

Cite This

This thesis proposes vocabulary-aware distillation of vision-language models for efficient domain-specific scene understanding. Its central contribution is Vocabulary-Aware Recall (VAR), a proposed loss that retains cross-entropy and adds a soft recall penalty over token positions matching a predefined domain vocabulary. The objective gives domain concepts explicit training emphasis within autoregressive sequences. A teacher model provides annotations that undergo human review, and a compact student model learns from them using the vocabulary-aware objective. The method is evaluated on SenBen, a new AI safety benchmark and scene-graph dataset containing 13,999 frames from 157 films. Its structured annotations link objects, attributes, and relationships to sensitivity tags. SenBen-Score measures the quality of these predictions. VAR improves SenBen Recall by 1.85 percentage points over plain cross-entropy and contributes 3.44 points in the full-decoder comparison. The complete training recipe gains 5.2 points on average across six seeds. Compared with an 8.3B-parameter local baseline, the 241M-parameter student improves SenBen F1 by 8.8 percentage points (42.8% versus 34.0%) while running 7.6 times faster and using about one-sixteenth of the GPU memory.

Subject Keywords

Vision-Language Models, Knowledge Distillation, Scene Graph Generation, Vocabulary-Aware Recall, AI Safety

URI

https://hdl.handle.net/11511/121012

Collections

Graduate School of Informatics, Thesis