Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning

Date

2025-01-01

Author

Topçuoğlu, Utku Mert
Akagündüz, Erdem

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

57
views

0
downloads

Cite This

This paper presents an innovative approach to self-supervised learning for Vision Transformers (ViTs), integrating local masked image modeling with progressive layer freezing. This method enhances the efficiency and speed of initial layer training in ViTs. By systematically freezing specific layers at strategic points during training, we reduce computational demands while maintaining learning capabilities. Our approach employs a novel multi-scale reconstruction process that fosters efficient learning in initial layers and enhances semantic comprehension across scales. The results demonstrate a substantial reduction in training time (12.5%) with a minimal impact on model accuracy (decrease in top-1 accuracy by 0.6%). Our method achieves top-1 and top-5 accuracies of 82.6% and 96.2%, respectively, underscoring its potential in scenarios where computational resources and time are critical. The implementation of our approach is available at our project's GitHub repository: https://github.com/utkutpcgl/ViTFreeze.

Subject Keywords

Computer Vision, Deep Learning, Masked Image Modeling, Self-Supervised Learning, Training Efficiency

URI

https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=105000170796&origin=inward
https://hdl.handle.net/11511/114317

Collections

Graduate School of Informatics, Conference / Seminar