EFFICIENT MULTIMODAL CNN-TRANSFORMER ARCHITECTURE WITH FEATURE DISTILLATION FOR REMOTE SENSING UNDER MISSING MODALITIES

Date

2026-9-3

Author

Moridsedaghat, Laya

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

89
views

0
downloads

Cite This

Multimodal remote sensing semantic segmentation can benefit from combining different types of spectral information, such as RGB, near-infrared (NIR), and shortwave infrared (SWIR) bands. Since each modality provides different information about the same area, using them together can improve segmentation performance. However, in real-world applications, one or more modalities may be unavailable because of sensor failures, differences in data acquisition time, or incomplete observations. This can reduce the reliability of models that depend on having all modalities available. This thesis proposes the CBC-Distilled, an efficient multimodal CNN–Transformer model developed based on the previously proposed CBC architecture. The aim is to improve the model’s ability to handle missing modalities while reducing its computational cost. In the new architecture we use a feature distillation strategy to transfer feature level knowledge from a complete-modality teacher branch to unimodal student encoders during training. The cross-band correlation block is used to model interactions between homogeneous spectral bands and enhance multimodal feature fusion. Random modality masking is also applied during training to expose the model to different missing-modality situations. The teacher and auxiliary branches are removed during inference, keeping the final model lightweight. Experiments on DSTL dataset show that the proposed model improves the F1 score compared with the CBC baseline while using fewer parameters and lower computational cost. Overall, the proposed model provides a good balance between segmentation accuracy, robustness to missing modalities, and computational efficiency.

Subject Keywords

Semantic segmentation, Missing modality, Feature distillation, Remote sensing, Multimodality

URI

https://hdl.handle.net/11511/120680

Collections

Graduate School of Informatics, Thesis