Weakly supervised instance attention for multisource fine-grained object recognition with an application to tree species classification

Date

2021-06-01

Author

Aygunes, Bulut
Cinbiş, Ramazan Gökberk
Aksoy, Selim

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

263
views

0
downloads

Multisource image analysis that leverages complementary spectral, spatial, and structural information benefits fine-grained object recognition that aims to classify an object into one of many similar subcategories. However, for multisource tasks that involve relatively small objects, even the smallest registration errors can introduce high uncertainty in the classification process. We approach this problem from a weakly supervised learning perspective in which the input images correspond to larger neighborhoods around the expected object locations where an object with a given class label is present in the neighborhood without any knowledge of its exact location. The proposed method uses a single-source deep instance attention model with parallel branches for joint localization and classification of objects, and extends this model into a multisource setting where a reference source that is assumed to have no location uncertainty is used to aid the fusion of multiple sources in four different levels: probability level, logit level, feature level, and pixel level. We show that all levels of fusion provide higher accuracies compared to the state-of-the-art, with the best performing method of feature-level fusion resulting in 53% accuracy for the recognition of 40 different types of trees, corresponding to an improvement of 5.7% over the best performing baseline when RGB, multispectral, and LiDAR data are used. We also provide an in-depth comparison by evaluating each model at various parameter complexity settings, where the increased model capacity results in a further improvement of 6.3% over the default capacity setting.

Subject Keywords

Multisource classification, Fine-grained object recognition, Weakly supervised learning, Deep learning

URI

https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85106225010&origin=inward
https://hdl.handle.net/11511/91088

Journal

ISPRS Journal of Photogrammetry and Remote Sensing

DOI

https://doi.org/10.1016/j.isprsjprs.2021.03.021

Collections

Department of Computer Engineering, Article

Suggestions

OpenMETU
Core

WEAKLY SUPERVISED DEEP CONVOLUTIONAL NETWORKS FOR FINE-GRAINED OBJECT RECOGNITION IN MULTISPECTRAL IMAGES Aygunes, Bulut; AKSOY, SELİM; Cinbiş, Ramazan Gökberk (2019-01-01) The challenging task of training object detectors for fine-grained classification faces additional difficulties when there are registration errors between the image data and the ground truth. We propose a weakly supervised learning methodology for the classification of 40 types of trees by using fixed-sized multispectral images with a class label but with no exact knowledge of the object location. Our approach consists of an end-to-end trainable convolutional neural network with separate branches for learni...
Multisource region attention network for fine-grained object recognition in remote sensing imagery Sümbül, Gencer; Cinbiş, Ramazan Gökberk; Aksoy, Selim (Institute of Electrical and Electronics Engineers (IEEE), 2019-07) Fine-grained object recognition concerns the identification of the type of an object among a large number of closely related subcategories. Multisource data analysis that aims to leverage the complementary spectral, spatial, and structural information embedded in different sources is a promising direction toward solving the fine-grained recognition problem that involves low between-class variance, small training set sizes for rare classes, and class imbalance. However, the common assumption of coregistered ...
Fine-Grained Object Recognition and Zero-Shot Learning in Remote Sensing Imagery Sumbul, Gencer; Cinbiş, Ramazan Gökberk; Aksoy, Selim (2018-02-01) Fine-grained object recognition that aims to identify the type of an object among a large number of subcategories is an emerging application with the increasing resolution that exposes new details in image data. Traditional fully supervised algorithms fail to handle this problem where there is low betweenclass variance and high within-class variance for the classes of interest with small sample sizes. We study an even more extreme scenario named zero-shot learning (ZSL) in which no training example exists f...
Deep metric learning with distance sensitive entangled triplet losses Karaman, Kaan; Alatan, Abdullah Aydın; Department of Electrical and Electronics Engineering (2021-2-12) Metric learning aims to define a distance that is able to measure the semantic difference between the instances in a dataset. The most recent approaches in this area mostly utilize deep neural networks as their models to map the input data into a feature space by finding appropriate distance metrics between the features. A number of loss functions are already defined in the literature based on these similarity metrics to discriminate instances in the feature space. In this thesis, we particularly focus on t...
Training object detectors by directly optimizing lrp metric Çam, Barış Can; Akbaş, Emre; Kalkan, Sinan; Department of Computer Engineering (2020-9) This thesis focuses on training deep object detection networks by directly optimizing the localisation-recall-precision (LRP) performance metric that can evaluate classification and localisation performance of an object detector in a unified manner (Oksuz et al., 2018). To achieve this goal, unlike the commonly used linear weighting approach, we aim to implicitly optimize the LRP metric first by using a bounded localisation loss from previous works and proposing a loss function that can bound the range ...

Citation Formats

B. Aygunes, R. G. Cinbiş, and S. Aksoy, “Weakly supervised instance attention for multisource fine-grained object recognition with an application to tree species classification,” ISPRS Journal of Photogrammetry and Remote Sensing, pp. 262–274, 2021, Accessed: 00, 2021. [Online]. Available: https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85106225010&origin=inward.