Date

2020-8

Author

Kalafoğlu, Muhammet Esat

Metadata

Show full item record

Item Usage Stats

304
views

178
downloads

Cite This

Visual action recognition (AR) is the problem of identifying the labels of activities that occur in a video. In this thesis, different spatio-temporal representations are analyzed and the factors making these representations better suited for AR are determined. To be specific, three main concepts are analyzed in this thesis study which are the effects of different architectural selections, the input modalities (RGB, optical flow, human pose), and temporal modeling concepts. Additionally, the joint utilization of BERT-based late temporal modeling with 3D CNN architectures is proposed and a novel distillation concept is recommended within this approach. Firstly, for architectural analysis, both 2D and 3D CNN structures are considered. For 3D CNN architectures, the effects of clip length, input spatial resolution, group convolution, and separable 3D convolution are analyzed. During this analysis, popular 3D CNN architectures for AR, such as MFNET, SlowFast Networks, R(2+1)D networks, I3D, MARS networks (knowledge distillation), and various ResNet architectures are all considered. Temporal shift modules are also investigated as an extension to 2D CNN architectures. For input modality analysis, popular two-stream architectures (RGB+Flow) are analyzed within both 2D and 3D CNN architectures. Moreover, as an extension to RGB v and flow modalities, pose input modality is utilized with a different approach from the literature and studied within the 2D CNN architectures in this thesis. For the temporal modeling analysis, various techniques are analyzed such as average pooling, LSTM, convolutional GRU, BERT, and non-local blocks within 2D CNN architectures. As a novel extension, conventional 3D convolutions are combined with late temporal modeling for AR. The popular temporal global average pooling layer (TGAP) at the end of 3D convolutional architecture is replaced with the recent Bidirectional Encoder epresentations from Transformers (BERT) layer in order to better exploit the attention mechanism of BERT. Such a replacement is shown to improve the performances of many popular 3D convolution architectures, including ResNeXt, I3D, SlowFast, and R(2+1)D. The-state-of-the-art performances are obtained on both HMDB51 and UCF101 datasets with 85.10% and 98.69% Top-1 accuracy, respectively. Finally, anovel knowledge distillation concept is proposed using a 3D-BERT architecture that yields quite promising performances.

Subject Keywords

Spatio-Temporal Features, Two-Stream Networks, 3D Convolution, Temporal Modeling, Action Recognition,, Activity Recognition,, Aktivite Tespiti, Uzamsal-Zamansal Öznitelikler, Zamansal Modelleme, 3D Evrişim, İki Kanallı Ağlar

URI

https://hdl.handle.net/11511/69033

Collections

Graduate School of Natural and Applied Sciences, Thesis