Towards Zero-shot Sign Language Recognition

Download

index.pdf

Date

2022-01-01

Author

Bilge, Yunus Can
Cinbiş, Ramazan Gökberk
Ikizler-Cinbis, Nazli

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

232
views

164
downloads

This paper tackles the problem of zero-shot sign language recognition (ZSSLR), where the goal is to leverage models learned over the seen sign classes to recognize the instances of unseen sign classes. In this context, readily available textual sign descriptions and attributes collected from sign language dictionaries are utilized as semantic class representations for knowledge transfer. For this novel problem setup, we introduce three benchmark datasets with their accompanying textual and attribute descriptions to analyze the problem in detail. Our proposed approach builds spatiotemporal models of body and hand regions. By leveraging the descriptive text and attribute embeddings along with these visual representations within a zero-shot learning framework, we show that textual and attribute based class definitions can provide effective knowledge for the recognition of previously unseen sign classes. We additionally introduce techniques to analyze the influence of binary attributes in correct and incorrect zero-shot predictions. We anticipate that the introduced approaches and the accompanying datasets will provide a basis for further exploration of zero-shot learning in sign language recognition.

Subject Keywords

Assistive technologies, Benchmark testing, Gesture recognition, Hidden Markov models, Semantics, Sign language recognition, Videos, Visualization, zero-shot learning

URI

https://hdl.handle.net/11511/97709

Journal

IEEE Transactions on Pattern Analysis and Machine Intelligence

DOI

https://doi.org/10.1109/tpami.2022.3143074

Collections

Department of Computer Engineering, Article

Suggestions

OpenMETU
Core

Zero-Shot Sign Language Recognition: Can Textual Data Uncover Sign Languages? Bilge, Yunus Can; İkizler Cinbiş, Nazlı; Cinbiş, Ramazan Gökberk (2019-09-12) We introduce the problem of zero-shot sign language recognition (ZSSLR), where the goal is to leverage models learned over the seen sign class examples to recognize the instances of unseen signs. To this end, we propose to utilize the readily available descriptions in sign language dictionaries as an intermediate-level semantic representation for knowledge transfer. We introduce a new benchmark dataset called ASL-Text that consists of 250 sign language classes and their accompanying textual descriptions. Co...
Towards a new typology of coordinated wh-questions Citko, Barbara; Gracanın Yüksek, Martına (Cambridge University Press (CUP), 2013-03-01) In this paper, we develop a new typology of multiple wh-questions with coordinated wh-pronouns. We motivate the existence of three distinct structures for such questions: one mono-clausal and two bi-clausal. We use four kinds of diagnostics to determine which of the three structures is available in a particular language: the availability of both multiple wh-questions and wh-questions with coordinated wh-pronouns, coordination of two argument wh-phrases, transitivity restrictions and superiority effects.
Face classification with support vector machine Kepenekci, B; Akar, Gözde (2004-04-30) A new approach to feature based frontal face recognition with Gabor wavelets and support vector machines is presented in this paper. The feature points are automatically extracted using the local characteristics of each individual face. A kernel that computes the similarity between two feature vectors, is used to map the face features to a space with higher dimension. To find the identity of a test face, the possible labels of each feature vector of that face is found with support vector machines, then the ...
Accelerating Learning of Special Education Studentswith Intellectual Disability via Technology Enhanced Extracurriculum Kaplan, Göknur; Doğan, Sibel (null; 2015-11-30) This study illustrates an effective practice utilizing an innovative instructional design, namely, technologyenhanced extracurriculum (TEE) created for special education students with intellectual disability. A formative research with post-facto multiple cases was designed to find out how a TEE affects students with intellectual disability in terms of cognitive and physical development; along with teachers’ perceptions about technology use in special education. Findings showed that TEE accelerates learning ...
Sign Language Recognition By Image Analysis Buyuksarac, Buket; Bulut, Mehmet Mete; Akar, Gözde (2016-05-19) The Sign Language Recognition (SLR) Problem is a highly important research topic, because of its ability to increase the interaction between the people who are hearing-impaired or impediment in speech. We propose a simple but robust system. The proposed system consists of three main steps. First we apply segmentation to the face and hand region by using Fuzzy C-Means Clustering (FCM) and Thresholding. FCM is a clustering technique which employs fuzzy partitioning, in an iterative algorithm. After the face a...

Citation Formats

Y. C. Bilge, R. G. Cinbiş, and N. Ikizler-Cinbis, “Towards Zero-shot Sign Language Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 0–0, 2022, Accessed: 00, 2022. [Online]. Available: https://hdl.handle.net/11511/97709.