Hierarchical human activity recognition with fusion of audio and multiple inertial sensor modalities

Download
2022-2-08
Yılmaz, Tuğçe Alara
People perform a wide variety of activities every day. Systems that can automatically distinguish these activities, i.e. human activity recognition models, have improved markedly, especially in the last decade. Deep learning is demonstrating increasingly promising outcomes in overcoming the problem of human activity detection as technology improves at a rapid pace. However, validating activity recognition in real-world situations is critical for practical solutions that work in natural contexts. Establishing systems that could achieve automatic activity recognition with real-life settings such as the devices that people use every day naturally and the environment they live in, might require lots of computational complexity. A lightweight neural network model is adopted for this purpose, one that could run swiftly even in the background process without taking up a lot of space when embedded into smartphones. Four inertial sensory data are represented in color coded image form and fused with three channelled audio data image representation to perform recognition task. The resulting fusion images allow rapid recognition performance because the size of the each image is so small. This thesis also underlines that audio sensor data, which require considerably bigger window sizes for identification on their own, improve automated recognition performance when used in conjunction with inertial sensors, even when divided into small window sizes to interact with other sensors simultaneously. In addition, this thesis provides a strategy that helps the computer better discern real-world behaviors by introducing activities, contexts, and placements in a hierarchical manner to perform accurate activity recognition. By merging auditory images with inertial color coded images, representing multiple activity pairs with hierarchical groups according to activity-context-placement, and employing a lightweight model, high accurate recognition performance score competitive to state-of-art, nearly 91\% success rate is achieved. We believe that this research could be classified as a "quality of experience" because it presents a lightweight model that could be used to predict behavior of individual by data collected from devices such as smartphone and smartwatch that everyone uses each day naturally.

Suggestions

Implementation of a Generic Framework on Crowd Simulation: A New Environment to Model Crowd Behavior and Design Video Games
Yücel, Furkan; Sürer, Elif (2020-12-01)
Crowd behavior is the collective act and gathering of a group of individuals to achieve a shared purpose. Swarm intelligence-based optimization algorithms are usually used to solve complex problems for crowd behavior. Crowd simulations are often used for the analyses that require precision in different domains such as complex structural analysis, image recognition, creating nature-inspired non-player character movements in video games, and more. In this study, a generic crowd simulation framework that can b...
Volunteers motivation and sense of community in a sports event
Yıldız, Alper; Koçak, Mehmet Settar; Yıldırım, Sinan; Devrilmez, Erhan (null; 2016-11-25)
Volunteers are indispensable for sports events. Success of the event is related with their well management and coordination. It is expected that keeping volunteers’ motivation and sense of community levels high will affect both volunteers and event positively. Hence, aim of this research was investigating the perceived senses of community and motivations of volunteers, and examining correlations between sense of community and motivation of sports event volunteers. Four hundred seventy-seven of whom voluntee...
A Workflow-based mobile guidance framework for managing personal activities
Tüysüz, Gökhan; Eren, Pekin Erhan; Department of Information Systems (2013)
In daily life, people have to perform a large number of activities typically in a limited amount of time. Thus, they may need help and guidance provided by support systems in order to accomplish these activities accurately and in the correct order. Accordingly, in this study, we propose a software framework based on workflows and supported by a mobile application to assist users in pervasive environments for managing their personal activities. Pervasive computing enables to ease and automate the execution o...
A Workflow Based Mobile Guidance Framework for Managing Personal Activities
TÜYSÜZ, GÖKHAN; AVENOĞLU, BİLGİN; Eren, Pekin Erhan (2013-09-27)
In daily life, people have to perform a large number of activities typically in a limited amount of time. Accordingly, they may benefit from help and guidance provided by support systems in order to accomplish these activities accurately and in the correct order. In order to satisfy such needs we develop a software framework which also incorporates a mobile application. Within the framework, workflows are used for modeling user activities due to their successful structuring and verification capabilities. Ho...
Conflict Resolution Styles Scale in Romantic Relationship The Validity and Reliability Study
Özen, Ayça; Salman Elgin, Selin; Sakallı, Nuray (2016-06-01)
Bu çalışmanın amacı çiftlerin çatışma çözüm stillerini ölçmek amacıyla geçerli ve güvenilir bir ölçek geliştirmektir. Bu amaçla 2 ayrı çalışma yürütülmüştür. İlk çalışmada, romantik ilişki yaşayan ya da evli kişilerle yarı yapılandırılmış mülakatlar gerçekleştirilmiştir. Hem bu mülakatlardan oluşturulan maddeler, hem de alanda uzman kişiler tarafından ilgili yazın incelenerek eklenen maddelerle geniş bir madde havuzu oluşturulmuştur. Romantik ilişki yaşayan 200 katılımcıyla (Ort.yaş = 23.13, S.yaş = 2.96) y...
Citation Formats
T. A. Yılmaz, “Hierarchical human activity recognition with fusion of audio and multiple inertial sensor modalities,” M.S. - Master of Science, Middle East Technical University, 2022.