Multimedia data modeling and semantic analysis by multimodal decision fusion

Güder, Mennan
In this thesis, we propose a multi-modal event recognition framework based on the integration of event modeling, fusion, deep learning and, association rule mining. Event modeling is achieved through visual concept learning, scene segmentation and association rule mining. Visual concept learning is employed to reveal the semantic gap between the visual content and the textual descriptors of the events. Association rules are discovered by a specialized association rule mining algorithm where the proposed strategy integrates temporality into the rule discovery process. In addition to physical parts of video, the concept of scene segment is proposed to define and extract elements of association rules. Various feature sources such as audio, motion, keypoint descriptors, temporal occurrence characteristics and fully connected layer outputs of CNN model are combined into the feature fusion. The proposed decision fusion approach employs logistic regression to formulate the relation between dependent variable (event type) and independent variables (classifiers’ outputs) in terms of decision weights. The main motivation in this thesis is to construct a multimodal fusion system which detects events in video by examining feature and decision sources. Various feature sets such as audio, visual, motion and deep learning are investigated. The proposed system employs a decision fusion methodology as the final step of semantic analysis. The main issues that are investigated throughout this study are robustness to uncertainty, better event recognition by use of multi-modal fusion, deep learning outputs, extracted rules, and flexibility in representation.


Multi-modal video event recognition based on association rules and decision fusion
Guder, Mennan; Çiçekli, Fehime Nihan (2018-02-01)
In this paper, we propose a multi-modal event recognition framework based on the integration of feature fusion, deep learning, scene classification and decision fusion. Frames, shots, and scenes are identified through the video decomposition process. Events are modeled utilizing features of and relations between the physical video parts. Event modeling is achieved through visual concept learning, scene segmentation and association rule mining. Visual concept learning is employed to reveal the semantic gap b...
Hierarchical multitasking control of discrete event systems: Computation of projections and maximal permissiveness
Schmidt, Klaus Verner; Cury, José E.r. (null; 2010-12-01)
This paper extends previous results on the hierarchical and decentralized control of multitasking discrete event systems (MTDES). Colored observers, a generalization of the observer property, together with local control consistency, allow to derive sufficient conditions for synthesizing modular and hierarchical control that are both strongly nonblocking (SNB) and maximally permissive. A polynomial procedure to verify if a projection fulfills the above properties is proposed and in the case they fail for a g...
Multi-objective decision making using fuzzy discrete event systems: A mobile robot example
Boutalis, Yiannis; Schmidt, Klaus Verner (2010-09-29)
In this paper, we propose an approach for the multi-objective control of sampled data systems that can be modeled as fuzzy discrete event systems (FDES). In our work, the choice of a fuzzy system representation is justified by the assumption of a controller realization that depends on various potentially imprecise sensor measurements. Our approach consists of three basic steps that are performed in each sampling instant. First, the current fuzzy state of the system is determined by a sensor evaluation. Seco...
Applied supervisory control for a flexible manufacturing system
Moor, Thomas; Schmidt, Klaus Verner; Perk, Sebastian (2010-12-01)
This paper presents a case study in the design and implementation of a discrete event system (DES) of real-world complexity. Our DES plant is a flexible manufacturing system (FMS) laboratory model that consists of 29 interacting components and is controlled via 107 digital signals. Regarding controller design, we apply a hierarchical and decentralised synthesis method from earlier work in order to achieve nonblocking and safe closed-loop behaviour. Regarding implementation, we discuss how digital signals tr...
Multi-camera video surveillance : detection, occlusion handling, tracking and event recognition
Akman, Oytun; Alatan, Abdullah Aydın; Department of Electrical and Electronics Engineering (2007)
In this thesis, novel methods for background modeling, tracking, occlusion handling and event recognition via multi-camera configurations are presented. As the initial step, building blocks of typical single camera surveillance systems that are moving object detection, tracking and event recognition, are discussed and various widely accepted methods for these building blocks are tested to asses on their performance. Next, for the multi-camera surveillance systems, background modeling, occlusion handling, tr...
Citation Formats
M. Güder, “Multimedia data modeling and semantic analysis by multimodal decision fusion,” Ph.D. - Doctoral Program, Middle East Technical University, 2015.