Multimedia data modeling and semantic analysis by multimodal decision fusion

Download

index.pdf

Date

2015

Author

Güder, Mennan

Metadata

Show full item record

Item Usage Stats

191
views

88
downloads

In this thesis, we propose a multi-modal event recognition framework based on the integration of event modeling, fusion, deep learning and, association rule mining. Event modeling is achieved through visual concept learning, scene segmentation and association rule mining. Visual concept learning is employed to reveal the semantic gap between the visual content and the textual descriptors of the events. Association rules are discovered by a specialized association rule mining algorithm where the proposed strategy integrates temporality into the rule discovery process. In addition to physical parts of video, the concept of scene segment is proposed to define and extract elements of association rules. Various feature sources such as audio, motion, keypoint descriptors, temporal occurrence characteristics and fully connected layer outputs of CNN model are combined into the feature fusion. The proposed decision fusion approach employs logistic regression to formulate the relation between dependent variable (event type) and independent variables (classifiers’ outputs) in terms of decision weights. The main motivation in this thesis is to construct a multimodal fusion system which detects events in video by examining feature and decision sources. Various feature sets such as audio, visual, motion and deep learning are investigated. The proposed system employs a decision fusion methodology as the final step of semantic analysis. The main issues that are investigated throughout this study are robustness to uncertainty, better event recognition by use of multi-modal fusion, deep learning outputs, extracted rules, and flexibility in representation.

Subject Keywords

Event processing (Computer science)., Semantic computing., Artificial intelligence.

URI

http://etd.lib.metu.edu.tr/upload/12619519/index.pdf
https://hdl.handle.net/11511/25263

Collections

Graduate School of Natural and Applied Sciences, Thesis

Suggestions

OpenMETU
Core

Multi-modal video event recognition based on association rules and decision fusion Guder, Mennan; Çiçekli, Fehime Nihan (2018-02-01) In this paper, we propose a multi-modal event recognition framework based on the integration of feature fusion, deep learning, scene classification and decision fusion. Frames, shots, and scenes are identified through the video decomposition process. Events are modeled utilizing features of and relations between the physical video parts. Event modeling is achieved through visual concept learning, scene segmentation and association rule mining. Visual concept learning is employed to reveal the semantic gap b...
Hierarchical multitasking control of discrete event systems: Computation of projections and maximal permissiveness Schmidt, Klaus Verner; Cury, José E.r. (null; 2010-12-01) This paper extends previous results on the hierarchical and decentralized control of multitasking discrete event systems (MTDES). Colored observers, a generalization of the observer property, together with local control consistency, allow to derive sufficient conditions for synthesizing modular and hierarchical control that are both strongly nonblocking (SNB) and maximally permissive. A polynomial procedure to verify if a projection fulfills the above properties is proposed and in the case they fail for a g...
Multi-objective decision making using fuzzy discrete event systems: A mobile robot example Boutalis, Yiannis; Schmidt, Klaus Verner (2010-09-29) In this paper, we propose an approach for the multi-objective control of sampled data systems that can be modeled as fuzzy discrete event systems (FDES). In our work, the choice of a fuzzy system representation is justified by the assumption of a controller realization that depends on various potentially imprecise sensor measurements. Our approach consists of three basic steps that are performed in each sampling instant. First, the current fuzzy state of the system is determined by a sensor evaluation. Seco...
Semantic information-based alternative plan generation for multiple query optimization Polat, Faruk; Alhajj, R (Elsevier BV, 2001-09-01) This paper addresses the impact of semantic information about queries on alternative plan generation (APG) for multiple query optimization (MQO). MQO covers optimizing the execution of a set of queries together where each query in the set to be optimized has several alternative execution plans. A multiple query optimizer selects an alternative plan for each query to obtain an optimal global execution plan. Our approach uses information such as common relations, common possible joins and common conditions to...
Generation of cyclic/toroidal chaos by Hopfield neural networks Akhmet, Marat (Elsevier BV, 2014-12-05) We discuss the appearance of cyclic and toroidal chaos in Hopfield neural networks. The theoretical results may strongly relate to investigations of brain activities performed by neurobiologists. As new phenomena, extension of chaos by entrainment of several limit cycles as well as the attraction of cyclic chaos by an equilibrium are discussed. Appropriate simulations that support the theoretical results are depicted. Stabilization of tori in a chaotic attractor is realized not only for neural networks, but...

Citation Formats

M. Güder, “Multimedia data modeling and semantic analysis by multimodal decision fusion,” Ph.D. - Doctoral Program, Middle East Technical University, 2015.