Abstraction in Model Based Partially Observable Reinforcement Learning using Extended Sequence Trees

Date

2012-12-07

Author

Cilden, Erkin
Polat, Faruk

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

157
views

0
downloads

Extended sequence tree is a direct method for automatic generation of useful abstractions in reinforcement learning, designed for problems that can be modelled as Markov decision process. This paper proposes a method to expand the extended sequence tree method over reinforcement learning to cover partial observability formalized via partially observable Markov decision process through belief state formalism. This expansion requires a reasonable approximation of information state. Inspired by statistical ranking, a simple but effective discretization schema over belief state space is defined. Extended sequence tree method is modified to make use of this schema under partial observability, and effectiveness of resulting algorithm is shown by experiments on some benchmark problems.

Subject Keywords

Reinforcement learning, Learning abstractions, Partially observable markov decision process, Extended sequence tree

URI

https://hdl.handle.net/11511/39411

DOI

https://doi.org/10.1109/wi-iat.2012.161

Collections

Department of Computer Engineering, Conference / Seminar

Suggestions

OpenMETU
Core

Toward Generalization of Automated Temporal Abstraction to Partially Observable Reinforcement Learning Cilden, Erkin; Polat, Faruk (2015-08-01) Temporal abstraction for reinforcement learning (RL) aims to decrease learning time by making use of repeated sub-policy patterns in the learning task. Automatic extraction of abstractions during RL process is difficult but has many challenges such as dealing with the curse of dimensionality. Various studies have explored the subject under the assumption that the problem domain is fully observable by the learning agent. Learning abstractions for partially observable RL is a relatively less explored area. In...
Abstraction in reinforcement learning in partially observable environments Çilden, Erkin; Polat, Faruk; Department of Computer Engineering (2014) Reinforcement learning defines a prominent family of unsupervised machine learning methods in autonomous agents perspective. Markov decision process model provides a solid formal basis for reinforcement learning algorithms. Temporal abstraction mechanisms can be built on reinforcement learning and significant performance gain can be achieved. If the full observability assumption of Markov decision process model is relaxed, the resulting model is partially observable Markov decision process, which constitute...
Learning by Automatic Option Discovery from Conditionally Terminating Sequences Girgin, Sertan; Polat, Faruk; Alhajj, Reda (2006-08-28) This paper proposes a novel approach to discover options in the form of conditionally terminating sequences, and shows how they can be integrated into reinforcement learning framework to improve the learning performance. The method utilizes stored histories of possible optimal policies and constructs a specialized tree structure online in order to identify action sequences which are used frequently together with states that are visited during the execution of such sequences. The tree is then used to implici...
Recursive Compositional Reinforcement Learning for Continuous Control Sürekli Kontrol Uygulamalari için Özyinelemeli Bileşimsel Pekiştirmeli Öǧrenme Tanik, Guven Orkun; Ertekin Bolelli, Şeyda (2022-01-01) Compositional and temporal abstraction is the key to improving learning and planning in reinforcement learning. Modern real-world control problems call for continuous control domains and robust, sample efficient and explainable control frameworks. We are presenting a framework for recursively composing control skills to solve compositional and progressively complex tasks. The framework promotes reuse of skills, and as a result quickly adaptable to new tasks. The decision-tree can be observed, providing insi...
EXTRACTION OF INTERPRETABLE DECISION RULES FROM BLACK-BOX MODELS FOR CLASSIFICATION TASKS GALATALI, EGEMEN BERK; ALEMDAR, HANDE; Department of Computer Engineering (2022-8-31) In this work, we have proposed a new method and ready to use workflow to extract simplified rule sets for a given Machine Learning (ML) model trained on a classifi- cation task. Those rules are both human readable and in the form of software code pieces thanks to the syntax of Python programming language. We have inspired from the power of Shapley Values as our source of truth to select most prominent features for our rule sets. The aim of this work to select the key interval points in given data in order t...

Citation Formats

E. Cilden and F. Polat, “Abstraction in Model Based Partially Observable Reinforcement Learning using Extended Sequence Trees,” 2012, Accessed: 00, 2020. [Online]. Available: https://hdl.handle.net/11511/39411.