Improving reinforcement learning using distinctive clues of the environment

Date

2019

Author

Demir, Alper

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

230
views

0
downloads

Effective decomposition and abstraction has been shown to improve the performance of Reinforcement Learning. An agent can use the clues from the environment to either partition the problem into sub-problems or get informed about its progress in a given task. In a fully observable environment such clues may come from subgoals while in a partially observable environment they may be provided by unique experiences. The contribution of this thesis is two fold; first improvements over automatic subgoal identification and option generation in fully observable environments is proposed, then an automatic landmark identification and an anchor based guiding mechanism in partially observable environments is introduced. Moreover, for both type of problems, the thesis proposes an overall framework that is shown to outperform baseline learning algorithms on several benchmark domains.

Subject Keywords

Reinforcement learning (Machine learning), Keywords: Reinforcement Learning, Automatic Subgoal Identification, Options Framework, Automatic Landmark Identification, Anchor Based Guiding.

URI

http://etd.lib.metu.edu.tr/upload/12624698/index.pdf
https://hdl.handle.net/11511/45192

Collections

Graduate School of Natural and Applied Sciences, Thesis

Suggestions

OpenMETU
Core

Recursive Compositional Reinforcement Learning for Continuous Control Sürekli Kontrol Uygulamalari için Özyinelemeli Bileşimsel Pekiştirmeli Öǧrenme Tanik, Guven Orkun; Ertekin Bolelli, Şeyda (2022-01-01) Compositional and temporal abstraction is the key to improving learning and planning in reinforcement learning. Modern real-world control problems call for continuous control domains and robust, sample efficient and explainable control frameworks. We are presenting a framework for recursively composing control skills to solve compositional and progressively complex tasks. The framework promotes reuse of skills, and as a result quickly adaptable to new tasks. The decision-tree can be observed, providing insi...
A Concept Filtering Approach for Diverse Density to Discover Subgoals in Reinforcement Learning DEMİR, ALPER; Cilden, Erkin; Polat, Faruk (2017-11-08) In the reinforcement learning context, subgoal discovery methods aim to find bottlenecks in problem state space so that the problem can naturally be decomposed into smaller subproblems. In this paper, we propose a concept filtering method that extends an existing subgoal discovery method, namely diverse density, to be used for both fully and partially observable RL problems. The proposed method is successful in discovering useful subgoals with the help of multiple instance learning. Compared to the original...
Using chains of bottleneck transitions to decompose and solve reinforcement learning tasks with hidden states Aydın, Hüseyin; Çilden, Erkin; Polat, Faruk (2022-08-01) Reinforcement learning is known to underperform in large and ambiguous problem domains under partial observability. In such cases, a proper decomposition of the task can improve and accelerate the learning process. Even ambiguous and complex problems that are not solvable by conventional methods turn out to be easier to handle by using a convenient problem decomposition, followed by the incorporation of machine learning methods for the sub-problems. Like in most real-life problems, the decomposition of a ta...
Local Roots A Tree Based Subgoal Discovery Method to Accelerate Reinforcement Learning Demir, Alper; Polat, Faruk; Cilden, Erkin (2016-12-04) Subgoal discovery in reinforcement learning is an effective way of partitioning a problem domain with large state space. Recent research mainly focuses on automatic identification of such subgoals during learning, making use of state transition information gathered during exploration. Mostly based on the options framework, an identified subgoal leads the learning agent to an intermediate region which is known to be useful on the way to goal. In this paper, we propose a novel automatic subgoal discovery meth...
Positive impact of state similarity on reinforcement learning performance Girgin, Sertan; Polat, Faruk; Alhaj, Reda (Institute of Electrical and Electronics Engineers (IEEE), 2007-10-01) In this paper, we propose a novel approach to identify states with similar subpolicies and show how they can be integrated into the reinforcement learning framework to improve learning performance. The method utilizes a specialized tree structure to identify common action sequences of states, which are derived from possible optimal policies, and defines a similarity function between two states based on the number of such sequences. Using this similarity function, updates on the action-value function of a st...

Citation Formats

A. Demir, “Improving reinforcement learning using distinctive clues of the environment,” Thesis (Ph.D.) -- Graduate School of Natural and Applied Sciences. Computer Engineering., Middle East Technical University, 2019.