Local Roots A Tree Based Subgoal Discovery Method to Accelerate Reinforcement Learning

Date

2016-12-04

Author

Demir, Alper
Polat, Faruk
Cilden, Erkin

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

227
views

0
downloads

Subgoal discovery in reinforcement learning is an effective way of partitioning a problem domain with large state space. Recent research mainly focuses on automatic identification of such subgoals during learning, making use of state transition information gathered during exploration. Mostly based on the options framework, an identified subgoal leads the learning agent to an intermediate region which is known to be useful on the way to goal. In this paper, we propose a novel automatic subgoal discovery method which is based on analysis of predicted shortcut history segments derived from experience, which are then used to generate useful options to speed up learning. Compared to similar existing methods, it performs significantly better in terms of time complexity and usefulness of the subgoals identified, without sacrificing solution quality. The effectiveness of the method is empirically shown via experimentation on various benchmark problems compared to well known subgoal identification methods.

Subject Keywords

Abstraction in reinforcement learning, Subgoal discovery, Options framework

URI

https://hdl.handle.net/11511/68833

DOI

https://doi.org/10.1007/978-3-319-46227-1_23

Collections

Department of Computer Engineering, Conference / Seminar

Suggestions

OpenMETU
Core

A Concept Filtering Approach for Diverse Density to Discover Subgoals in Reinforcement Learning DEMİR, ALPER; Cilden, Erkin; Polat, Faruk (2017-11-08) In the reinforcement learning context, subgoal discovery methods aim to find bottlenecks in problem state space so that the problem can naturally be decomposed into smaller subproblems. In this paper, we propose a concept filtering method that extends an existing subgoal discovery method, namely diverse density, to be used for both fully and partially observable RL problems. The proposed method is successful in discovering useful subgoals with the help of multiple instance learning. Compared to the original...
Improving reinforcement learning using distinctive clues of the environment Demir, Alper; Polat, Faruk; Department of Computer Engineering (2019) Effective decomposition and abstraction has been shown to improve the performance of Reinforcement Learning. An agent can use the clues from the environment to either partition the problem into sub-problems or get informed about its progress in a given task. In a fully observable environment such clues may come from subgoals while in a partially observable environment they may be provided by unique experiences. The contribution of this thesis is two fold; first improvements over automatic subgoal identifica...
Finite bisimulations for switched linear systems Aydın Göl, Ebru; Lazar, Mircea; Belta, Calin (2013-02-04) In this paper, we consider the problem of constructing a finite bisimulation quotient for a discrete-time switched linear system in a bounded subset of its state space. Given a set of observations over polytopic subsets of the state space and a switched linear system with stable subsystems, the proposed algorithm generates the bisimulation quotient in a finite number of steps with the aid of sublevel sets of a polyhedral Lyapunov function. Starting from a sublevel set that includes the origin in its interio...
Finite Bisimulations for Switched Linear Systems Aydın Göl, Ebru; Lazar, Mircea; Belta, Calin (2014-12-01) In this paper, we consider the problem of constructing a finite bisimulation quotient for a discrete-time switched linear system in a bounded subset of its state space. Given a set of observations over polytopic subsets of the state space and a switched linear system with stable subsystems, the proposed algorithm generates the bisimulation quotient in a finite number of steps with the aid of sublevel sets of a polyhedral Lyapunov function. Starting from a sublevel set that includes the origin in its interio...
Near-field performance analysis of locally-conformal perfectly matched absorbers via Monte Carlo simulations Ozgun, Ozlem; Kuzuoğlu, Mustafa (2007-12-10) In the numerical solution of some boundary value problems by the finite element method (FEM), the unbounded domain must be truncated by an artificial absorbing boundary or layer to have a bounded computational domain. The perfectly matched layer (PML) approach is based on the truncation of the computational domain by a reflectionless artificial layer which absorbs outgoing waves regardless of their frequency and angle of incidence. In this paper, we present the near-field numerical performance analysis of o...

Citation Formats

A. Demir, F. Polat, and E. Cilden, “Local Roots A Tree Based Subgoal Discovery Method to Accelerate Reinforcement Learning,” 2016, Accessed: 00, 2020. [Online]. Available: https://hdl.handle.net/11511/68833.