Automated learning rate search using batch-level cross-validation

Download

index.pdf

Date

2021-04-01

Author

KABAKÇI, Duygu
Akbaş, Emre

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

119
views

45
downloads

Deep learning researchers and practitioners have accumulated a significant amount of experience on training a wide variety of architectures on various datasets. However, given anetwork architecture and a dataset, obtaining the best model (i.e. the model giving the smallest test set error) while keeping the training time complexity low is still a challenging task. Hyper-parameters of deep neural networks, especially the learning rate and its (decay) schedule, highly affect the network's final performance. The general approach is to search the best learning rate and learning rate decay parameters within a cross-validation framework, a process that usually requires a significant amount ofexperimentation with extensive time cost. In classical cross-validation (CV), a random part of the dataset is reserved for the evaluation of model performance on unseen data. This technique is usually run multiple times to decide learning rate settings with random validation sets. In this paper, we explorebatch-level cross-validation as an alternative to the classical dataset-level, hence macro, CV. The advantage of batch-level or micro CV methods is that the gradient computed during training is re-used to evaluate several different learning rates. We propose an algorithm based on micro CV and stochastic gradient descent with momentum, which produces a learning rate schedule during trainingby selectinga learning rate per epoch,automatically. In our algorithm, a random half of the current batch (of examples) is used for training and the other half is used for validating several different step sizes or learning rates. We conductedcomprehensive experimentson three datasets (CIFAR10, SVHN and Adience) using three different network architectures (a custom CNN, ResNet and VGG) to compare the performances of our micro-CV algorithm and thewidely used stochastic gradient descent with momentum in a early-stopping macro-CV setup. The results show that, our micro-CV algorithm achieves comparable test accuracy to macro-CV witha much lower computational cost.

URI

https://hdl.handle.net/11511/102738

Journal

Sakarya University Journal of Computer and Information Sciences (Online)

DOI

https://doi.org/10.35377/saucis...935353

Collections

Department of Computer Engineering, Article

Suggestions

OpenMETU
Core

Automated learning rate search using batch-level cross-validation Kabakcı, Duygu; Akbaş, Emre; Department of Computer Engineering (2019) Deep convolutional neural networks are being widely used in computer vision tasks, such as object recognition and detection, image segmentation and face recognition, with a variety of architectures. Deep learning researchers and practitioners have accumulated a significant amount of experience on training a wide variety of architectures on various datasets. However, given a specific network model and a dataset, obtaining the best model (i.e. the model giving the smallest test set error) while keeping the tr...
Visual Object Tracking with Autoencoder Representations Besbinar, Beril; Alatan, Abdullah Aydın (2016-05-19) Deep learning is the discipline of training computational models that are composed of multiple layers and these methods have recently improved the state of the art in many areas as a virtue of large labeled datasets, increase in the computational power of current hardware and unsupervised training methods. Although such a dataset may not be available for lots of application areas, the representations obtained by the well-designed networks that have a large representation capacity and trained with enough dat...
Deep Learning-Enabled Technologies for Bioimage Analysis Rabbi, Fazle; Dabbagh, Sajjad Rahmani; Angın, Pelin; Yetisen, Ali Kemal; Tasoglu, Savas (2022-02-01) Deep learning (DL) is a subfield of machine learning (ML), which has recently demon-strated its potency to significantly improve the quantification and classification workflows in bio-medical and clinical applications. Among the end applications profoundly benefitting from DL, cellular morphology quantification is one of the pioneers. Here, we first briefly explain fundamental concepts in DL and then we review some of the emerging DL-enabled applications in cell morphology quantification in the fields of em...
Computational representation of protein sequences for homology detection and classification Oğul, Hasan; Mumcuoğlu, Ünal Erkan; Department of Information Systems (2006) Machine learning techniques have been widely used for classification problems in computational biology. They require that the input must be a collection of fixedlength feature vectors. Since proteins are of varying lengths, there is a need for a means of representing protein sequences by a fixed-number of features. This thesis introduces three novel methods for this purpose: n-peptide compositions with reduced alphabets, pairwise similarity scores by maximal unique matches, and pairwise similarity scores by...
Automated biological data acquisition and integration using machine learning techniques Çarkacıoğlu, Levent; Atalay, Mehmet Volkan; Department of Computer Engineering (2009) Since the initial genome sequencing projects along with the recent advances on technology, molecular biology and large scale transcriptome analysis result in data accumulation at a large scale. These data have been provided in different platforms and come from different laboratories therefore, there is a need for compilation and comprehensive analysis. In this thesis, we addressed the automatization of biological data acquisition and integration from these non-uniform data using machine learning techniques....

Citation Formats

D. KABAKÇI and E. Akbaş, “Automated learning rate search using batch-level cross-validation,” Sakarya University Journal of Computer and Information Sciences (Online), vol. 4, no. 3, pp. 312–325, 2021, Accessed: 00, 2023. [Online]. Available: https://hdl.handle.net/11511/102738.