Hyperparameter Optimization logopeterindia.net
Knowledge Hub

Hyperparameter Optimization Techniques

A curated index on hyperparameter optimization (HPO) — how to search for the best learning rate, depth and regularization settings — with surveys, methods from random search to Hyperband, benchmarks and libraries.

29 topics•7 categories•Updated November 14, 2026

What Is Hyperparameter Optimization?

Hyperparameter optimization (HPO) automates the search for hyperparameters such as learning rate, batch size, tree depth and regularization strength. The simplest methods are grid and random search. Bayesian optimization builds a surrogate model of the objective, for example with Gaussian processes, random forests or the Tree-structured Parzen Estimator, to choose promising configurations. Multi-fidelity methods such as successive halving, Hyperband, BOHB and ASHA stop poor runs early, while evolutionary methods, CMA-ES and population based training evolve configurations, and gradient-based HPO differentiates the validation loss directly. This page lists surveys, search methods, benchmarks and libraries.

Surveys, Books & Practical Guidance

6 entries

Reading on how HPO works and how to apply it.

01Automated Machine Learning: Methods, Systems, Challenges (open-access book)Open-access Springer book from the AutoML community, with chapters on hyperparameter optimization. 02Tunability: Importance of Hyperparameters of Machine Learning AlgorithmsPaper that measures how much tuning each hyperparameter is worth for common algorithms. 03Practical Recommendations for Gradient-Based Training of Deep Architectures (Bengio)Chapter of practical advice on setting hyperparameters for deep networks. 04On Empirical Comparisons of Optimizers for Deep LearningPaper showing that comparisons between optimizers depend on how well each one is tuned. 05Multi-Objective Hyperparameter Optimization in Machine Learning: An OverviewOverview of HPO that balances several objectives, such as accuracy and cost. 06Deep Learning Tuning Playbook (Google Research)Playbook on systematically tuning deep learning models.

Random Search & Evolutionary Methods

3 entries

Methods that need no model of the objective.

07Random Search for Hyper-Parameter Optimization (Bergstra and Bengio)JMLR paper showing that random search is more efficient than grid search. 08The CMA Evolution Strategy: A Tutorial (Nikolaus Hansen)Tutorial on CMA-ES, an evolution strategy widely used for black-box optimization. 09Population Based Training of Neural NetworksPaper on training a population of models while tuning their hyperparameters online.

Bayesian Optimization

6 entries

Surrogate-model methods that choose the next configuration to try.

10A Tutorial on Bayesian Optimization (Peter Frazier)Tutorial on Gaussian-process Bayesian optimization and acquisition functions. 11Practical Bayesian Optimization of Machine Learning Algorithms (Snoek et al.)Paper on Gaussian-process Bayesian optimization for tuning learning algorithms. 12Algorithms for Hyper-Parameter Optimization (Bergstra et al.)NeurIPS 2011 paper on sequential model-based optimization, including the Tree-structured Parzen Estimator. 13SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter OptimizationPaper describing SMAC3, which uses random-forest surrogates. 14BoTorch: A Framework for Efficient Monte-Carlo Bayesian OptimizationPaper on the PyTorch-based Bayesian optimization library behind Ax. 15Google Vizier: A Service for Black-Box OptimizationGoogle Research paper on its internal service for black-box optimization.

Multi-Fidelity & Early Stopping

6 entries

Methods that spend little compute on poor configurations.

16Non-stochastic Best Arm Identification and Hyperparameter Optimization (Jamieson and Talwalkar)Paper that introduced successive halving for hyperparameter optimization. 17Hyperband: A Novel Bandit-Based Approach to Hyperparameter OptimizationPaper on tuning by early-stopping poor configurations with a bandit strategy. 18BOHB: Robust and Efficient Hyperparameter Optimization at ScalePaper that combines Bayesian optimization with Hyperband. 19A System for Massively Parallel Hyperparameter Tuning (ASHA)Paper on the asynchronous successive halving algorithm for parallel tuning. 20DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter OptimizationPaper that combines differential evolution with Hyperband. 21Freeze-Thaw Bayesian OptimizationPaper on pausing and resuming training runs guided by learning-curve predictions.

Gradient-Based HPO

2 entries

Methods that differentiate the validation loss with respect to hyperparameters.

22Gradient-based Hyperparameter Optimization through Reversible Learning (Maclaurin et al.)Paper that computes hyperparameter gradients by reversing the training procedure. 23Optimizing Millions of Hyperparameters by Implicit Differentiation (Lorraine et al.)Paper on scaling hyperparameter gradients to very large numbers of hyperparameters.

Benchmarks

2 entries

Reproducible problems for comparing HPO methods.

24HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOPaper introducing a collection of multi-fidelity HPO benchmarks. 25YAHPO Gym: An Efficient Multi-Objective Multi-Fidelity Benchmark for HPOPaper introducing a surrogate-based benchmark suite for HPO.

Libraries

4 entries

Open-source frameworks for running HPO.

26OptunaHyperparameter optimization framework with pruning and several sampling algorithms. 27SMAC3Open-source Bayesian optimization package for hyperparameter optimization. 28Syne TuneLibrary for large-scale, asynchronous hyperparameter and architecture optimization. 29Ray TuneDistributed hyperparameter tuning library in the Ray ecosystem.

Related Pages

5 entries

Related directories on this site.

30Hyperparameter Optimisation (HPO) ToolsA curated directory of 20 Hyperparameter Optimisation (HPO) Tools — covering Optuna, Hyperopt, KerasTuner, Google Vizier, Ray, NNI, scikit-optimize, FLAML. 31Parameter Optimization TechniquesGuide to parameter optimization in machine learning for 2026: gradient-based optimizers such as Adam and AdamW, learning-rate schedules, hyperparameter search from random search to Hyperband and PBT, surveys and libraries. 32AutoML ToolsDirectory of AutoML tools: open-source libraries such as auto-sklearn, TPOT, AutoGluon, FLAML, Ludwig, PyCaret, AutoKeras and H2O AutoML. 33AI Model Optimization TechniquesAI model optimization techniques for 2026: pruning, quantization, knowledge distillation, LoRA, efficient attention and speculative decoding. 34Machine Learning AlgorithmsResources for learning machine learning algorithms: introductory tours and guides, and the scikit-learn user guide for supervised and unsupervised learning.

Frequently Asked Questions

What is hyperparameter optimization?

Hyperparameter optimization (HPO) is the search for the hyperparameter values, such as learning rate, depth or regularization strength, that give the best validation performance. Each candidate needs a training run, so the cost of evaluation drives the choice of method.

Is random search better than grid search?

Often, yes. Bergstra and Bengio showed that random search is more efficient than grid search when only a few hyperparameters matter, because it tries more distinct values of each one.

When should I use Bayesian optimization?

Use it when each training run is expensive and the number of hyperparameters is modest. A surrogate model such as a Gaussian process, a random forest or a TPE density guides which configuration to try next.

How do Hyperband and ASHA save compute?

They use successive halving: many configurations start with a small budget, such as a few epochs, and only the best continue to larger budgets. ASHA runs this asynchronously so many workers stay busy.

How should I set up an HPO run in practice?

Search the learning rate on a log scale, score candidates on a validation set or by cross-validation, and keep the test set untouched. Spend the budget on the hyperparameters that matter most, since the Tunability study found they differ between algorithms.