Hello.
Since September 2020, I'm a PhD student at the Wallenberg AI, Autonomous Systems and Software Program
and the University of Lund Sweden.
I'm working on the optimization of black-box functions with Bayesian Optimization with a focus on
high-dimensional functions with hundreds on input parameters.
I'm interested in exploring the relevance of black-box optimization and
developing scalable and reliable algorithms for black-box optimization and applying them to real-world problems.
Research Output.
Recent work reported that simple Bayesian optimization methods perform well for high-dimensional real-world tasks, seemingly contradicting prior work and tribal knowledge.
This paper investigates the 'why'.
We identify fundamental challenges that arise in high-dimensional Bayesian optimization and explain why recent methods succeed. Our analysis shows that vanishing gradients caused by Gaussian process initialization schemes play a major role in the failures of high-dimensional Bayesian optimization and that methods that promote local search behaviors are better suited for the task.
We find that maximum likelihood estimation of Gaussian process length scales suffices for state-of-the-art performance. Based on this, we propose a simple variant of maximum likelihood estimation called MSR that leverages these findings to achieve state-of-the-art performance on a comprehensive set of real-world applications.
We also present targeted experiments to illustrate and confirm our findings.
A well-balanced exploration-exploitation trade-off is crucial for successful acquisition functions in Bayesian optimization.
However, there is a lack of quantitative measures for exploration, making it difficult to analyze and compare different acquisition functions.
This work introduces two novel approaches - observation traveling salesman distance and observation entropy - to quantify the exploration characteristics of acquisition functions based on their selected observations.
Using these measures, we examine the explorative nature of several well-known acquisition functions across a diverse set of black-box problems, uncover links between exploration and empirical performance, and reveal new relationships among existing acquisition functions.
Beyond enabling a deeper understanding of acquisition functions, these measures also provide a foundation for guiding their design in a more principled and systematic manner.
Bayesian optimization is a widely used method for optimizing expensive black-box functions, with Expected Improvement being one of the most commonly used acquisition functions.
In contrast, information-theoretic acquisition functions aim to reduce uncertainty about the function's optimum and are often considered fundamentally distinct from EI. In this work, we challenge this prevailing perspective by introducing a unified theoretical framework, Variational Entropy Search, which reveals that EI and information-theoretic acquisition functions are more closely related than previously recognized.
We demonstrate that EI can be interpreted as a variational inference approximation of the popular information-theoretic acquisition function, named Max-value Entropy Search. Building on this insight, we propose VES-Gamma, a novel acquisition function that balances the strengths of EI and MES.
Extensive empirical evaluations across both low- and high-dimensional synthetic and real-world benchmarks demonstrate that VES-Gamma is competitive with state-of-the-art acquisition functions and in many cases outperforms EI and MES.
Impactful applications such as materials discovery, hardware design, neural architecture search, or
portfolio optimization require optimizing high-dimensional black-box functions with mixed and combinatorial
input spaces.
While Bayesian optimization has recently made significant progress in solving such problems, an in-depth
analysis reveals that the current state-of-the-art methods are not reliable.
Their performances degrade substantially when the unknown optima of the function do not have a certain
structure.
To fill the need for a reliable algorithm for combinatorial and mixed spaces, this paper proposes Bounce
that relies on a novel map of various variable types into nested embeddings of increasing dimensionality.
Comprehensive experiments show that Bounce reliably achieves and often even improves upon state-of-the-art
performance on a variety of high-dimensional problems.
Bayesian optimization is an effective method for optimizing expensive-to-evaluate black-box functions.
High-dimensional problems are particularly challenging as the surrogate model of the objective suffers from
the
curse of dimensionality, which makes accurate modeling difficult.
We propose a group testing approach to identify active variables to facilitate efficient optimization in
these
domains.
The proposed algorithm, Group Testing Bayesian Optimization (GTBO), first runs a testing phase where groups
of variables are systematically selected and tested on whether they influence the objective.
To that end, we extend the well-established theory of group testing to functions of continuous ranges.
In the second phase, GTBO guides optimization by placing more importance on the active dimensions.
By exploiting the axis-aligned subspace assumption, GTBO is competitive against state-of-the-art methods on
several synthetic and real-world high-dimensional optimization tasks.
Furthermore, GTBO aids in the discovery of active parameters in applications, thereby enhancing
practitioners'
understanding of the problem at hand.
We present an algorithm for high-dimensional Bayesian Optimization using nested random embeddings
(BAxUS).
BAxUS starts the optimization in a very low-dimensional sparse embedding (typically only 1-3
dimensions)
and
increases the embedding as it optimizes.
Using the sparse embedding, we can keep previous observations when increasing the embedding
throughout
the
optimization.