Data Types & Structures

The following diagram showcases the type hierarchy of all BOSS inputs and hyperparameters.

This reminder of this page contains documentation for all exported types and structures.

Problem Definition

The BossProblem structure contains the whole problem definition, the model definition, and the data together with the current parameter and hyperparameter values.

BOSS.BossProblem — Type

BossProblem(; kwargs...)

Defines the whole optimization problem for the BOSS algorithm.

Problem Definition

There is some (noisy) blackbox function `y = f(x) = f_true(x) + ϵ` where `ϵ ~ Normal`.

We have some surrogate model `y = model(x) ≈ f_true(x)`
describing our knowledge (or lack of it) about the blackbox function.

We wish to find `x ∈ domain` such that `fitness(f(x))` is maximized
while satisfying the constraints `f(x) <= y_max`.

Keywords

fitness::Fitness: The Fitness function.
f::Union{Function, Missing}: The objective blackbox function.
model::SurrogateModel: The SurrogateModel.
data::ExperimentData: The initial data of objective function evaluations. See ExperimentDataPrior.
domain::Domain: The Domain of x.
y_max::AbstractVector{<:Real}: The constraints on y. (See the definition above.)

Fitness

The Fitness type is used to define the fitness function $\text{fit}(y) \rightarrow \mathbb{R}$.

The NoFitness can be used in problems without defined fitness (such as active learning problems). It is the default option used if no fitness is provided to BossProblem. The NoFitness can only be used with AcquisitionFunction that does not require fitness.

The LinFitness can be used to define a simple linear fitness function

\[\text{fit}(y) = \alpha^T y \;.\]

Using LinFitness instead of NonlinFitness may allow for simpler/faster computation of some acquisition functions.

The NonlinFitness can be used to define an arbitrary fitness function

\[\text{fit}(y) \rightarrow \mathbb{R} \;.\]

BOSS.Fitness — Type

An abstract type for a fitness function measuring the quality of an output y of the objective function.

Fitness is used by the AcquisitionFunction to determine promising points for future evaluations.

All fitness functions should implement:

(::CustomFitness)(y::AbstractVector{<:Real}) -> fitness::Real

An exception is the NoFitness, which can be used for problem without a well defined fitness. In such case, an AcquisitionFunction which does not depend on Fitness must be used.

source

BOSS.NoFitness — Type

NoFitness()

Placeholder for problems with no defined fitness.

BossProblem defined with NoFitness can only be solved with AcquisitionFunction not dependent on Fitness.

source

BOSS.LinFitness — Type

LinFitness(coefs::AbstractVector{<:Real})

Used to define a linear fitness function measuring the quality of an output y of the objective function.

May provide better performance than the more general NonlinFitness as some acquisition functions can be calculated analytically with linear fitness functions whereas this may not be possible with a nonlinear fitness function.

Input Domain

The Domain structure is used to define the input domain $x \in \text{Domain}$. The domain is formalized as

\[\begin{aligned} & lb < x < ub \\ & d_i \implies (x_i \in \mathbb{Z}) \\ & \text{cons}(x) > 0 \;. \end{aligned}\]

BOSS.Domain — Type

Domain(; kwargs...)

Describes the optimization domain.

Keywords

bounds::AbstractBounds: The basic box-constraints on x. This field is mandatory.
discrete::AbstractVector{<:Bool}: Can be used to designate some dimensions of the domain as discrete.
cons::Union{Nothing, Function}: Used to define arbitrary nonlinear constraints on x. Feasible points x must satisfy all(cons(x) .> 0.). An appropriate acquisition maximizer which can handle nonlinear constraints must be used if cons is provided. (See AcquisitionMaximizer.)

source

BOSS.AbstractBounds — Type

const AbstractBounds = Tuple{<:AbstractVector{<:Real}, <:AbstractVector{<:Real}}

Defines box constraints.

Example: ([0, 0], [1, 1]) isa AbstractBounds

source

Output Constraints

Constraints on output vector y can be defined using the y_max field. Providing y_max to BossProblem defines the linear constraints y < y_max.

Arbitrary nonlinear constraints can be defined by augmenting the objective function. For example to define the constraint y[1] * y[2] < c, one can define an augmented objective function

function f_c(x)
    y = f(x)  # the original objective function
    y_c = [y..., y[1] * y[2]]
    return y_c
end

and use

y_max = [fill(Inf, y_dim)..., c]

where y_dim is the output dimension of the original objective function. Note that defining nonlinear constraints this way increases the output dimension of the objective function and thus the model definition has to be modified accordingly.

Surrogate Model

The surrogate model is defined using the SurrogateModel type.

BOSS.SurrogateModel — Type

An abstract type for a surrogate model approximating the objective function.

Example usage: struct CustomModel <: SurrogateModel ... end

All models should implement:

make_discrete(model::CustomModel, discrete::AbstractVector{<:Bool}) -> discrete_model::CustomModel
model_posterior(model::CustomModel, data::ExperimentDataMAP) -> (x -> mean, std)
model_loglike(model::CustomModel, data::ExperimentData) -> (::ModelParams -> ::Real)
sample_params(model::CustomModel) -> ::ModelParams
param_priors(model::CustomModel) -> ::ParamPriors

Models may implement:

sliceable(::CustomModel) -> ::Bool
model_posterior_slice(model::CustomModel, data::ExperimentDataMAP, slice::Int) -> (x -> mean, std)

If sliceable(::CustomModel) == true, then the model should additionally implement:

model_loglike_slice(model::SliceableModel, data::ExperimentData, slice::Int) -> (::ModelParams -> ::Real)
θ_slice(model::SliceableModel, idx::Int) -> Union{Nothing, UnitRange{<:Int}}

source

The LinModel and NonlinModel structures are used to define parametric models. (Some compuatations are simpler/faster with linear model, so the LinModel might provide better performance in the future. This functionality is not implemented yet, so using the NonlinModel is equiavalent for now.)

BOSS.Parametric — Type

An abstract type for parametric surrogate models.

Parameters & Hyperparameters

The BOSS.ModelParams and BOSS.ParamPriors type aliases are used throughout the package to pass around model (hyper)parameters and their priors. These types are only important for advanced usage of BOSS.jl. (E.g. implementing custom surrogate models.)

BOSS.ModelParams — Type

const ModelParams = Tuple{
    <:Theta,
    <:LengthScales,
    <:Amplitudes,
    <:NoiseStd,
}

Represents all model (hyper)parameters.

Example:

params = (nothing, [1.;π;; 1.;π;;], [1., 1.5], [0.1, 1.])
params isa BOSS.ModelParams

θ, λ, α, noise = params
θ isa BOSS.Theta
λ isa BOSS.LengthScales
α isa BOSS.Amplitudes
noise isa BOSS.NoiseStd

See: Theta, LengthScales, Amplitudes, NoiseStd

source

BOSS.Theta — Type

const Theta = Union{Nothing, AbstractVector{<:Real}}

Parameters of the parametric model, or nothing if the model is nonparametric.

Example: [1., 2., 3.] isa Theta

source

BOSS.LengthScales — Type

const LengthScales = Union{Nothing, <:AbstractMatrix{<:Real}}

Length scales of the GP as a x_dim×y_dim matrix, or nothing if the model is purely parametric.

Example: [1.;5.;; 1.;5.;;] isa LengthScales

source

BOSS.Amplitudes — Type

const Amplitudes = Union{Nothing, <:AbstractVector{<:Real}}

Amplitudes of the GP, or nothing if the model is purely parametric.

Example: [1., 5.] isa Amplitudes

source

BOSS.NoiseStd — Type

const NoiseStd = AbstractVector{<:Real}

Noise standard deviations of each y dimension.

Example: [0.1, 1.] isa NoiseStd

source

BOSS.ParamPriors — Type

const ParamPriors = Tuple{
    <:ThetaPriors,
    <:LengthScalePriors,
    <:AmplitudePriors,
    <:NoiseStdPriors,
}

Represents all (hyper)parameter priors.

See: ThetaPriors, LengthScalePriors, AmplitudePriors, NoiseStdPriors

source

BOSS.ThetaPriors — Type

const ThetaPriors = Union{Nothing, AbstractVector{<:UnivariateDistribution}}

Prior of Theta.

source

BOSS.LengthScalePriors — Type

const LengthScalePriors = Union{Nothing, <:AbstractVector{<:MultivariateDistribution}}

Prior of LengthScales.

source

BOSS.AmplitudePriors — Type

const AmplitudePriors = Union{Nothing, <:AbstractVector{<:UnivariateDistribution}}

Prior of Amplitudes.

source

BOSS.NoiseStdPriors — Type

const NoiseStdPriors = AbstractVector{<:UnivariateDistribution}

Prior of NoiseStd.

source

All (hyper)parameter priors are defined as a part of the surrogate model definition. All surrogate models share the noise_std_priors field, but other priors may be missing depending on the particular model.

Experiment Data

The data from all past objective function evaluations as well as estimated parameter and/or hyperparameter values of the surrogate model are stored in the ExperimentData types.

BOSS.ExperimentData — Type

Stores all the data collected during the optimization as well as the parameters and hyperparameters of the model.

source

The ExperimentDataPrior structure is used to pass the initial dataset to the BossProblem.

BOSS.ExperimentDataPrior — Type

Stores the initial data.

At least one initial datapoint has to be provided (purely for implementation reasons). One can for example use LatinHypercubeSampling.jl to obtain a small intial grid, or provide a single random initial datapoint.

Fields

X::AbstractMatrix{<:Real}: Contains the objective function inputs as columns.
Y::AbstractMatrix{<:Real}: Contains the objective function outputs as columns.

Model Fitter

The ModelFitter type defines the algorithm used to estimate the model (hyper)parameters.

BOSS.ModelFitter — Type

Specifies the library/algorithm used for model parameter estimation. Inherit this type to define a custom model-fitting algorithms.

Example: struct CustomFitter <: ModelFitter{MAP} ... end or struct CustomFitter <: ModelFitter{BI} ... end

All model fitters should implement: estimate_parameters(model_fitter::CustomFitter, problem::BossProblem; info::Bool).

This method should return a tuple (params, val). The returned params should be a ModelParams (if CustomAlg <: ModelFitter{MAP}) or a AbstractVector{<:ModelParams} (if CustomAlg <: ModelFitter{BI}). The returned val should be the log likelihood of the parameters (if CustomAlg <: ModelFitter{MAP}), or a vector of log likelihoods of the individual parameter samples (if CustomAlg <: ModelFitter{BI}), or nothing.

Acquisition Maximizer

The AcquisitionMaximizer type is used to define the algorithm used to maximize the acquisition function.

BOSS.AcquisitionMaximizer — Type

Specifies the library/algorithm used for acquisition function optimization. Inherit this type to define a custom acquisition maximizer.

Example: struct CustomAlg <: AcquisitionMaximizer ... end

All acquisition maximizers should implement: maximize_acquisition(acq_maximizer::CustomAlg, acq::AcquisitionFunction, problem::BossProblem, options::BossOptions).

This method should return a tuple (x, val). The returned x is the point of the input domain which maximizes the given acquisition function acq (as a vector), or a batch of points (as a column-wise matrix). The returned val is the acquisition value acq(x), or the values acq.(eachcol(x)) for each point of the batch, or nothing (depending on the acquisition maximizer implementation).

Acquisition Function

The acquisition function is defined using the AcquisitionFunction type.

BOSS.AcquisitionFunction — Type

Specifies the acquisition function describing the "quality" of a potential next evaluation point. Inherit this type to define a custom acquisition function.

Example: struct CustomAcq <: AcquisitionFunction ... end

All acquisition functions should implement: (acquisition::CustomAcq)(problem::BossProblem, options::BossOptions)

This method should return a function acq(x::AbstractVector{<:Real}) = val::Real, which is maximized to select the next evaluation function of blackbox function in each iteration.

Termination Conditions

The TermCond type is used to define the termination condition of the BO procedure.

BOSS.TermCond — Type

Specifies the termination condition of the whole BOSS algorithm. Inherit this type to define a custom termination condition.

Example: struct CustomCond <: TermCond ... end

All termination conditions should implement: (cond::CustomCond)(problem::BossProblem)

This method should return true to keep the optimization running and return false once the optimization is to be terminated.

Miscellaneous

The BossOptions structure is used to define miscellaneous hyperparameters of the BOSS.jl package.

BOSS.BossOptions — Type

BossOptions(; kwargs...)

Stores miscellaneous settings of the BOSS algorithm.

Keywords

info::Bool: Setting info=false silences the BOSS algorithm.
debug::Bool: Set debug=true to print stactraces of caught optimization errors.
parallel_evals::Symbol: Possible values: :serial, :parallel, :distributed. Defaults to :parallel. Determines whether to run multiple objective function evaluations within one batch in serial, parallel, or distributed fashion. (Only has an effect if batching AM is used.)
callback::BossCallback: If provided, callback(::BossProblem; kwargs...) will be called before the BO procedure starts and after every iteration.

References

[1] Bobak Shahriari et al. “Taking the human out of the loop: A review of Bayesian optimization”. In: Proceedings of the IEEE 104.1 (2015), pp. 148–175