An optimization problem may be expressed as the maximum value, $f^*$, to be obtained from the objective function, $f$, which is given as a function of a vector of optimization variables, $\bold x$. Optimization must occur across constraints, which may entail
(1) bounds on the optimization variables, $\bold x_{lb}\le\bold x\le\bold x_{ub}$, (2) boundaries on an arbitrary evaluation of the state, $g_i(\bold x)\le0,~i=1,\cdots,m$, (3) equivalency on an arbitrary evaluation of the state, $h_j(\bold x)=0,~j=1,\cdots,p$
$$ f^*=\max~f(\bold x) $$
Alternatively, we may obtain the state which maximizes $f$ by the following notation.
$$ \bold x^*=\arg\max~f(\bold x) $$
An analytical solution would solve $f'(\bold x^)=0$ for $\bold x^$, but this requires a differentiable expression for the objective function, and is difficult to automate.
When $\bold x$ has only one optimization variable, there are a number of optimization methods we may use, similar in structure to root finding methods.
This bracketed optimization method maintains $x_l,x_u$. At each stage, for $x_u-x_l=l_0$ it evaluates two points, at $x_l+l_2$ and $x_l+l_1$, with $l_1,l_2$ such that$l_0=l_1+l_2$ and $l_1/l_0=l_2/l_1$. It then retains point with the lower evaluation so that the local maxima is kept in the interval. This is analogous to the bisection method.
This method may simply be used in the neighborhood of the maxima. Because three points define a unique parabola, and a parabola has a unique maxima, three points may be used to give a direct estimate for $x^*$ based on the $x$ location of the parabola’s vertex.
Applying Newton’s method to the derivative of $f$, a local maxima may be obtained.
$$
x_{i+1}=x_i-\frac{f'(x_i)}{f''(x_i)}~\Rarr~x_{i+1}=x_a-\frac{h(f(x_b)-f(x_a))}{f(x_c)-2f(x_b)+f(x_a)}
$$
Multiple samples of $f$ must be taken to estimate higher derivatives. These must be taken at $x_i,x_i+h,x_i+2h$ for each iteration
$$ x_a<x_b<x_c~~~~~~~~~x_b-x_a=x_c-x_b $$