If we have imperfect data but a hypothesis for the relationship the data should model, regression is the method with which we can resolve a curve form our data. If are very confident in each individual measurement but don’t know the form the curve should take, (5) Interpolation is the method that should be used.
Linear regression is not restricted to fitting a straight line to a data set of $(x_k,y_k)$. Rather, it may be used to fit any relationship $f_k=a_0f_0(x)+\cdots+a_nf_n(x)$ to a corresponding dataset of $y_k$. We will primarily consider linear regressions of polynomial forms.
Linear regression minimizes the sum of squared error. This may be solved analytically as other optimization problems, finding where $S_r'=0$.
$$ S_r=\Sigma(y_k-f_k)^2~~~\Rarr~~~\frac{\partial S_r}{\partial a_1}=0~~~\cdots~~~\frac{\partial S_r}{\partial a_n}=0 $$
We may also solve the regression by linear algebra for any model in the following form.
$$ g(y)=a_1f_1(x)+\cdots+a_mf_m(x) $$
We will use a column vector of the transformed $y$ dataset as well as a column vector of our regression variables.
$$
\bold y=[~g(y_1)\cdotsg(y_n)~]^T~~~~~\bold a=[~a_1~~\cdots~~a_m~]^T
$$
We will also use our dataset of $x$ as transformed by our model functions.
$$ \bold Z=\begin{bmatrix} f_1(x_1)&\cdots&f_m(x_1)\\ \cdots&\cdots&\cdots\\ f_1(x_n)&\cdots&f_m(x_n) \end{bmatrix} $$
Finally, we may solve for our coefficient matrix.
$$ \bold{Za}=\bold y~~~\Rarr~~~\bold Z^T\bold{Za}=\bold Z^T\bold y~~~\Rarr~~~\bold a=(\bold Z^T\bold Z)^{-1}\bold Z^T\bold y $$
Applying this technique to a polynomial regression, the process is as follows.
$$
\bold y=[~y_1~~\cdotsy_n~]^T~~~\bold a=[~a_1~~\cdots~~a_m~]^T
$$
$$ \bold Z_p=\begin{bmatrix} 1&\cdots&x_1^m\\ \cdots&\cdots&\cdots\\ 1&\cdots&x_n^m \end{bmatrix} $$
$$ \bold a=(\bold Z^T\bold Z)^{-1}\bold Z^T\bold y $$