If the basis functions are orthonormalthen ai = <f, fi>.
Suppose we are given "training" data points
(x2, y2), ...,
where the xi ∈ [lo, hi], and
want to find a function, f(x), that is a "good" fit to the data,
that is each f(xi) is close to its corresponding yi.
A common approach is to try to find some function, f(), expressed
as a series in terms of a set of "simple" functions
(orthonormal basis functions have advantages), which
amounts to solving the resulting linear regression problem
to minimise the sum of the
squared errors,∑i (yi - f(xi))2.
(A polynomial can
only approximate a sawtooth.)
A high order approximation gives a smaller RMS error than a
low order one in general.
This of course opens the well known over-fitting trap;
showed how to stop before falling into it
(in the real world there is no looking at
<true_f, est_f> because invariably
true_f is unknown in a real inference problem).
The Legendre polynomials are orthogonal polynomials on [-1, 1]: