Orthogonal

For functions f and g defined on the range [lo..hi], define their inner product to be

<f, g> = ∫_lo..hi f(x) g(x) dx,

and the norm

||f||₂ = <f, f>^1/2     (also written as just ||f||);

 

hence

<c f, g> = <f, c g> = c <f, g>

and

||c f||₂ = c ||f||₂,   where c is a constant.

 

Functions {f_i | i ≥ 0} are orthogonal if

<f_i, f_j> = 0,   ∀ i, j s.t. i≠j

and are orthonormal if also

||f_i|| = 1,   ∀ i.

 

Note that

∫_-1..+1 f(x) dx = ¹/_a ∫_-a..+a f(y/a) dy

and that

∫_-a..+a f(x) dx = ∫_-a+b..a+b f(y-b) dy

so one can scale and shift a set of orthogonal functions to another range, [lo, hi], and normalise them (individually).

 

Series of Functions

If a function f(x) has an expansion

f(x) = a₀f₀(x) + a₁f₁(x) + ...

over [lo, hi] in terms of orthogonal basis functions, {f₀(x), f₁(x), ...}, then

<f, f_i> = ∫_[lo..hi] f(x) f_i(x) dx

= ∫_[lo..hi] a₀ f₀(x) f_i(x) + a₁ f₁(x) f_i(x) + ... dx

= a₀.<f₀,f_i> + a₁.<f₁,f_i> + ...

= a₀ . 0 + ... + a_i-1 . 0 + a_i . ||f_i||² + a_i+i . 0 + ...

= a_i . ||f_i||²

so

a_i = <f, f_i> / ||f_i||²

If the basis functions are orthonormal then a_i = <f, f_i>.

 

Interpolation

Suppose we are given "training" data points {(x₁, y₁), (x₂, y₂), ..., (x_N, y_N)}, where the x_i ∈ [lo, hi], and want to find a function, f(x), that is a "good" fit to the data, that is each f(x_i) is close to its corresponding y_i. A common approach is to try to find some function, f(), expressed as a series in terms of a set of "simple" functions (orthonormal basis functions have advantages), which amounts to solving the resulting linear regression problem to minimise the sum of the squared errors, ∑_i (y_i - f(x_i))².

A random example computed by JavaScript (if it is on) ...