Least Squares

LA home
Computing
Maths
 Glossary
 Calculus
 Stats
 N χ
 Trig
 Vectors
 Matrices
 Eigen v.
 Ortho.
 Least squares
 Quaternions
 Lagrange mult.
 Least squares

also see
algorithms
 numerical
  matrices
MML
Least squares, e.g., for linear regression.
 
Model
X W ~ Y,
X W + E = Y,
 
x1,1, ..., x1,K
..., ...,  ...
..., ...,  ...
xN,1, ..., xN,K
w1
...
wK
+
e1
...
...
eN
=
y1
...
...
yN
 
N > K, we hope.
 
Problem: Given X and Y, find weights, W, so as to minimise the sum of the squared errors.
 
Errors
e1
...
...
eN
=
y1 - ∑k x1,kwk
...
...
yN - ∑k xN,kwk
 
Squared errors
e12
...
...
eN2
=
y12 - {2y1k x1,kwk} + {∑k x1,kwk}2
...
...
yN2 - {2yNk xN,kwk} + {∑k xN,kwk}2
The sum of the squared errors (a scalar) is S = ∑n en2.
 
Differentiate S wrt wm, 1≤m≤K, and set to zero
d S / d wm
= - 2 {∑n yn xn,m} + 2 {∑n {∑k xn,k wk} xn,m}
= - 2 {∑n xTm,n yn} + 2 {∑n xTm,n {∑k xn,k wk}},     ∀ m = 1, ..., K
= 0,
i.e.,
XT Y = (XT X) W,       where T is transpose,
W = (XT X)-1 XT Y,     if XT X is invertible.
 
(Note that X is not square in general; do not be tempted to write W=X-1Y, but XTX is square with shape K×K.)
www:


© L. Allison   http://www.allisons.org/ll/   (or as otherwise indicated),
Created with "vi (Linux or Solaris)",  charset=iso-8859-1,  fetched Thursday, 21-Sep-2017 01:08:53 EDT.

free: Linux, Ubuntu operating-sys, OpenOffice office-suite, The GIMP ~photoshop,
Firefox web-browser, FlashBlock flash on/off.