Least Squares and Straight Line

Projection

$P=A(A^TA)^{-1}A^T$

• extreme cases
  • If $b$ is in columns space , $Pb=b$
    • $b$ is column combination of $A$
    • $b=Ax$ ⇒ $Pb= A(A^TA)^{-1}A^TAx=Ax=b$
  • If $bㅗ$columns space, $Pb=0$
    • $b$ is in left null space ⇒ $A^Tb=0$

Projection onto left null space

$I-P$ is projection matrix onto left null space

Find the best straight line

line : $b=C+Dt$

data points : (1,1), (2,1) ,(3,1)

$C+D=1$

$C+2D=2$

$C+3D=2$

$Ax=b$

$\begin{bmatrix}1&& 1\\ 1&&2\\1&&3 \end{bmatrix}\begin{bmatrix}C\\D \end{bmatrix}=\begin{bmatrix}1\\2\\2 \end{bmatrix}$

No solutions exist,so minimize $||Ax-b||^2=||e||^2$

1. Take Partial Derivative

   $||C+D-1||^2+||C+2D-2||^2+||C+3D-2||^2$

2. Use Projection Matrix

   $A^TA\hat{x}=A^Tb$

⇒ Both yields same equation

$3C+6D=5$

$6C+14D=11$

• Verifying $p$ and $e$ are perpendicular

$A^TA$ is always invertible?

If $A$ has independent columns, then $A^TA$ is invertible.

Proof ) Suppose $A^TAx=0$

$x$ must be zero if $A^TA$ is invertible.

1. Trick $x^TA^TAx=0=(Ax)^T(Ax)$ ⇒ $Ax=0$ ⇒ $x=0$

2. Columns are definitely independent if they are perpendicular unit vectors(orthonomal vectors).