Matrix Calculus

Derivative of a scalar wrt matrix

Let \(\nabla_X f\) be the matrix;

\[(\nabla_X f)_{ij} = \frac{\partial f}{\partial X_{ij}}\]

and

\[\frac{\partial f}{\partial X}=(\nabla_X f)^T\]

so

\[(\frac{\partial f}{\partial X})_{ij}=\frac{\partial f}{\partial X_{ji}}\]

\(\nabla_X f\) is commonly called the derivative in denominator layout and \(\frac{\partial f}{\partial X}\) is the derivative in numerator layout.

For a function \(f:\mathbb{R}^n \to \mathbb{R}\) the differential is;

\[df=<\nabla_x f, h>=(\nabla_x f)^T h\]

for \(f:\mathbb{R}^{m\times n}\to \mathbb{R}\)

the differential is;

\[df=<\nabla_X f, H>=\mathop{\mathrm{Tr}}((\nabla_X f)^T H) = \mathop{\mathrm{Tr}}(\frac{\partial f}{\partial X}H)\]

i.e. it is the inner product of the gradient and the matrix \(H\) using the standard inner product for matrices namely the Frobenius inner product of the gradient and H.

\[df=<\nabla_X f, H>=\mathop{\mathrm{Tr}}((\nabla_X f)^T H) = \mathop{\mathrm{Tr}}(\frac{\partial f}{\partial X}H)=\sum_{i=1}^n\sum_{j=1}^m \frac{\partial f}{\partial X_{ij}}H_{ij}\]

The derivative is the linear transformation;

\[L(H)=\sum_{i=1}^n\sum_{j=1}^m \frac{\partial f}{\partial X_{ij}}H_{ij}=\mathop{\mathrm{Tr}}((\nabla_X f)^T H)\]

where \(H\) is a matrix of the same shape as \(X\) just as \(dx\) is a column vector like \(x\) and

\[\frac{\partial f}{\partial X}\]

is a matrix with the same shape as \(X^T\)

as

\[\frac{\partial f}{\partial x}\]

is the transpose of the gradient and has the same shape as \(x^T\)

\[\lim_{H \to 0} \frac{1}{|H|}[f(X+H)-f(X)-\mathop{\mathrm{Tr}}(\frac{\partial f}{\partial X}H)]=0\]

Row-Column Rule

The row-column rule for matrix multiplication can be written as:

\[\begin{aligned} (AB)_{ij} = \sum_k A_{ik} B_{kj} \\ \end{aligned}\]

We can apply the row-column rule twice to find an expression for the \((i,j)^{th}\) element of a product of three matrices:

\[\begin{aligned} (ABC)_{ij} &= \sum_m (AB)_{im} C_{mj} \\ &= \sum_m \sum_k A_{ik} B_{km} C_{mj} \\ &= \sum_k \sum_m A_{ik} B_{km} C_{mj} \\ \end{aligned}\]

Similarly for a product of four matrices:

\[\begin{aligned} (ABCD)_{ij} = \sum_k \sum_m \sum_u A_{ik} B_{km} C_{mu} D_{uj} \end{aligned}\]

and so on.

The following rules for expressing the \(i^{th}\) element of a matrix multiplied by a column vector or a row vector multiplied by a matrix can sometimes be useful.

Matrix times column vector rule:

\[\begin{aligned} (Ax)_i = \sum_k A_{ik} x_k \\ \end{aligned}\]

Row vector times matrix rule:

\[\begin{aligned} (x^T A)_i = \sum_k A_{ki} x_k \\ \end{aligned}\]

Traces

We can use the row column rule to write expressions for traces of the product of several matrices in terms of summation notation.

Trace of product of two matrices:

\[\begin{aligned} \mathop{\mathrm{Tr}}(AB) &= \sum_i (AB)_{ii} \\ &= \sum_i \sum_k A_{ik} B_{ki} \\ \end{aligned}\]

Trace of product of three matrices:

\[\begin{aligned} \mathop{\mathrm{Tr}}(ABC) &= \sum_i (ABC)_{ii} \\ &= \sum_i \sum_k (AB)_{ik} C_{ki} \\ &= \sum_i \sum_k \sum_m A_{im} B_{mk} C_{ki} \\ \end{aligned}\]

Trace of product of four matrices:

\[\begin{aligned} \mathop{\mathrm{Tr}}(ABCD) &= \sum_i (ABCD)_{ii} \\ &= \sum_i \sum_k (ABC)_{ik} D_{ki} \\ &= \sum_i \sum_k \sum_m (AB)_{im}C_{mk} D_{ki} \\ &= \sum_i \sum_k \sum_m \sum_u A_{iu} B_{um} C_{mk} D_{ki} \\ \end{aligned}\]

Other useful identities

We can express the quadratic form \(x^T A x\) as a trace:

\[\begin{aligned} x^TAx = \mathop{\mathrm{Tr}}\{ Axx^T \} \\ \end{aligned}\]

This is useful because quadratic forms such as this come up a lot in the exponent of multivariate Gaussians which are used extensively in statistical machine learning.

To prove this we use the row-column rule and the expression for the element of the outer product from earlier \((uv^T)_{ij}=u_i v_j\);

\[\begin{aligned} \mathop{\mathrm{Tr}}(Axx^T) &= \sum_i (Axx^T)_{ii} \\ &= \sum_i \sum_k A_{ik} (xx^T)_{ki} \\ &= \sum_i \sum_k A_{ik} x_k x_i \\ \end{aligned}\]

\(x^T Ax\) is the dot product of the vectors \(x\) and \(Ax\) so we can write;

\[x^T Ax = \sum_i x_i (Ax)_i\]

Then we use \((Ax)_i = \sum_k A_{ik} x_k\) for the matrix vector product from earlier;

Therefore;

\[\begin{aligned} x^T A x &= \sum_i x_i (Ax)_i \\ &= \sum_i x_i \sum_k A_{ik} x_k \\ &= \sum_i \sum_k x_i A_{ik} x_k \\ &= \mathop{\mathrm{Tr}}(Axx^T) \end{aligned}\]

We can also express the squared Euclidean norm of a vector as a trace (use \(x^TAx=\mathop{\mathrm{Tr}}(Axx^T)\) with \(A\) equal to the identity):

\[\begin{aligned} \lVert x \rVert ^2 = x^Tx = \mathop{\mathrm{Tr}}\{ xx^T \} \\ \end{aligned}\]

We can express the quadratic form \(x^T A x\) in summation notation as follows:

\[\begin{aligned} x^TAx = \sum_i \sum_j x_i A_{ij} x_j \end{aligned}\]

We can find the \((i,j)^{th}\) element of the outer product \(uv^T\) as follows;

\[\begin{aligned} (uv^T)_{ij}=u_i v_j \end{aligned}\]

Trace of \(A^TA\)

\[\begin{aligned} \mathop{\mathrm{Tr}}(A^TA) = \sum A_{ij}^2 \\ \end{aligned}\]

Differentials

Definition of differential

\[\begin{aligned} \mathop{\mathrm{d}}f = f'(x) \mathop{\mathrm{d}}x \\ \end{aligned}\]

Sum Rule for Differentials

\[\begin{aligned} \mathop{\mathrm{d}}(f+g) &= (f+g)'(x) \mathop{\mathrm{d}}x \\ &= f'(x) \mathop{\mathrm{d}}x + g'(x) \mathop{\mathrm{d}}x \\ &= \mathop{\mathrm{d}}f + \mathop{\mathrm{d}}g \\ \end{aligned}\]

Constant Multiple Rule for Differentials

\[\begin{aligned} \mathop{\mathrm{d}}(cf) &= (cf)'(x) \mathop{\mathrm{d}}x \\ &= cf'(x) \mathop{\mathrm{d}}x \\ &= c (\mathop{\mathrm{d}}f) \\ \end{aligned}\]

Product Rule for Differentials

\[\begin{aligned} \mathop{\mathrm{d}}(fg) &= (fg)'(x) \mathop{\mathrm{d}}x \\ &= (f'(x)g(x)+f(x)g'(x)) \mathop{\mathrm{d}}x \\ &= f'(x)g(x) \mathop{\mathrm{d}}x + f(x) g'(x) \mathop{\mathrm{d}}x \\ &= (f'(x)\mathop{\mathrm{d}}x) g(x) + f(x) (g'(x) \mathop{\mathrm{d}}x) \\ &= (\mathop{\mathrm{d}}f) g + f (\mathop{\mathrm{d}}g) \end{aligned}\]

Differential of constant

\[\begin{aligned} \mathop{\mathrm{d}}c &= (c)' \mathop{\mathrm{d}}x \\ &= 0 \mathop{\mathrm{d}}x \\ &= 0 \\ \end{aligned}\]

Vector Differentials

If \(a\) is a constant vector;

\[\begin{aligned} \mathop{\mathrm{d}}a=0 \\ \mathop{\mathrm{d}}(x^T)=(\mathop{\mathrm{d}}x)^T \\ \mathop{\mathrm{d}}(a^T x) = a^T \mathop{\mathrm{d}}x \\ \mathop{\mathrm{d}}(x^T a) = (\mathop{\mathrm{d}}x^T) a=(\mathop{\mathrm{d}}x)^T a \\ \mathop{\mathrm{d}}(x+y) = \mathop{\mathrm{d}}x + \mathop{\mathrm{d}}y \\ \mathop{\mathrm{d}}(x-y) = \mathop{\mathrm{d}}x - \mathop{\mathrm{d}}y \\ \mathop{\mathrm{d}}(x^Ty) = (\mathop{\mathrm{d}}x^T)y + x^T (\mathop{\mathrm{d}}y)=(\mathop{\mathrm{d}}x)^Ty+x^T (\mathop{\mathrm{d}}y)\\ \end{aligned}\]

Matrix Differentials

If \(A\) is a constant matrix;

\[\begin{aligned} \mathop{\mathrm{d}}A = 0 \\ \mathop{\mathrm{d}}cX = c \mathop{\mathrm{d}}X \\ \mathop{\mathrm{d}}(X^T) = (\mathop{\mathrm{d}}X)^T \\ \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}(X) = \mathop{\mathrm{Tr}}\mathop{\mathrm{d}}X \\ \mathop{\mathrm{d}}(X+Y) = \mathop{\mathrm{d}}X + \mathop{\mathrm{d}}Y \\ \mathop{\mathrm{d}}(X-Y) = \mathop{\mathrm{d}}X - \mathop{\mathrm{d}}Y \\ \mathop{\mathrm{d}}(XY) = (\mathop{\mathrm{d}}X)Y + X (\mathop{\mathrm{d}}Y) \\ \mathop{\mathrm{d}}|X| = |X| \mathop{\mathrm{Tr}}\{ X^{-1} \mathop{\mathrm{d}}X \} \\ \mathop{\mathrm{d}}\log |X| = \frac{1}{|X|}(\mathop{\mathrm{d}}|X|) = \mathop{\mathrm{Tr}}\{ X^{-1} \mathop{\mathrm{d}}X \} \\ \mathop{\mathrm{d}}X^{-1} = - X^{-1} (\mathop{\mathrm{d}}X) X^{-1} \\ \end{aligned}\]

Finding the Hessian (Second Identification Theorem)

\[\mathop{\mathrm{d}}^2 \phi = (\mathop{\mathrm{d}}x)^T B(x) \mathop{\mathrm{d}}x \iff H\phi = \frac{B(x)+B(x)^T}{2}\]

Conversion between differential and derivatives

We can use the following rules to convert between differentials and derivatives;

\[\begin{aligned} \mathop{\mathrm{d}}y = a \mathop{\mathrm{d}}x \iff \frac{dy}{dx}=a \\ \mathop{\mathrm{d}}y =\vec{a}^T \mathop{\mathrm{d}}\vec{x} \iff \frac{dy}{d\vec{x}}=\vec{a}^T \\ \mathop{\mathrm{d}}y = \mathop{\mathrm{Tr}}\{ A \mathop{\mathrm{d}}X\} \iff \frac{\partial y}{\partial X}=A \\ \mathop{\mathrm{d}}\vec{y} = \vec{a} \mathop{\mathrm{d}}x \iff \frac{ d \vec{y}}{dx}=\vec{a} \\ \mathop{\mathrm{d}}\vec{y} = A \mathop{\mathrm{d}}\vec{x} \iff \frac{d \vec{y}}{d \vec{x}}=A \\ \mathop{\mathrm{d}}Y = A \mathop{\mathrm{d}}x \iff \frac{d Y}{dx}=A \\ \end{aligned}\]

Numerator and Denominator Layout and Indices

If

\[\frac{\partial f(X)}{\partial X_{ij}}= A_{ij}\]

then in denominator layout

\[\frac{\partial f(X)}{\partial X} = A\]

or in numerator layout;

\[\frac{\partial f(X)}{\partial X} = A^T\]

If

\[\frac{\partial f(X)}{\partial X_{ij}}= A_{ji}\]

then in numerator layout

\[\frac{\partial f(X)}{\partial X} = A\]

or in denominator layout

\[\frac{\partial f(X)}{\partial X} = A^T\]

Examples

Example 1\(\frac{\partial x^Ta}{\partial x}=a^T\)

\[\begin{aligned} \mathop{\mathrm{d}}(x^Ta) &= \mathop{\mathrm{d}}(a^T x) \\ &= a^T \mathop{\mathrm{d}}x \\ \end{aligned}\]

Therefore

\[\frac{\partial x^Ta}{\partial x}=a^T\]

Example 2\(\frac{\partial x^Ta}{\partial x}=a^T\)

\[f(x)=x^Ta\]

\[\begin{aligned}f(x+h)-f(x)&=(x+h)^Ta-x^Ta\\&=x^Ta+h^Ta-x^Ta\\&=h^Ta\\&=a^Th \\&= a^Th+o(h) \end{aligned}\]

\[\frac{\partial x^Ta}{\partial x}=a^T\]

Example 3\(\frac{\partial \|x\|^2 }{\partial x}=\frac{\partial x^Tx}{\partial x}=2x^T\)

\[\begin{aligned} \mathop{\mathrm{d}}(x^Tx) &= (\mathop{\mathrm{d}}x^T)x + x^T (\mathop{\mathrm{d}}x) \\ &= (\mathop{\mathrm{d}}x)^T x + x^T (\mathop{\mathrm{d}}x) \\ &= 2x^T (\mathop{\mathrm{d}}x) \end{aligned}\]

Therefore

\[\frac{\partial \|x\|^2 }{\partial x}=\frac{\partial x^Tx}{\partial x}=2x^T\]

Example 4\(\frac{\partial \|x\|^2 }{\partial x}=\frac{\partial x^Tx}{\partial x}=2x^T\)

\[f(x)=x^Tx\]

\[\begin{aligned}f(x+h)-f(x)&=(x+h)^T(x+h)-x^Tx \\ &= x^Tx+x^Th+h^Tx+h^Th-x^Tx\\&= x^Th+h^Tx+h^Th\\&= 2x^Th+h^Th\\&=2x^Th+o(h) \end{aligned}\]

Therefore

\[\frac{\partial \|x\|^2 }{\partial x}=\frac{\partial x^Tx}{\partial x}=2x^T\]

Example 5\(\frac{\partial b^T Ax}{\partial x}=b^TA\)

\[\begin{aligned} \mathop{\mathrm{d}}(b^T A x) &= b^T A \mathop{\mathrm{d}}x \\ \end{aligned}\]

Therefore

\[\frac{\partial b^T Ax}{\partial x}=b^TA\]

Example 6\(\frac{\partial b^T Ax}{\partial x}=b^TA\)

\[f(x)=b^TAx\]

\[\begin{aligned}f(x+h)-f(x)&= b^TA(x+h)-b^TAx \\ &= b^TAh \\ &= b^TAh+o(h)\end{aligned}\]

Therefore

\[\frac{\partial b^T Ax}{\partial x}=b^TA\]

Example 7\(\frac{\partial Ax}{\partial x}=A\)

\[\begin{aligned} \mathop{\mathrm{d}}(Ax) = A \mathop{\mathrm{d}}x \\ \end{aligned}\]

Therefore \[\frac{\partial Ax}{\partial x}=A\]

Example 8\(\frac{\partial Ax}{\partial x}=A\)

\[f(x)=Ax\]

\[\begin{aligned}f(x+h)-f(x)&=A(x+h)-Ax\\&=Ah\\&=Ah+o(h)\end{aligned}\]

Therefore \[\frac{\partial Ax}{\partial x}=A\]

Example 9\(\frac{\partial x^TAx}{\partial x}=x^T(A+A^T)\)

\[\begin{aligned} \mathop{\mathrm{d}}(x^TAx) &= (\mathop{\mathrm{d}}x^T)Ax + x^T (\mathop{\mathrm{d}}Ax) \\ &= (\mathop{\mathrm{d}}x)^TAx + x^T A (\mathop{\mathrm{d}}x) \\ &= x^TA^T (\mathop{\mathrm{d}}x) + x^T A (\mathop{\mathrm{d}}x) \\ &= x^T(A+A^T) (\mathop{\mathrm{d}}x) \\ \end{aligned}\]

Therefore

\[\frac{\partial x^TAx}{\partial x}=x^T(A+A^T)\]

If \(A\) is symmetric then this reduces to \(2x^TA\).

Example 10\(\frac{\partial x^TAx}{\partial x}=x^T(A+A^T)\)

\[f(x)=x^TAx\]

\[\begin{aligned}f(x+h)-f(x)&=(x+h)^TA(x+h)-x^TAx \\ &= x^TAx +h^TAx +x^TAh +h^TAh -x^TAx \\&= h^TAx+x^TAh+h^TAh \\ &= x^TA^Th+x^TAh+h^TAh \\ &= x^T(A^T+A)h + h^TAh \\ &= x^T(A^T+A)h+o(h)\end{aligned}\]

Therefore

\[\frac{\partial x^TAx}{\partial x}=x^T(A+A^T)\]

Example 11\(\frac{\partial x^TAx}{\partial A}=xx^T\)

We have

\[x^TAx=\mathop{\mathrm{Tr}}\{ A xx^T\}\]

The differential of \(x^TAx\) wrt \(A\) is

\[\begin{aligned} \mathop{\mathrm{d}}(x^T A x) &= \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}\{ Axx^T\} \\ &= \mathop{\mathrm{Tr}}\{ \mathop{\mathrm{d}}(Axx^T)\} \\ &= \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}A) xx^T \} \\ &= \mathop{\mathrm{Tr}}\{ xx^T (\mathop{\mathrm{d}}A) \} \end{aligned}\]

Therefore

\[\frac{\partial x^TAx}{\partial A}=xx^T\]

Example 12\(\frac{\partial x^TAx}{\partial A}=xx^T\)

\[f(A)=x^TAx\]

\[\begin{aligned}f(A+H)-f(A)&=x^T(A+H)x-x^TAx\\&=x^THx\\&=\mathop{\mathrm{Tr}}(Hxx^T)\\&=\mathop{\mathrm{Tr}}(xx^TH)\\&=\mathop{\mathrm{Tr}}(xx^TH)+o(H)\end{aligned}\]

Therefore

\[\frac{\partial x^TAx}{\partial A}=xx^T\]

Example 13\(\frac{\partial \mathop{\mathrm{Tr}}(AX)}{\partial X}=A\)

\[\begin{aligned} \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}(AX)&=\mathop{\mathrm{Tr}}(\mathop{\mathrm{d}}(AX)) \\ &= \mathop{\mathrm{Tr}}(A \mathop{\mathrm{d}}X) \\ \end{aligned}\]

Therefore

\[\frac{\partial \mathop{\mathrm{Tr}}(AX)}{\partial X}=A\]

Example 14\(\frac{\partial \mathop{\mathrm{Tr}}(AX^T)}{\partial X}=A^T\)

\[\begin{aligned} \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}(AX^T) &= \mathop{\mathrm{Tr}}(\mathop{\mathrm{d}}(AX^T)) \\ &= \mathop{\mathrm{Tr}}(A \mathop{\mathrm{d}}X^T) \\ &= \mathop{\mathrm{Tr}}(A (\mathop{\mathrm{d}}X)^T) \\ &= \mathop{\mathrm{Tr}}( (\mathop{\mathrm{d}}X) A^T) \\ &= \mathop{\mathrm{Tr}}( A^T \mathop{\mathrm{d}}X) \end{aligned}\]

Therefore \[\frac{\partial \mathop{\mathrm{Tr}}(AX^T)}{\partial X}=A^T\]

Example 15\(\frac{\partial \mathop{\mathrm{Tr}}(X^TAX)}{\partial X}=X^T(A+A^T)\)

\[\begin{aligned} \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}(X^TAX)&= \mathop{\mathrm{Tr}}\mathop{\mathrm{d}}(X^TAX) \\ &= \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}X^T)AX + X^T (\mathop{\mathrm{d}}AX) \} \\ &= \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}X^T)AX + X^T A (\mathop{\mathrm{d}}X) \} \\ &= \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}X^T)AX \}+ \mathop{\mathrm{Tr}}\{ X^T A (\mathop{\mathrm{d}}X) \} \\ &= \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}X)^TAX \}+ \mathop{\mathrm{Tr}}\{ X^T A (\mathop{\mathrm{d}}X) \} \\ &= \mathop{\mathrm{Tr}}\{ X^T A^T (\mathop{\mathrm{d}}X) \}+ \mathop{\mathrm{Tr}}\{ X^T A (\mathop{\mathrm{d}}X) \} \\ &= \mathop{\mathrm{Tr}}\{ X^T A^T (\mathop{\mathrm{d}}X) + X^T A (\mathop{\mathrm{d}}X) \} \\ &= \mathop{\mathrm{Tr}}\{ X^T (A+A^T) (\mathop{\mathrm{d}}X) \} \end{aligned}\]

Therefore \[\frac{\partial \mathop{\mathrm{Tr}}(X^TAX)}{\partial X}=X^T(A+A^T)\]

Example 16\(\frac{\partial \mathop{\mathrm{Tr}}(X^{-1}A)}{\partial X}=-X^{-1}AX^{-1}\)

\[\begin{aligned} \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}\{ X^{-1}A \} &= \mathop{\mathrm{Tr}}\{ \mathop{\mathrm{d}}(X^{-1}A) \}\\ &= \mathop{\mathrm{Tr}}\{ \mathop{\mathrm{d}}(X^{-1}A)^T \} \\ &= \mathop{\mathrm{Tr}}\{ \mathop{\mathrm{d}}(A^T X^{-T}) \} \\ &= \mathop{\mathrm{Tr}}\{ A^T \mathop{\mathrm{d}}( X^{-T}) \} \\ &= \mathop{\mathrm{Tr}}\{ A^T (\mathop{\mathrm{d}}X^{-1})^T \} \\ &= \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}X^{-1}) A \} \\ &= \mathop{\mathrm{Tr}}\{ A (\mathop{\mathrm{d}}X^{-1}) \} \\ &= \mathop{\mathrm{Tr}}\{ -AX^{-1}(\mathop{\mathrm{d}}X) X^{-1} \} \\ &= \mathop{\mathrm{Tr}}\{ -X^{-1}AX^{-1}(\mathop{\mathrm{d}}X) \} \\ \end{aligned}\]

Therefore

\[\frac{\partial \mathop{\mathrm{Tr}}(X^{-1}A)}{\partial X}=-X^{-1}AX^{-1}\]

Example 17\(\frac{\partial \mathop{\mathrm{Tr}}(AXBX^T C) }{\partial X}= BX^T C A + B^T X^T A^T C^T\)

\[\begin{aligned} \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}(AXBX^TC)&= \mathop{\mathrm{Tr}}\{ \mathop{\mathrm{d}}(AXBX^TC) \} \\ &= \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}(AX))BX^TC + AX(\mathop{\mathrm{d}}(BX^TC)) \} \\ &= \mathop{\mathrm{Tr}}\{ A(\mathop{\mathrm{d}}X)BX^TC + AXB(\mathop{\mathrm{d}}X^T)C \} \\ &= \mathop{\mathrm{Tr}}\{ A(\mathop{\mathrm{d}}X)BX^TC\} + \mathop{\mathrm{Tr}}\{AXB(\mathop{\mathrm{d}}X^T)C \} \\ &= \mathop{\mathrm{Tr}}\{ BX^TCA (\mathop{\mathrm{d}}X)\} + \mathop{\mathrm{Tr}}\{CAXB(\mathop{\mathrm{d}}X^T) \} \\ &= \mathop{\mathrm{Tr}}\{ BX^TCA (\mathop{\mathrm{d}}X)\} + \mathop{\mathrm{Tr}}\{(\mathop{\mathrm{d}}X)B^TX^TA^TC^T \} \\ &= \mathop{\mathrm{Tr}}\{ BX^TCA (\mathop{\mathrm{d}}X)\} + \mathop{\mathrm{Tr}}\{B^TX^TA^TC^T(\mathop{\mathrm{d}}X) \} \\ &= \mathop{\mathrm{Tr}}\{ BX^TCA (\mathop{\mathrm{d}}X) + B^TX^TA^TC^T(\mathop{\mathrm{d}}X) \} \\ &= \mathop{\mathrm{Tr}}\{ (BX^TCA + B^TX^TA^TC^T)(\mathop{\mathrm{d}}X) \} \\ \end{aligned}\]

Therefore \[\frac{\partial \mathop{\mathrm{Tr}}(AXBX^T C) }{\partial X}= BX^T C A + B^T X^T A^T C^T\]

Example 18\(\frac{\partial (Xa+b)^TC(Xa+b) }{\partial X} = a(Xa+b)^T(C+C^T) = [(C+C^T)(Xa+b)a^T]^T\)

Let

\[y=(Xa+b)^TC(Xa+b)\]

and \(w=Xa+b\), then \(y=w^TCw\).

\[\frac{\partial y}{\partial w}=w^T (C+C^T)\]

\[\begin{aligned} dy &= \mathop{\mathrm{Tr}}\{ w^T (C+C^T) (\mathop{\mathrm{d}}w)\} \\ &= \mathop{\mathrm{Tr}}\{ w^T (C+C^T) \mathop{\mathrm{d}}(Xa+b) \} \\ &= \mathop{\mathrm{Tr}}\{ w^T (C+C^T) (\mathop{\mathrm{d}}X) a\} \\ &= \mathop{\mathrm{Tr}}\{ a w^T (C+C^T) (\mathop{\mathrm{d}}X)\} \\ \end{aligned}\]

Therefore

\[\begin{aligned} \frac{\partial y}{\partial X}&=aw^T (C+C^T) \\ &= a (Xa+b)^T (C+C^T) \\ &= [(C+C^T) (Xa+b) a^T]^T \end{aligned}\]

Example 19\(\frac{\partial (Bx+b)^T C (Dx+d)}{\partial x}=[B^T C (Dx+d)+D^T C^T (Bx+b)]^T\)

\[\begin{aligned} (Bx+b)^T C (Dx+d)&= (x^T B^T + b^T) C (Dx+d) \\ &= (x^T B^T C + b^T C) (Dx+d) \\ &= x^T B^T C Dx + x^T B^T C d + b^T C D x + b^T C d \\ \end{aligned}\]

Therefore we have;

\[\begin{aligned} \mathop{\mathrm{d}}[(Bx+b)^T C (Dx+d)] &= \mathop{\mathrm{d}}[x^T B^T C Dx + x^T B^T C d + b^T C D x + b^T C d] \\ &= \mathop{\mathrm{d}}(x^T B^T C Dx) + \mathop{\mathrm{d}}(x^T B^T C d) + \mathop{\mathrm{d}}(b^T C D x) \\ &= x^T (B^T C D + D^T C^T B) \mathop{\mathrm{d}}x + (\mathop{\mathrm{d}}x^T) (B^T C d) + (b^T C D)(\mathop{\mathrm{d}}x) \\ &= x^T (B^T C D + D^T C^T B) \mathop{\mathrm{d}}x + (d^T C^T B) (\mathop{\mathrm{d}}x) + (b^T C D)(\mathop{\mathrm{d}}x) \\ &= [x^T (B^T C D + D^T C^T B) + (d^T C^T B) + (b^T C D)](\mathop{\mathrm{d}}x) \\ &= [x^T (B^T C D + D^T C^T B) + (d^T C^T B) + (b^T C D)](\mathop{\mathrm{d}}x) \end{aligned}\]

Therefore;

\[\begin{aligned} \frac{\partial (Bx+b)^T C (Dx+d)}{\partial x} &= x^T B^T C D + x^T D^T C^T B + d^T C^T B + b^T C D \\ &= (Dx+d)^TC^TB + (Bx+b)^T CD\\ &= [B^T C(Dx+d)+D^TC^T(Bx+b)]^T \end{aligned}\]

Therefore

\[\frac{\partial (Bx+b)^T C (Dx+d)}{\partial x}=[B^T C (Dx+d)+D^T C^T (Bx+b)]^T\]

Example 20\(\frac{\partial b^T X^T DXc}{\partial X}=[D^TXbc^T+DXcb^T]^T=cb^TX^TD+bc^TX^TD^T\)

Let

\[y=b^TX^TDXc\]

\[A=X^TDX\]

Therefore

\[y=b^TAc\]

and let

\[z=Ac\]

then

\[y=b^Tz\]

\[\begin{aligned} \mathop{\mathrm{d}}y &= \mathop{\mathrm{d}}(b^Tz) \\ &= b^T \mathop{\mathrm{d}}z \\ &= \mathop{\mathrm{Tr}}\{ b^T \mathop{\mathrm{d}}z \} \\ &= \mathop{\mathrm{Tr}}\{ b^T \mathop{\mathrm{d}}(Ac)\} \\ &= \mathop{\mathrm{Tr}}\{ b^T (\mathop{\mathrm{d}}A) c\} \\ &= \mathop{\mathrm{Tr}}\{ cb^T (\mathop{\mathrm{d}}A) \} \\ &= \mathop{\mathrm{Tr}}\{ cb^T \mathop{\mathrm{d}}(X^TDX) \} \\ &= \mathop{\mathrm{Tr}}\{ cb^T (\mathop{\mathrm{d}}X^T) DX + cb^T X^T D (\mathop{\mathrm{d}}X) \} \\ &= \mathop{\mathrm{Tr}}\{ DXcb^T (\mathop{\mathrm{d}}X^T) \} +\mathop{\mathrm{Tr}}\{ cb^T X^T D (\mathop{\mathrm{d}}X)\} \\ &= \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}X) bc^TX^TD^T \} + \mathop{\mathrm{Tr}}\{ cb^T X^T D (\mathop{\mathrm{d}}X)\} \\ &= \mathop{\mathrm{Tr}}\{ bc^TX^TD^T (\mathop{\mathrm{d}}X)\} + \mathop{\mathrm{Tr}}\{ cb^T X^T D (\mathop{\mathrm{d}}X)\} \\ &= \mathop{\mathrm{Tr}}\{ (bc^TX^TD^T +cb^T X^T D) (\mathop{\mathrm{d}}X)\} \\ \end{aligned}\]

Therefore

\[\frac{\partial b^T X^T DXc}{\partial X}=cb^TX^TD+bc^TX^TD^T=[D^TXbc^T+DXcb^T]^T\]

Inverse and Determinant

Differential of Inverse

The differential of \(X^{-1}\) wrt \(X\) is:

\[\mathop{\mathrm{d}}X^{-1} = - X^{-1} (\mathop{\mathrm{d}}X) X^{-1}\]

Differential and Derivative of Determinant

In numerator layout;

\[\frac{\partial det(X)}{\partial X}= adj(X) = det(X) X^{-1}\]

and in denominator layout;

\[\frac{\partial det(X)}{\partial X}= adj(X) = det(X) (X^{-1})^T\]

Derivative of Inverse \[\frac{\partial A^{-1}_{mn}}{\partial A_{ij}}=-A^{-1}_{mi} A^{-1}_{jn}\]

We use

\[\frac{\partial A^{-1}}{\partial x}=-A^{-1} \frac{\partial A}{\partial x} A^{-1}\]

Therefore

\[\frac{\partial A^{-1}}{\partial A_{ij}}=-A^{-1} \frac{\partial A}{\partial A_{ij}} A^{-1}\]

Therefore

\[\begin{aligned} \frac{\partial A^{-1}_{mn}}{\partial A_{ij}} &=\Bigg [ -A^{-1} \frac{\partial A}{\partial A_{ij}} A^{-1} \Bigg ]_{mn} \\ &= - \sum_k A^{-1}_{mk} [ \frac{\partial A}{\partial A_{ij}} A^{-1} ]_{kn} \\ &= -\sum_k A^{-1}_{mk} \sum_l (\frac{\partial A}{\partial A_{ij}})_{kl} A^{-1}_{ln} \\ &= -\sum_k \sum_l A^{-1}_{mk} \frac{\partial A_{kl}}{\partial A_{ij}} A^{-1}_{ln} \\ &= -\sum_k \sum_l A^{-1}_{mk} \delta_{ki} \delta_{lj} A^{-1}_{ln} \\ &= -\sum_k A^{-1}_{mk} \delta_{ki} A^{-1}_{jn} \\ &= -A^{-1}_{mi} A^{-1}_{jn} \\ \end{aligned}\]

Therefore

\[\frac{\partial A^{-1}_{mn}}{\partial A_{ij}}=-A^{-1}_{mi} A^{-1}_{jn}\]

Derivative of Determinant \[\frac{\partial det(X)}{\partial X_{ij}}= det(X) (X^{-1})^T_{ij}\]

In this section we will use Jacobi’s formula (see Wikipedia)

In denominator layout we have;

\[\frac{\partial det(X)}{\partial X_{ij}}=\Bigg ( \frac{\partial det(X)}{\partial X} \Bigg )_{ij}\]

The cofactor matrix \(C\) of a matrix \(X\) is given by:

\[X^{-1}=\frac{1}{det(X)}C^T\]

Therefore \[C^T=det(X)X^{-1}\]

i.e.

\[C=det(X) (X^{-1})^T\]

The adjugate of \(X\) is the transpose of the cofactor matrix therefore;

\[adj(X)=det(X) X^{-1}\]

The differential of \(det(X)\) is

\[\mathop{\mathrm{d}}det(X) = \mathop{\mathrm{Tr}}\{ adj(X) \mathop{\mathrm{d}}X\} = \mathop{\mathrm{Tr}}\{ det(X) X^{-1} \mathop{\mathrm{d}}X\}\]

Therefore in numerator layout

\[\frac{\partial det(X)}{\partial X}=adj(X)=det(X) X^{-1}\]

and in denominator layout \[\frac{\partial det(X)}{\partial X}=adj(X)^T=det(X)(X^{-1})^T\]

Therefore

\[\frac{\partial det(X)}{\partial X_{ij}}=[det(X)(X^{-1})^T]_{ij} = det(X) (X^{-1})^T_{ij} = det(X) (X^{-1})_{ji}\]

Therefore

\[\frac{\partial det(X)}{\partial X_{ij}}= det(X) (X^{-1})^T_{ij}\]

Machine Learning Examples

Least Squares

We want to minimize \((Y-X\beta)^T(Y-X\beta)\).

We use differentials to find the derivative.

\[\begin{aligned} \mathop{\mathrm{d}}(Y-X\beta)^T(Y-X\beta) &= \mathop{\mathrm{d}}(Y^TY-Y^TX\beta -\beta^T X^T Y + \beta^T X^TX \beta) \\ &= \mathop{\mathrm{d}}( -2 Y^T X \beta) + \mathop{\mathrm{d}}(\beta^T X^TX \beta) \\ &= -2Y^T X (\mathop{\mathrm{d}}\beta) + (\mathop{\mathrm{d}}\beta)^T X^TX \beta + \beta^T X^TX (\mathop{\mathrm{d}}\beta) \\ &= - 2 Y^T X (\mathop{\mathrm{d}}\beta) + 2\beta^T X^TX (\mathop{\mathrm{d}}\beta) \\ &= \Bigg [ 2 \beta^T X^T X - 2 Y^T X \Bigg ] (\mathop{\mathrm{d}}\beta) \\ \end{aligned}\]

Therefore

\[\frac{\partial (Y-X\beta)^T(Y-X\beta)}{\partial \beta}=2 \beta^T X^T X - 2 Y^T X\]

Setting this derivative to zero we get:

\[\begin{aligned} & 2 \beta^T X^T X - 2 Y^T X = 0 \\ & \Rightarrow X^TX \beta = X^T Y \\ & \Rightarrow \beta = (X^TX)^{-1} X^T Y \\ \end{aligned}\]

Linear Regression with Multiple Outputs (PRML 3.1.5)

The log likelihood can be expressed using a trace since:

\[\|x\|^2=x^Tx=\mathop{\mathrm{Tr}}\{xx^T \}\]

Therefore the log likelihood is:

\[\begin{aligned} \log p(T|X,W,\beta) &= \frac{NK}{2} \log \{ \frac{\beta }{2\pi}\} - \frac{\beta}{2}\sum_{n=1}^N \| t_n - W^T \phi_n \|^2 \\ &= \frac{NK}{2} \log \{ \frac{\beta }{2\pi}\} - \frac{\beta}{2}\sum_{n=1}^N (t_n - W^T \phi_n )^T(t_n-W^T\phi_n) \\ &= \frac{NK}{2} \log \{ \frac{\beta }{2\pi}\} - \frac{\beta}{2}\sum_{n=1}^N \mathop{\mathrm{Tr}}\{(t_n - W^T \phi_n )(t_n-W^T\phi_n)^T \} \\ \end{aligned}\]

Concentrating on the trace we have:

\[\begin{aligned} \mathop{\mathrm{Tr}}\{(t_n - W^T \phi_n )(t_n-W^T\phi_n)^T\} &=\mathop{\mathrm{Tr}}\{(t_n - W^T \phi_n )(t_n^T-\phi_n^T W)\} \\ &= \mathop{\mathrm{Tr}}\{t_n t_n^T - t_n \phi_n^T W - W^T \phi_n t_n^T + W^T \phi_n \phi_n^T W \} \\ &= \mathop{\mathrm{Tr}}\{t_n t_n^T \} - \mathop{\mathrm{Tr}}\{ t_n \phi_n^T W \} - \mathop{\mathrm{Tr}}\{W^T \phi_n t_n^T \} +\mathop{\mathrm{Tr}}\{W^T \phi_n \phi_n^T W \} \\ \end{aligned}\]

Now we apply other formulas we have derived to find the derivatives of these traces.

\[\frac{\partial \mathop{\mathrm{Tr}}\{ t_n \phi_n^T W\}}{\partial W}=t_n \phi_n^T\]

\[\frac{\partial \mathop{\mathrm{Tr}}\{ W^T \phi_n t_n^T \} }{\partial W}=t_n \phi_n^T\]

\[\frac{\partial W^T \phi_n \phi_n^T W }{\partial W}=2 W^T \phi_n \phi_n^T\]

Therefore

\[\begin{aligned} \frac{\partial \log p (T|X,W,\beta)}{\partial W}&=-\frac{\beta}{2}\sum_{n=1}^N \Bigg \{ -2 t_n \phi_n^T + 2 W^T \phi_n \phi_n^T \Bigg \} \\ &= \beta \sum_{n=1}^N \Bigg \{ t_n \phi_n^T - W^T \phi_n \phi_n^T \Bigg \} \\ \end{aligned}\]

Setting this derivative equal to zero we get:

\[\begin{aligned} & W^T \sum_{n=1}^N \phi_n \phi_n^T = \sum_{n=1}^N t_n \phi_n^T \\ & \Rightarrow (\sum_{n=1}^N \phi_n \phi_n^T) W = \sum_{n=1}^N \phi_n t_n^T \\ \end{aligned}\]

\(\sum_{n=1}^N \phi_n t_n^T\) is the standard column-row expansion of the matrix \(\Phi^T T\) where \(\Phi\) is a matrix with rows \(\phi_n^T\) and \(T\) is a matrix with rows \(t_n^T\).

Using the expression for the \((i,j)^{th}\) element of the outer product:

\[(uv^T)_{ij}=u_i v_j\]

we have

\[(\phi(x_n)t_n^T)_{ij}=\phi_i(x_n)(t_n)_j\]

Therefore

\[(\sum_{n=1}^N \phi(x_n)t_n^T)_{ij}=\sum_{n=1}^N \phi_i(x_n)(t_n)_j\]

Now consider \(\Phi^T T\). We have:

\[\begin{aligned} (\Phi^T T)_{ij}&=\sum_k \Phi^T_{ik} T_{kj} \\ &= \sum_k \Phi_{ki} T_{kj} \\ &= \sum_k \phi_i(x_k) (t_k)_j \\ &= (\sum_k \phi(x_k) t_k^T)_{ij} \end{aligned}\]

Therefore

\[\sum_{n=1}^N \phi(x_n)t_n^T=\Phi^T T\]

Now let’s examine \(\sum_{n=1}^N \phi_n \phi_n^T\).

We have

\[(uv^T)_{ij}=u_i v_j\]

therefore

\[(\phi(x_n) \phi(x_n)^T)_{ij} = \phi_i(x_n) \phi_j(x_n)\]

therefore

\[(\sum_{n=1}^N \phi(x_n) \phi(x_n)^T)_{ij} = \sum_{n=1}^N \phi_i(x_n) \phi_j(x_n)\]

Also if \(\Phi\) is the matrix with rows \(\phi(x_n)\) then by the row-column rule:

\[\begin{aligned} (\Phi^T\Phi)_{ij} &= \sum_k (\Phi^T)_{ik} \Phi_{kj} \\ &= \sum_k \Phi_{ki} \Phi_{kj} \\ &= \sum_k \phi_i(x_k) \phi_j (x_k) \\ \end{aligned}\]

Therefore \(\Phi^T\Phi=\sum_{n=1}^N \phi(x_n)\phi(x_n)^T\).

Therefore

\[\Phi^T \Phi W = \Phi^T T\]

or

\[W = (\Phi^T \Phi)^{-1} \Phi^T T\]

Maximum Likelihood for the Gaussian (Ch 2.3.4 PRML)

The log likelihood is given by

\[\log p(X|\mu, \Sigma)=-\frac{ND}{2}\log (2\pi) - \frac{N}{2} \log |\Sigma| - \frac{1}{2}\sum_{n=1}^N (x_n-\mu)^T \Sigma^{-1} (x_n - \mu)\]

Now use the differential of \(x^TAx\) wrt \(A\) which is given by

\[\mathop{\mathrm{d}}(x^TAx) = \mathop{\mathrm{Tr}}\{xx^T (\mathop{\mathrm{d}}A)\}\]

and the differential of the log of the determinant which is

\[\mathop{\mathrm{d}}\log |X| = \mathop{\mathrm{Tr}}\{ X^{-1} (\mathop{\mathrm{d}}X)\}\]

then the differential of the log likelihood with respect to \(\Sigma^{-1}\) is

\[\begin{aligned} \mathop{\mathrm{d}}\log p(X|\mu, \Sigma) &= \frac{N}{2} \mathop{\mathrm{d}}\log |\Sigma^{-1}| -\frac{1}{2} \sum_{n=1}^N \mathop{\mathrm{d}}[(x_n-\mu)^T \Sigma^{-1} (x_n-\mu)] \\ &= \frac{N}{2} \mathop{\mathrm{Tr}}\{ \Sigma (\mathop{\mathrm{d}}\Sigma^{-1}) \} - \frac{1}{2} \sum_{n=1}^N \mathop{\mathrm{Tr}}\{ (x_n-\mu)(x_n-\mu)^T (\mathop{\mathrm{d}}\Sigma^{-1})\} \\ &= \mathop{\mathrm{Tr}}\Bigg \{ \Bigg [\frac{N}{2} \Sigma - \frac{1}{2}\sum_{n=1}^N (x_n-\mu)(x_n-\mu)^T \Bigg ] (\mathop{\mathrm{d}}\Sigma^{-1}) \Bigg \} \\ \end{aligned}\]

Therefore

\[\frac{\partial \log p(X|\mu, \Sigma)}{\partial \Sigma^{-1}}=\frac{N}{2}\Sigma - \frac{1}{2} \sum_{n=1}^N (x_n-\mu)(x_n-\mu)^T\]

Setting this derivative equal to zero we get

\[\Sigma = \frac{1}{N} \sum_{n=1}^N (x_n-\mu)(x_n-\mu)^T\]

PRML Ch 3.51 Equation 3.83

Let

\[E(w)=\frac{\beta}{2} \| t-\Phi w \|^2 + \frac{\alpha}{2}w^Tw\]

We want to show

\[\nabla \nabla E(w) = \alpha I + \beta \Phi^T\Phi\]

The first differential of \(\frac{\alpha}{2} w^Tw\) is:

\[\begin{aligned} \mathop{\mathrm{d}}(\frac{\alpha}{2} w^Tw) &= \frac{\alpha}{2} \mathop{\mathrm{d}}(w^Tw)\\ &= \frac{\alpha}{2} (\mathop{\mathrm{d}}w)^T w + \frac{\alpha}{2} w^T (\mathop{\mathrm{d}}w) \\ &= \alpha w^T (\mathop{\mathrm{d}}w) \\ \end{aligned}\]

The second differential of \(\frac{\alpha}{2} w^Tw\) is:

\[\begin{aligned} \mathop{\mathrm{d}}^2 (\frac{\alpha}{2} w^Tw) &= \mathop{\mathrm{d}}( \alpha w^T (\mathop{\mathrm{d}}w)) \\ &= \alpha (\mathop{\mathrm{d}}w^T) (\mathop{\mathrm{d}}w) + \alpha w^T (\mathop{\mathrm{d}}^2 w) \\ &= \alpha (\mathop{\mathrm{d}}w)^T (\mathop{\mathrm{d}}w) \\ &= (\mathop{\mathrm{d}}w)^T (\alpha I) (\mathop{\mathrm{d}}w) \\ \end{aligned}\]

Therefore

\[\nabla \nabla \frac{\alpha}{2} w^Tw = \alpha I\]

Now consider \(\frac{\beta}{2} \| t-\Phi w \|^2\)

We have

\[\begin{aligned} \frac{\beta}{2} \| t-\Phi w \|^2 &= \frac{\beta}{2} (t-\Phi w )^T (t-\Phi w) \\ &= \frac{\beta}{2} (t^T t- 2 t^T \Phi w + w^T \Phi^T \Phi w) \\ \end{aligned}\]

Therefore the first differential of \(\frac{\beta}{2} \| t-\Phi w \|^2\) is:

\[\begin{aligned} \mathop{\mathrm{d}}(\frac{\beta}{2} \| t-\Phi w \|^2) &= \mathop{\mathrm{d}}(\frac{\beta}{2} (t^T t- 2 t^T \Phi w + w^T \Phi^T \Phi w)) \\ &= (\frac{\beta}{2} \mathop{\mathrm{d}}(t^T t- 2 t^T \Phi w + w^T \Phi^T \Phi w)) \\ &= \frac{\beta}{2} (- 2 t^T \Phi(\mathop{\mathrm{d}}w) + (\mathop{\mathrm{d}}w)^T \Phi^T \Phi w +w^T \Phi^T \Phi (\mathop{\mathrm{d}}w)) \\ &= -\beta t^T \Phi(\mathop{\mathrm{d}}w)+\frac{\beta}{2}(\mathop{\mathrm{d}}w)^T \Phi^T \Phi w +\frac{\beta}{2}w^T \Phi^T \Phi (\mathop{\mathrm{d}}w) \\ &= -\beta t^T \Phi(\mathop{\mathrm{d}}w)+\beta w^T \Phi^T \Phi (\mathop{\mathrm{d}}w) \\ \end{aligned}\]

Therefore the second differential of \(\frac{\beta}{2} \| t-\Phi w \|^2\) is:

\[\begin{aligned} \mathop{\mathrm{d}}^2 (\frac{\beta}{2} \| t-\Phi w \|^2) &= \mathop{\mathrm{d}}(-\beta t^T \Phi(\mathop{\mathrm{d}}w)+\beta w^T \Phi^T \Phi (\mathop{\mathrm{d}}w)) \\ &= -\beta t^T \Phi (\mathop{\mathrm{d}}^2 w) + \beta (\mathop{\mathrm{d}}w^T) \Phi^T \Phi (\mathop{\mathrm{d}}w) + \beta w^T \Phi ^T \Phi (\mathop{\mathrm{d}}^2 w) \\ &= \beta (\mathop{\mathrm{d}}w^T) \Phi^T \Phi (\mathop{\mathrm{d}}w) \\ &= \beta (\mathop{\mathrm{d}}w)^T \Phi^T \Phi (\mathop{\mathrm{d}}w) \\ &= (\mathop{\mathrm{d}}w)^T \beta \Phi^T \Phi (\mathop{\mathrm{d}}w) \\ \end{aligned}\]

Therefore

\[\nabla \nabla \frac{\beta}{2} \| t-\Phi w \|^2 = \beta \Phi^T \Phi\]

Therefore

\[\nabla \nabla E(w) = \alpha I + \beta \Phi^T\Phi\]

which is equation 3.83 in Pattern Recognition and Machine Learning.

Ch 4.1.3 Least squares for classification PRML

Let

\[E_D(\widetilde{W})=\frac{1}{2} \mathop{\mathrm{Tr}}\{ (\widetilde{X}\widetilde{W}-T)^T (\widetilde{X}\widetilde{W}-T) \}=\frac{1}{2} \mathop{\mathrm{Tr}}\{ \widetilde{W}^T \widetilde{X}^T \widetilde{X} \widetilde{W}\}-\mathop{\mathrm{Tr}}\{ \widetilde{W}^T \widetilde{X}^T T \} + \frac{1}{2} \mathop{\mathrm{Tr}}\{ T^T T\}\]

Now use what we proved earlier:

\[\frac{\partial \mathop{\mathrm{Tr}}\{X^T A X\} }{\partial X}=X^T (A+A^T)\]

and

\[\frac{\partial \mathop{\mathrm{Tr}}\{ X^TA \} }{\partial X}=A^T\]

Therefore

\[\frac{\partial E_D(\widetilde{W})}{\partial \widetilde{W}}= \widetilde{W}^T \widetilde{X}^T \widetilde{X}-T^T \widetilde{X}\]

Setting this derivative to zero we get

\[\widetilde{W} = (\widetilde{X}^T \widetilde{X})^{-1} \widetilde{X}^T T\]

which is equation 4.16 in PRML.

Ch 4.2.2 Probabilistic Generative Models

Let

\[\begin{aligned} \log p(t | \pi, \mu_1, \mu_2, \Sigma) &= -\frac{1}{2}\sum_{n=1}^N t_n (x_n - \mu_1)^T \Sigma ^{-1} (x_n - \mu_1) - \frac{1}{2} \sum_{n=1}^N t_n \log |\Sigma | \\& - \frac{1}{2} \sum_{n=1}^N (1-t_n) (x_n-\mu_2)^T \Sigma ^{-1} (x_n-\mu_2) - \frac{1}{2} \sum_{n=1}^N (1-t_n) \log | \Sigma |+const. \end{aligned}\]

Also \(t_n=1\) for \(n \in C_1\) and \(t_n=0\) for \(n \in C_2\) and also \(\sum_{n=1}^N t_n = N_1\) and \(\sum_{n=1}^N (1-t_n)=N_2\) and \(N_1+N_2=N\).

Then using \[x^TAx = \mathop{\mathrm{Tr}}\{ A x x^T\}\] we have:

\[\begin{aligned} \log p(t | \pi, \mu_1, \mu_2, \Sigma)&=-\frac{1}{2}\sum_{n\in C_1} \mathop{\mathrm{Tr}}\{\Sigma ^{-1} (x_n - \mu_1)(x_n - \mu_1)^T \} +\frac{N_1}{2} \log |\Sigma^{-1} | \\ & - \frac{1}{2} \sum_{n \in C_2} \mathop{\mathrm{Tr}}\{ \Sigma ^{-1} (x_n-\mu_2)(x_n-\mu_2)^T\} + \frac{N_2}{2} \log | \Sigma^{-1} |+const. \\ \end{aligned}\]

Now using \[\frac{\partial \mathop{\mathrm{Tr}}(XA)}{\partial X}=A\]

and \[\frac{\partial \log |X|}{\partial X}= X^{-1}\]

we have

\[\frac{\partial \log p(t | \pi, \mu_1, \mu_2, \Sigma ) }{\partial \Sigma^{-1}}=-\frac{1}{2}\sum_{n\in C_1} (x_n-\mu_1)(x_n-\mu_1)^T-\frac{1}{2}\sum_{n \in C_2} (x_n-\mu_2)(x_n-\mu_2)^T + \frac{N}{2}\Sigma\]

Setting this derivative equal to zero we get:

\[\Sigma = \frac{1}{N}\sum_{n\in C_1} (x_n-\mu_1)(x_n-\mu_1)^T + \frac{1}{N} \sum_{n \in C_2} (x_n-\mu_2)(x_n-\mu_2)^T\]

COMP4670 ANU Example

\(f(X) = \frac{1}{2} \mathop{\mathrm{Tr}}\{X^T C X N\} + \mu \frac{1}{4} \| N - X^T X \|^2_F\)

where \(C^T=C\) and \(N\) is diagonal therefore \(N^T=N\)

\[\begin{aligned} \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}(X^TCXN)&= \mathop{\mathrm{Tr}}\{ \mathop{\mathrm{d}}(X^TCXN) \}\\ & = \mathop{\mathrm{Tr}}\{(\mathop{\mathrm{d}}X^T)CXN+X^TC(\mathop{\mathrm{d}}X)N\} \\ &= \mathop{\mathrm{Tr}}\{(\mathop{\mathrm{d}}X^T)CXN\}+\mathop{\mathrm{Tr}}\{X^TC(\mathop{\mathrm{d}}X)N\} \\ &= \mathop{\mathrm{Tr}}\{N^TX^TC^T(\mathop{\mathrm{d}}X)\}+\mathop{\mathrm{Tr}}\{NX^TC(\mathop{\mathrm{d}}X)\} \\ &= \mathop{\mathrm{Tr}}\{[N^TX^TC^T+NX^TC](\mathop{\mathrm{d}}X)\} \end{aligned}\]

Therefore in numerator layout:

\[\frac{\partial \space \mathop{\mathrm{Tr}}(X^TCXN)}{\partial X}=N^TX^TC^T+NX^TC\]

or in denominator layout (taking the transpose):

\[\frac{\partial \space \mathop{\mathrm{Tr}}(X^TCXN)}{\partial X}=CXN+C^TXN^T\]

\[\begin{aligned} \| N - X^T X \|^2_F &= \mathop{\mathrm{Tr}}\{ (N-X^TX)^T (N-X^TX) \} \\ &= \mathop{\mathrm{Tr}}\{ (N^T - X^TX)(N-X^TX)\} \\ &= \mathop{\mathrm{Tr}}\{ N^TN - N^T X^TX - X^TX N + X^TX X^TX\} \\ &= \mathop{\mathrm{Tr}}\{ N^T N\} - \mathop{\mathrm{Tr}}\{ N^T X^TX\} - \mathop{\mathrm{Tr}}\{ X^T X N\} + \mathop{\mathrm{Tr}}\{ X^T X X^TX\} \\ &= \mathop{\mathrm{Tr}}\{ N^T N\} -2 \mathop{\mathrm{Tr}}\{ N^T X^T X\} + \mathop{\mathrm{Tr}}\{ X^TX X^TX\} \end{aligned}\]

Now

\[\begin{aligned} \mathop{\mathrm{d}}\| N - X^T X \|^2_F &= \mathop{\mathrm{d}}[\mathop{\mathrm{Tr}}\{ N^T N\} -2 \mathop{\mathrm{Tr}}\{ N^T X^T X\} + \mathop{\mathrm{Tr}}\{ X^TX X^TX\} ] \\ &= \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}\{ N^T N\} -2 \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}\{ N^T X^T X\} + \mathop{\mathrm{d}}\mathop{\mathrm{Tr}}\{ X^TX X^TX\} \\ &= -2 \mathop{\mathrm{Tr}}\{\mathop{\mathrm{d}}(N^TX^TX)\} + \mathop{\mathrm{Tr}}\{ \mathop{\mathrm{d}}(X^TXX^TX)\} \\ &= - 2 \mathop{\mathrm{Tr}}\{ N^T(\mathop{\mathrm{d}}X^T) X + N^T X^T (\mathop{\mathrm{d}}X)\} \\ &\space+\mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}X^T)XX^TX + X^T (\mathop{\mathrm{d}}X)X^TX + X^TX (\mathop{\mathrm{d}}X^T)X + X^TXX^T (\mathop{\mathrm{d}}X)\} \\ &= - 2 \mathop{\mathrm{Tr}}\{ N^T(\mathop{\mathrm{d}}X^T) X \} -2 \mathop{\mathrm{Tr}}\{ N^T X^T (\mathop{\mathrm{d}}X)\} + \mathop{\mathrm{Tr}}\{ (\mathop{\mathrm{d}}X^T)XX^TX \} \\ &+ \mathop{\mathrm{Tr}}\{X^T (\mathop{\mathrm{d}}X)X^TX \} + \mathop{\mathrm{Tr}}\{X^TX (\mathop{\mathrm{d}}X^T)X \} +\mathop{\mathrm{Tr}}\{ X^TXX^T (\mathop{\mathrm{d}}X)\} \\ &= -2 \mathop{\mathrm{Tr}}\{ X^T (\mathop{\mathrm{d}}X) N\} - 2 \mathop{\mathrm{Tr}}\{ N^T X^T (\mathop{\mathrm{d}}X)\} + \mathop{\mathrm{Tr}}\{ X^TXX^T (\mathop{\mathrm{d}}X)\} \\ & + \mathop{\mathrm{Tr}}\{ X^TX X^T (\mathop{\mathrm{d}}X)\} + \mathop{\mathrm{Tr}}\{ X^T(\mathop{\mathrm{d}}X) X^TX\} + \mathop{\mathrm{Tr}}\{X^TX X^T (\mathop{\mathrm{d}}X) \} \\ &= -2 \mathop{\mathrm{Tr}}\{ N X^T (\mathop{\mathrm{d}}X)\} -2 \mathop{\mathrm{Tr}}\{ N^T X^T (\mathop{\mathrm{d}}X)\} + 4 \mathop{\mathrm{Tr}}\{ X^TX X^T (\mathop{\mathrm{d}}X)\} \\ &= \mathop{\mathrm{Tr}}\{ -2 NX^T (\mathop{\mathrm{d}}X) -2 N^T X^T (\mathop{\mathrm{d}}X) + 4 X^TXX^T (\mathop{\mathrm{d}}X)\} \\ &= \mathop{\mathrm{Tr}}\{ [4 X^TX X^T - 2 (N+N^T)X^T](\mathop{\mathrm{d}}X)\} \\ \end{aligned}\]

Therefore in numerator layout:

\[\frac{\partial \| N - X^T X \|^2_F}{\partial X} = 4 X^TX X^T -2 (N+N^T) X^T\]

or in denominator layout (taking transpose):

\[\frac{\partial \| N - X^T X \|^2_F}{\partial X} = 4 X X^T X - 2 X(N^T+N)\]

Now since \(N^T=N\) and \(C^T=C\) we have:

\[\begin{aligned} \frac{\partial f}{\partial X}&=\frac{1}{2}(CXN+C^TXN^T)+\frac{\mu}{4}[4 X X^T X - 2 X(N^T+N)] \\ &= \frac{1}{2}(CXN+C^TXN^T)+\mu X X^T X - \frac{\mu}{4} \cdot 4 XN \\ &= \frac{1}{2}(CXN+C^TXN^T)+\mu X X^T X - \mu XN \\ &= \frac{1}{2}(2CXN) +\mu XX^TX - \mu XN \\ &= CXN + \mu XX^TX - \mu XN \end{aligned}\]

Back to top ↑