Numerator Layout of Matrix Calculus

Numerator Layout

	$y \in R^{1 \times 1}$	$y \in R^{n \times 1}$	$y \in R^{n \times m}$
$x \in R^{1 \times 1}$	$\frac{\partial y}{\partial x}$	$(\frac{\partial y}{\partial x})_{i, 1} = \frac{\partial y _{i}}{\partial x}$	$(\frac{\partial y}{\partial x})_{i, j} = \frac{\partial y _{i, j}}{\partial x}$
$x \in R^{m \times 1}$	$(\frac{\partial y}{\partial x})_{1, j} = \frac{\partial y}{\partial x _{j}}$	$(\frac{\partial y}{\partial x})_{i, j} = \frac{\partial y _{i}}{\partial x _{j}}$
$x \in R^{m \times n}$	$(\frac{\partial y}{\partial x})_{i, j} = \frac{\partial y}{\partial x _{j, i}}$

Some texts use $\frac{\partial y}{\partial x ^{T}}$ to represent $(\frac{\partial y}{\partial x})^{T}$ .

Suppose $x \in R^{m \times 1}$ and $y \in R$ s.t. $y = f (x)$ . Then the gradient is defined as follows.

\nabla f (x) = (\frac{\partial y}{\partial x})^{T}

Also we can define the Hessian as follows.

\nabla^{2} f (x) = (\frac{\partial}{\partial x} (\frac{\partial y}{\partial x})^{T})^{T}

Because of the Clairaut’s theorem, usually we use the following form.

\nabla^{2} f (x) = \frac{\partial}{\partial x} (\frac{\partial y}{\partial x})^{T}

Suppose $x \in R^{n \times 1}$ and $y \in R^{m \times 1}$ s.t. $y = f (x)$ . Then the Jacobian matrix is defined as follows.

J_{f} (x) = \frac{\partial y}{\partial x}

For any possible combination of shapes for $x, y, z$ , the following chain rule holds.

\frac{\partial z}{\partial x} = \frac{\partial z}{\partial y} \frac{\partial y}{\partial x}