Directional derivative

The directional derivative $D_{\hat{\mathbf{u}}}f$ of a scalar function $f(x,y,z)$ at a point, in the direction of a unit vector $\hat{\mathbf{u}}$, is the rate of change of $f$ along that direction:

$D_{\hat{\mathbf{u}}}f=\nabla f\cdot \hat{\mathbf{u}}=|\nabla f|\cos \theta ,$

where $\theta$ is the angle between $\nabla f$ (gradient) and $\hat{\mathbf{u}}$. Use the dot-product form to compute; the $|\nabla f|\cos \theta$ form gives the geometry.

What it means

The partial derivative $\partial f/\partial x$ tells you how $f$ changes when you move in $+\hat{\mathbf{x}}$. The directional derivative generalizes this to any direction $\hat{\mathbf{u}}$: it tells you how fast $f$ changes when you walk in the direction $\hat{\mathbf{u}}$, per unit distance traveled.

For $\hat{\mathbf{u}}=\hat{\mathbf{x}}$: $D_{\hat{\mathbf{x}}}f=\partial f/\partial x$. For $\hat{\mathbf{u}}=\hat{\mathbf{y}}$: $D_{\hat{\mathbf{y}}}f=\partial f/\partial y$. For arbitrary $\hat{\mathbf{u}}=u_x\hat{\mathbf{x}}+u_y\hat{\mathbf{y}}+u_z\hat{\mathbf{z}}$:

$D_{\hat{\mathbf{u}}}f=u_x\frac{\partial f}{\partial x}+u_y\frac{\partial f}{\partial y}+u_z\frac{\partial f}{\partial z}=\nabla f\cdot \hat{\mathbf{u}}.$

The partial derivatives along the three coordinate axes are components of the gradient; the directional derivative in an arbitrary direction is the projection of the gradient onto that direction.

Why $\hat{\mathbf{u}}$ must be a unit vector

The formula $D_{\hat{\mathbf{u}}}f=\nabla f\cdot \hat{\mathbf{u}}$ assumes $|\hat{\mathbf{u}}|=1$, i.e. “rate per unit distance.” Plug a non-unit vector $\mathbf{v}$ into the same formula and the result $\nabla f\cdot \mathbf{v}$ comes out scaled by $|\mathbf{v}|$. So for a non-unit $\mathbf{v}$:

$D_{\mathbf{v}}f=\nabla f\cdot \frac{\mathbf{v}}{|\mathbf{v}|}.$

Forgetting to normalize is an easy mistake here.

Geometric interpretation: steepest ascent

Writing $D_{\hat{\mathbf{u}}}f=|\nabla f|\cos \theta$:

• Maximized at $\theta =0$ ($\hat{\mathbf{u}}$ pointing along $\nabla f$): $D_{\hat{\mathbf{u}}}f=|\nabla f|$. This is the steepest-ascent direction.
• Zero at $\theta =\pi /2$ ($\hat{\mathbf{u}}$ perpendicular to $\nabla f$): $D_{\hat{\mathbf{u}}}f=0$. Moving along a level surface (where $f$ is constant) means $\nabla f\perp \hat{\mathbf{u}}$.
• Minimized at $\theta =\pi$ ($\hat{\mathbf{u}}$ antiparallel to $\nabla f$): $D_{\hat{\mathbf{u}}}f=-|\nabla f|$. The steepest-descent direction, which is what Gradient descent runs on.

The gradient is the steepest-ascent vector; its magnitude is the steepest slope; the directional derivative in any other direction is just the projection.

Worked example

Take $f(x,y,z)=x^2+2xy+z^3$.

Gradient:

$\nabla f=(2x+2y)\hat{\mathbf{x}}+2x\hat{\mathbf{y}}+3z^2\hat{\mathbf{z}}.$

At point $(1,1,1)$:

$\nabla f{|}_{(1,1,1)}=4\hat{\mathbf{x}}+2\hat{\mathbf{y}}+3\hat{\mathbf{z}}.$

Directional derivative in the direction of $\mathbf{v}=\hat{\mathbf{x}}+\hat{\mathbf{y}}-\hat{\mathbf{z}}$. First normalize: $|\mathbf{v}|=\sqrt{3}$, so $\hat{\mathbf{u}}=(\hat{\mathbf{x}}+\hat{\mathbf{y}}-\hat{\mathbf{z}})/\sqrt{3}$.

$D_{\hat{\mathbf{u}}}f=\nabla f\cdot \hat{\mathbf{u}}=\frac{4+2-3}{\sqrt{3}}=\frac{3}{\sqrt{3}}=\sqrt{3}\approx \mathrm{1.73.}$

Interpretation: walking from $(1,1,1)$ in the direction $(1,1,-1)/\sqrt{3}$ at unit speed, $f$ increases at rate $\sqrt{3}$ per unit distance.

The maximum possible rate at this point: $|\nabla f|=\sqrt{16+4+9}=\sqrt{29}\approx 5.39$, in the direction ${\hat{\mathbf{u}}}_{\max }=(4,2,3)/\sqrt{29}$.

Connection to chain rule

If $\mathbf{r}(t)$ is a parametrized curve passing through a point at $t=0$ with $\dot{\mathbf{r}}(0)=\mathbf{v}$, then

$\frac{d}{dt}f(\mathbf{r}(t)){|}_{t=0}=\nabla f\cdot \mathbf{v}.$

This is the chain rule. The directional derivative is the special case where $\mathbf{v}=\hat{\mathbf{u}}$ is a unit vector, measuring rate of change “per unit arc length” along the curve, at the starting point. Otherwise the rate is scaled by the speed $|\mathbf{v}|$.

In machine learning

In gradient-based optimization, the “step direction” question is: in which $\hat{\mathbf{u}}$ should we update parameters to decrease the loss $J(\mathbf{w})$ fastest? Answer: $\hat{\mathbf{u}}=-\nabla J/|\nabla J|$. The negative gradient direction has directional derivative $-|\nabla J|$, the most-negative possible value. See Gradient descent.

In adaptive methods (Adam, RMSprop, etc.), the effective $\hat{\mathbf{u}}$ at each step is a messier function of past gradients and squared gradients, but it’s the same idea: move in the direction of greatest local decrease.

In electromagnetics

The directional derivative shows up wherever you want the rate of change of a field along a specific direction, usually along a path, a boundary, or a streamline. The line integral ${\int }_C\mathbf{E}\cdot d\mathbf{l}$ can be viewed as integrating $D_{\hat{\mathbf{t}}}V$ along the path (where $\hat{\mathbf{t}}$ is the unit tangent and $\mathbf{E}=-\nabla V$): the integral accumulates the rate of potential change along the path, giving the total potential difference.