How to find Maxima and Minima of an n-th Dimensional Function

Local Maximum, Local Minimum, and Extremum

Consider an $n$-dimensional function $z=f(\mathbf{x})$ defined by $f: D\subseteq \R^n \to \R$, where $\mathbf{x}=(x_1, x_2, \ldots, x_n)$ is an $n$-tuple of real numbers. Also consider a point $\mathbf{a} = (a_1, a_2, \ldots, a_n)$ defined in the domain $D\subseteq\R^n$.

1. Local Maximum. The function $z=f(\mathbf{x})$ attains its local maximum at $\mathbf{a}$ if there exists a neighborhood $N_\varepsilon(\mathbf{a})\in D$ of $\mathbf{a}$ such that $f(\mathbf{x})\leq f(\mathbf{a})$, $\forall \mathbf{x}\in N_\varepsilon(\mathbf{a}) \backslash \{\mathbf{a}\}$. We call the plural of maximum maxima.
2. Local Minimum. The function $z=f(\mathbf{x})$ attains its local minimum at $\mathbf{a}$ if there exists a neighborhood $N_\varepsilon(\mathbf{a})\in D$ of $\mathbf{a}$ such that $f(\mathbf{x})\geq f(\mathbf{a})$, $\forall \mathbf{x}\in N_\varepsilon(\mathbf{a}) \backslash \{\mathbf{a}\}$. We call the plural of minimum minima.
3. Extremum. The function $z=f(\mathbf{x})$ attains its maximum, or minimum at point $\mathbf{a}$ is generally called an extremum at point $\mathbf{a}$. Point $\mathbf{a}$ might sometimes be called the extreme point of the function and $f(\mathbf{a})$ is called the extreme value of the function $z=f(\mathbf{x})$. We call the plural of extremum extrema.

Stationary Points, Critical Points, and Saddle Points

Consider an $n$-dimensional function $z=f(\mathbf{x})$, $f: D\subseteq \R^n \to \R$, where $\mathbf{x}=(x_1, x_2, \ldots, x_n)$ is an $n$-tuple of real numbers. Assume that the function is defined at a point $\mathbf{a} = (a_1, a_2, \ldots, a_n)\in D\subseteq\R^n$.

1. Stationary Point. The points where all of the partial derivatives of $f$ equals 0 is called the stationary points of $f$. That is,
   $$\pd{f}{x_i}(a_1,a_2,...,a_n)=\pd{f(\mathbf{a})}{x_i}=0\quad\text{for\;all}\; i=1,2,...,n$$
2. Critical Point. The point $\mathbf{a}=(a_1, a_2, \ldots ,a_n)\in D$ is called a critical point if at least one of the partial derivatives of $f$ at that point equals 0 or undefined.
3. Saddle Point. The point $\mathbf{a}=(a_1,a_2,\ldots,a_n)\in D$ is called a saddle point if it is a stationary point and for all $V_\varepsilon(\mathbf{a})$, and there exists $\mathbf{y}=(y_1, y_2,\ldots,y_n)\in V_\varepsilon$ and $\mathbf{z}=(z_1,z_2,\ldots,z_n)\in V_\varepsilon$ such that $$f(\mathbf{y})< f(\mathbf{a})< f(\mathbf{z}).$$

The Hessian Matrix and the Sylvester Criterion

The Hessian Matrix. Consider a function $z=f(\mathbf{x})$ and $\mathbf{x}$ defined as before, and let $f$ be a function whose second partial derivatives are continuous on its domain $D$. Consider also a point $\mathbf{a}=(a_1,a_2,...,a_n)\in D\subseteq \R^n$. By then, the matrix

is called the Hessian matrix of the function $f$ at point $\mathbf{a}$. Since $f$ has continuous second partial derivatives, $H_f(M)$ must be a symmetric square matrix; that is, $H_f(\mathbf{a})=H_f(\mathbf{a})^{\mathsf{T}}$.

Quadratic Form. A quadratic form, to put simply, is just a second-degree multi-variable polynomial such as

One interesting about quadratic forms is that every symmetric matrix defines a quadratic form; that is, given that $\mathbf{A} = [a]_{ij}$, the polynomial

will results in a quadratic form without requiring us to combining terms. Therefore, $H_f(\mathbf{a})$ can most definitely be treated as a matrix associated with a quadratic form.

Definite Matrix. A symmetric real matrix $\mathbf{A}$ is called positive definite if the associated quadratic form $$\phi( \mathbf{x})=\mathbf{x}^{\mathsf{T}}\mathbf{A}\mathbf{x}$$ is positive for every column ($n\times 1$), non-zero vector $\mathbf{x}$ in $\R^n$. Similarly, if $\phi(\mathbf{x})$ only yields negative values, then $\mathbf{A}$ is called negative definite. Finally, if $\phi$ produces both negative and positive values, then $\mathbf{A}$ is said to be indefinite. There are also other cases such as positive semi-definite or negative semi-definite matrices. In general,

1. $\mathbf{A}$ is positive-definite if and only if $\mathbf{x}^{\mathsf{T}}\mathbf{A}\mathbf{x}>0,\;\forall \mathbf{x}\in\R^n \backslash\{\mathbf{0}\}$.
2. $\mathbf{A}$ is negative-definite if and only if $\mathbf{x}^{\mathsf{T}}\mathbf{A}\mathbf{x}< 0,\;\forall \mathbf{x}\in\R^n \backslash\{\mathbf{0}\}$.
3. $\mathbf{A}$ is semi-positive-definite if and only if $\mathbf{x}^{\mathsf{T}}\mathbf{A}\mathbf{x}\geq 0,\;\forall \mathbf{x}\in\R^n$.
4. $\mathbf{A}$ is semi-negative-definite if and only if $\mathbf{x}^{\mathsf{T}}\mathbf{A}\mathbf{x}\leq 0,\;\forall \mathbf{x}\in\R^n$.

Sub-matrices. Given an $n\times n$ matrix $\mathbf{A}$ with $(i,j)$-th entry, where $a_{ij}=a_{ji}$. Let $A^{(k)}$ denote the $k\times k$ submatrix taken from the top left corner of $\mathbf{A}$. These matrices are called major sub-matrices of $A$. That is,

In particular,

Let $\Delta_k=\det(\mathbf{A}^{k})\;,1\leq k\leq n$. For instance, $\Delta_1=a_{11}$ and $\Delta_n =\det(\mathbf{A})$.

The Maximum/Minimum Algorithm for an n-Variable Function

The General Case

The General Second Derivative Test. To identify all the stationary (and critical points) of $f(\mathbf{x})$ and test if they are the extrema of $f$, we follow the following procedure:

1. Find $D(f)$, the domain of $f(\mathbf{x})$. This is for us to determine whether to keep a critical point or not.
2. Solve the equation $\nabla f(\mathbf{x})=\mathbf{0}$; that is, solve the system of $n$-equations
   \[ \left\{ \begin{array}{cc} \pd{f(\mathbf{x})}{x_1}=f_{x_1}(x_1,x_2,...,x_n)=0 \\[0.1em] \pd{f(\mathbf{x})}{x_2}=f_{x_2}(x_1,x_2,...,x_n)=0 \\[0.1em] \vdots \\[-0.1em] \pd{f(\mathbf{x})}{x_n}=f_{x_n}(x_1,x_2,...,x_n)=0 \end{array} \right., \]
   to find all stationary points of $f(\mathbf{x})$ (if they exist). Label these points, for instance, $\mathbf{a}_0=(a_1^0, \;a_2^0,...,\;a_n^0)$, $\mathbf{a}_1=(a_1^1,\;a_2^1,\ldots,\;a_n^1),\ldots$. Next, find all points where the partial derivatives $f_{x_1}(\mathbf{x}), f_{x_2}(\mathbf{x}),...,f_{x_n}(\mathbf{x})$ are undefined, we now have the set of all critical points of $f(\mathbf{x})$.
   
   Remark. One reminder to note is that all critical points that need to be considered must lie inside the domain $D$. If not, just discard them.
3. If $\mathbf{a_0}$ is a stationary point, calculate its Hessian matrix at $\mathbf{a_0}$; that is, find $H_f(\mathbf{a_0})$. After that, apply Sylvester's Criterion on $H_f(\mathbf{a_0})$ by calculating the values of $\Delta_1,\Delta_2,...,\Delta_n$ at $\mathbf{a_0}$.
   1. If $\mathrm{H}_f(\mathbf{a_0})\succ 0$, then $\mathbf{a_0}$ is a minimum of the function $f(\mathbf{x})$.
   2. If $\mathrm{H}_f(\mathbf{a_0})\prec 0$, then $\mathbf{a_0}$ is a maximum of the function $f(\mathbf{x})$.
   3. If $\det (\Delta_k)\neq 0,\;1\leq k\leq n,$ but $\mathrm{H}_f(\mathbf{a_0})$ itself is neither positive nor negative definite, then $\mathbf{a_0}$ is not an extremum. A similar statement is that if there exist two points $\mathbf{y}=(y_1,y_2,...,y_n)$ and $\mathbf{z}=(z_1,z_2,...,z_n)$ such that $\mathbf{y}H_f(\mathbf{a_0})\mathbf{y}^{\mathsf{T}}>0$ and $\mathbf{z}H_f(\mathbf{a_0})\mathbf{z}^{\mathsf{T}}< 0$, then $\mathbf{a_0}$ is not an extremum of $f(\mathbf{x})$.
   4. If there exists a point $\mathbf{y}=(y_1,y_2,...,y_n)$ such that $\mathbf{y}^{\mathrm{T}}\mathrm{H}_f(\mathbf{a_0})\mathbf{y}=0$; that is, if there is at least one major sub-matrix of $\mathrm{H}_f(\mathbf{a_0})$ equals to 0, then the test is inconclusive.
4. Repeat the same process again for the stationary point $\mathbf{a_1}$.

The Special Case: $n=2$

Example Problems