Method of Lagrange multipliers

-

We have studied how to identify extreme values of two-variable functions \(f(x,y)\). In practice, we may have additional constraints. For example,

Find the extreme values of \(f(x,y)\) subject to the constraint \(g(x,y) = 0\).

We can use the method of Lagrange multipliers to solve this type of problem.



Let's consider the following example.

Problem. If \((x,y)\) moves on the unit circle \(x^2 + y^2 = 1\), find the extreme values of \(f(x,y) = x^2 + xy + y^2\). □

This problem can be restated as follows: 

Problem (restated). Let \(g(x,y) = x^2 + y^2 - 1\). Find the extreme values of \(f(x,y) = x^2 + xy + y^2\) subject to the constraint \(g(x,y) = 0\). □

Let's solve this problem using an "explicit" method.

Solution 1 (Explicit method). Consider the implicit function \(y = \sqrt{1 - x^2}\) of \(g(x,y)\) on the open interval \((-1, 1)\) and substitute it to \(f(x, y)\) to have

\[h(x) = f(x,y(x)) = x^2 + x\sqrt{1-x^2} + (\sqrt{1-x^2})^2 = x\sqrt{1-x^2}+1.\]

Since

\[h'(x) = \frac{1 - 2x^2}{\sqrt{1 - x^2}},\]

\(h(x)\) has a local minimum at \(x = -\frac{1}{\sqrt{2}}\) and a local maximum at \(x = \frac{1}{\sqrt{2}}\). Therefore, \(f(x,y)\) has a local minimum value \(\frac{1}{2}\) at point \(\left(-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right)\), and a local maximum value \(\frac{3}{2}\) at point \(\left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right)\) on the unit circle.

Similarly, using the implicit function \(y = -\sqrt{1-x^2}\) on the open interval \((-1,1)\), we can define \(h(x) = -x\sqrt{1-x^2} + 1\) and find that \(f(x,y)\) has a local minimum value \(\frac{1}{2}\) at \(\left(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}\right)\) and a local maximum value \(\frac{3}{2}\) at \(\left(-\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}\right)\).

Next, we examine the neighbors of the points \((1, 0)\) and \((-1, 0)\). In these cases, we consider the implicit functions \(x = \sqrt{1 - y^2}\) (the branch passing through \((1,0)\)) and \(x = -\sqrt{1 - y^2}\) (the branch passing through \((-1, 0)\)), and find the same points as above to give local minimum and maximum.

In summary, \(f(x,y)\) has a local minimum value \(\frac{1}{2}\) at \((x,y) = \left(\pm\frac{1}{\sqrt{2}}, \mp\frac{1}{\sqrt{2}}\right)\), and a local maximum value \(\frac{3}{2}\) at \((x,y) = \left(\pm\frac{1}{\sqrt{2}}, \pm\frac{1}{\sqrt{2}}\right)\). □

The above solution is explicit and easy to understand. However, this method applies only when the implicit functions of \(g(x,y) = 0\) can be obtained explicitly. If that is not the case, the following method of Lagrange (undetermined) multipliers provides an effective approach. We first show how to solve the above problem using the method of Lagrange multipliers. We will prove the method in general later.

Solution 2 (Lagrange multiplier). First, let us define a new function

\[F(x,y,\lambda) = f(x,y) - \lambda g(x,y) = (x^2 + xy + y^2) - \lambda(x^2 + y^2 - 1)\]

where \(\lambda\) is a new variable (undetermined constant) called the Lagrange multiplier. Note that we have \(F(x,y) = f(x,y)\) if the constraint \(g(x,y) = 0\) is satisfied. We now consider the extreme value problem of this function.

Next, we differentiate \(F(x,y)\) with respect to \(x\) and \(y\) and solve

\[\begin{eqnarray} F_x(x,y,\lambda) &=& 2(1-\lambda)x + y = 0,\tag{Eq:lagx}\\ F_y(x,y,\lambda) &=& x + 2(1-\lambda)y = 0.\tag{Eq:lagy} \end{eqnarray}\]

Furthermore,

\[F_{\lambda}(x,y,\lambda) = -(x^2 + y^2 - 1) = 0 \tag{Eq:lambda}\]

should also be satisfied. The latter equation implies, in particular, that \((x,y) \neq (0,0)\) (i.e., \(x\) and \(y\) cannot be simultaneously equal to 0). However, from (Eq:lagx) and (Eq:lagy) above, if \(x = 0\), then \(y = 0\) and vice versa, which is a contradiction. Thus, both \(x\) and \(y\) are non-zero. Eliminating, for example, \(y\) from (Eq:lagx) and (Eq:lagy) and some rearrangement gives

\[x(2\lambda - 1)(2\lambda - 3) = 0.\]

Since \(x \neq 0\), we have \(\lambda = \frac{1}{2}, \frac{3}{2}\). For each of these values of \(\lambda\), we solve the simultaneous equations (Eq:lagx) and (Eq:lagy) for \(x\) and \(y\):

  • For \(\lambda = \frac{1}{2}\), we have \(x = -y\).
  • For \(\lambda = \frac{3}{2}\), we have  \(x = y\).

Combining with (Eq:lambda), we have

  • For \(\lambda = \frac{1}{2}\), we have \((x,y) = \left(\pm\frac{1}{\sqrt{2}}, \mp\frac{1}{\sqrt{2}}\right)\).
  • For \(\lambda = \frac{3}{2}\), we have  \((x,y) = \left(\pm\frac{1}{\sqrt{2}}, \pm\frac{1}{\sqrt{2}}\right)\).
In order to see if \(f(x,y)\) indeed has extreme values at these points, let us examine its behavior in the neighbor of these points \((a,b) = \left(\pm\frac{1}{\sqrt{2}}, \mp\frac{1}{\sqrt{2}}\right), \left(\pm\frac{1}{\sqrt{2}}, \pm\frac{1}{\sqrt{2}}\right)\).

Since \(b\neq 0\), each point on the unit circle \(g(x,y) = 0\) can be expressed as \((x, \varphi(x))\) where \(y = \varphi(x)\) is an implicit function of \(g(x,y) = 0\). Thus, we need to solve the extreme value problem of 
\[h(x) = f(x, \varphi(x)).\]
Differentiating the equation of constraint \(g(x,\varphi(x)) = 0\), or
\[x^2 + \{\varphi(x)\}^2 - 1 = 0\]
with respect to \(x\), we have
\[2x + 2\varphi(x)\varphi'(x) = 0.\]
From this, we have
\[\varphi'(x) = -\frac{x}{\varphi(x)}.\]
Using the multivariate chain rule, we obtain
\[\begin{eqnarray} h'(x) &=& f_x(x,\varphi(x)) + f_y(x,\varphi(x))\left(-\frac{x}{\varphi(x)}\right) = \frac{1-2x^2}{\varphi(x)},\\ h''(x) &=& \frac{-4x\varphi(x) - (1-2x^2)\varphi'(x)}{\{\varphi(x)\}^2} =\frac{-2x^3 + [1-4\{\varphi(x)\}^2]x}{\{\varphi(x)\}^3}. \end{eqnarray}\]

If \((x, \varphi(x)) = \left(\pm\frac{1}{\sqrt{2}}, \mp\frac{1}{\sqrt{2}}\right)\) (corresponding to \(\lambda = \frac{1}{2}\)),  we have \(h'(x) = 0\) and \(h''(x) = 4 > 0\) so \(h(x)\) has a local minimum value \(\frac{1}{2}\).

If \((x, \varphi(x)) = \left(\pm\frac{1}{\sqrt{2}}, \pm\frac{1}{\sqrt{2}}\right)\) (corresponding to \(\lambda = \frac{3}{2}\)),  we have \(h'(x) = 0\) and \(h''(x) = -4 < 0\) so \(h(x)\) has a local maximum value \(\frac{3}{2}\). □

It should be stressed that using the method of Lagrange multipliers, we did not need to know the explicit form of the implicit function \(\varphi(x)\). That is, the implicit function remains, in fact, implicit. The mere existence of the implicit function \(\varphi(x)\) is sufficient. Thus, this method is applicable even when the implicit function is hard (if possible) to find.

Remark. In general, constrained extreme value problems can be solved by the following procedure:
  1. Enumerate the possible extreme points by the method of Lagrange multipliers.
  2. For each possible extreme point, consider the implicit function in the neighbor of the point and evaluate the first and second derivatives.
□

We now give the method of Lagrange multipliers as a theorem.

Theorem [Method of Lagrange multipliers]

Let \(f(x,y)\) and \(g(x,y)\) be functions of class \(C^1\). With a new variable \(\lambda\), let us define
\[F(x,y,\lambda) = f(x,y) - \lambda g(x,y).\]
Suppose the point \((a,b)\) satisfies the following conditions:
  1. The function \(f(x,y)\) has an extreme value at \((x,y) = (a,b)\) subject to the constraint \(g(x,y) = 0\) (in particular, \(g(a,b) = 0\));
  2. \((a,b)\) is a regular point of \(g(x,y) = 0\) (i.e., it is not the case that \(g_x(a,b) = g_y(a,b) = 0\)).
Then, there exists \(\alpha \in \mathbb{R}\) such that
\[F_x(a,b, \alpha) = F_y(a,b, \alpha) = F_{\lambda}(a,b, \alpha) = 0.\]
Proof. By condition (1), \(F_{\lambda}(a,b,\alpha) = -g(a,b) = 0\). Thus, it suffices to show that there exists \(\alpha \in\mathbb{R}\) such that
\[\begin{eqnarray} F_x(a,b,\alpha) &=& f_x(a,b) - \alpha g_x(a,b) = 0,\tag{Eq:lagax}\\ F_y(a,b,\alpha) &=& f_y(a,b) - \alpha g_y(a,b) = 0.\tag{Eq:lagay} \end{eqnarray}\]

By condition (2), \(g_x(a,b) \neq 0\) or \(g_y(a,b) \neq 0\). Without loss of generality (what does this mean?), we may assume \(g_y(a,b) \neq 0\). By the Implicit Function Theorem, there exists a function \(y=\varphi(x)\) in a neighbor of \(x = a\) such that \(b = \varphi(a)\) and \(g(x, \varphi(x)) = 0\). In the neighbor of \((a,b)\), any point satisfying \(g(x,y) = 0\) is of the form \((x, \varphi(x))\). Therefore, we need to solve the extreme value problem of the univariate function \(f(x, \varphi(x))\) in the neighbor of \(x=a\). Note that
\[\varphi'(x) = -\frac{g_x(x,\varphi(x))}{g_y(x,\varphi(x))}.\]
Differentiating \(f(x,\varphi(x))\) with respect to \(x\), we have
\[f_x(x,\varphi(x)) + f_y(x,\varphi(x))\varphi'(x) = f_x(x,\varphi(x)) - f_y(x,\varphi(x))\frac{g_x(x,\varphi(x))}{g_y(x,\varphi(x))}.\]
By assumption, \(f(x,\varphi(x))\) has an extreme value at \(x = a\),
\[f_x(a, \varphi(a)) = f_y(a, \varphi(a)) = 0.\]
But \(b = \varphi(a)\) so that
\[f_x(a,b) - f_y(a,b)\frac{g_x(a,b)}{g_y(a,b)} = 0.\]
It follows that
\[f_x(a,b)g_y(a,b) - f_y(a,b) g_x(a,b) = 0.\]
Based on this, let us define
\[\alpha = \frac{f_y(a,b)}{g_y(a,b)}.\]
Clearly, this \(\alpha\) satisfies (Eq:lagax}) and (Eq:lagay). ■







Comments

Post a Comment

Popular posts from this blog

Open sets and closed sets in \(\mathbb{R}^n\)

-
Image
Open sets In \(\mathbb{R}\), we have the notion of an open interval such as \((a, b) = \{x \in \mathbb{R} | a < x < b\}\). We want to extend this idea to apply to \(\mathbb{R}^n\). We also introduce the notions of bounded sets and closed sets in \(\mathbb{R}^n\). Recall that the \(\varepsilon\)-neighbor of a point \(x\in\mathbb{R}^n\) is defined as \(N_{\varepsilon}(x) = \{y \in \mathbb{R}^n | d(x, y) < \varepsilon \}\) where \(d(x,y)\) is the distance between \(x\) and \(y\). Definition (Open set) A subset \(U\) of \(\mathbb{R}^n\) is said to be an  open set  if the following holds: \[\forall x \in U ~ \exists \delta > 0 ~ (N_{\delta}(x) \subset U).\tag{Eq:OpenSet}\] That is, for every point in an open set \(U\), we can always find an open ball centered at that point, that is included in \(U\). See the following figure. Perhaps, it is instructive to see what is  not  an open set. Negating (Eq:OpenSet), we have \[\exists x \in U ~ \forall \delta > 0 ~ (N_{\delta}(x) \not
Read more

Euclidean spaces

-
Image
We would like to study multivariate functions (i.e., functions of many variables), continuous multivariate functions in particular. To define continuity, we need a measure of "closeness" between points. One measure of closeness is the Euclidean distance. The set \(\mathbb{R}^n\) (with \(n \in \mathbb{N}\)) with the Euclidean distance function is called a Euclidean space. This is the space where our functions of interest live. The real line is a geometric representation of \(\mathbb{R}\), the set of all real numbers. That is, each \(a \in \mathbb{R}\) is represented as the point \(a\) on the real line. The coordinate plane , or the \(x\)-\(y\) plane , is a geometric representation of \(\mathbb{R}^2\), the set of all pairs of real numbers. Each pair of real numbers \((a, b)\) is visualized as the point \((a, b)\) in the plane. Remark . Recall that \(\mathbb{R}^2 = \mathbb{R}\times\mathbb{R} = \{(x, y) | x, y \in \mathbb{R}\}\) is the Cartesian product of \(\mathbb{R}\) with i
Read more

Newton's method

-
Image
Newton's method (the Newton-Raphson method) is a very powerful numerical method for solving nonlinear equations.  Suppose we'd like to solve a non-linear equation \(f(x) = 0\), where \(f(x)\) is a (twice) differentiable non-linear function. Newton's method generates a sequence of numbers \(c_1, c_2, c_3, \cdots\) that converges to a solution of the equation. That is, if \(\alpha\) is a solution (i.e., \(f(\alpha) = 0\)) then,  \[\lim_{n\to\infty}c_n = \alpha,\] and this sequence \(\{c_n\}\) is generated by a series of linear approximations of the function \(f(x)\). Theorem (Newton's method) Let \(f(x)\) be a function that is twice differentiable on an open interval \(I\) that contains the closed interval \([a, b]\) (i.e., \([a,b]\subset I\)) and satisfy the following conditions: \(f(a) < 0\) and \(f(b) > 0\); For all \(x\in [a, b]\), \(f'(x) > 0\) and \(f''(x) > 0\). Let us define the sequence \(\{c_n\}\) by \[ \begin{eqnarray} c_1 &
Read more