Brunei Math Club

The notion of  uniform continuity  is a ``stronger'' version of (simple) continuity. If a function is uniformly continuous, it is continuous, but the converse does not generally hold (that is, a continuous function may not be uniformly continuous). However, if we restrict a continuous function on a closed interval, it is always uniformly continuous. Definition (Uniform continuity) The function \(f(x)\) on an interval \(I\) is said to be  uniformly continuous  on \(I\) if it satisfies the following condition. For any \(\varepsilon > 0\), there exists \(\delta > 0\), such that, for all \(x, y\in I\), \(|x - y| < \delta\) implies \(|f(x) - f(y)|< \varepsilon\).   In a logical form, this condition is expressed as  \[ \forall \varepsilon > 0, \exists \delta > 0, \forall x,y\in I ~ (|x-y| < \delta \implies |f(x) - f(y)| < \varepsilon).\label{eq:unifcont} \] Remark . Compare the above condition for uniform continuity with the condition...

 Marginal distributions of a multivariate normal distribution are also normal distributions. Let's prove this. See also : Multivariate normal distribution [Wikipedia] The density function of a multivariate normal distribution is given as \[ f(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^n|\Sigma|}}\exp\left[-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^{\top}\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})\right] \] where \(\mathbf{x}\in\mathbb{R}^n\) is the random vector, \(\boldsymbol{\mu}\) is the mean vector and \(\Sigma\) is the covariance matrix. By changing the variables \(\mathbf{x} - \boldsymbol{\mu} \mapsto \mathbf{x}\), we can assume the mean is zero without losing generality. So, in the following, we only consider \[ f(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^n|\Sigma|}}\exp\left[-\frac{1}{2}\mathbf{x}^{\top}\Sigma^{-1}\mathbf{x}\right]. \] We need the following theorems from linear algebra. Theorem 1 Let \(A\) be an \(n\times n\) regular, \(D\) be \(m\times m\) regular, \(B\...

The goal of this post is to prove one of the practical foundations of Riemann-integrals. Theorem A function that is continuous on \([a,b]\) is integrable on \([a,b]\).   See also :  Riemann integral See also : Uniformly continuous functions Proof . Let \(f(x)\) be continuous on \([a,b]\). By the above theorem, \(f(x)\) is uniformly continuous on \([a,b]\). Therefore, for any \(\varepsilon > 0\), there exists a \(\delta > 0\) such that, for all \(x, y\in [a,b]\), \(|x-y| < \delta\) implies \(|f(x) - f(y)| < \frac{\varepsilon}{b - a}\). Let \(\Delta\) be a partition of \([a,b]\) such that \(a = x_0 < x_1 < \cdots < x_{n-1} < x_n = b\) and its mesh is less than \(\delta\) (i.e., \(x_{i+1} - x_{i} < \delta\) for all \(i= 0, 1, \cdots, n-1\)). Then, for each \(i=0, 1, \cdots, n-1\), if \(x, y \in [x_{i}, x_{i+1}]\), then \(|f(x) - f(y)| < \frac{\varepsilon}{b - a}\). Hence, if we define \[ ...

The use of integrals is not limited to computing areas and lengths. Integrals are also helpful for defining new functions. Here, we study two special functions : the Gamma and Beta functions. These functions are widely used in various fields of science and engineering, as well as statistics. Gamma function Lemma For any \(s > 0\), the improper integral \(\int_0^{\infty}e^{-x}x^{s-1}dx\) converges. Proof . Let \(f(x) = e^{-x}x^{s-1}\). We decompose the given integral into \(\int_{0}^{1}f(x)dx\) and \(\int_1^{\infty}f(x)dx\) and show that both of them converge. First, consider \(f(x)e^{\frac{x}{2}} = \frac{x^{s-1}}{e^{\frac{x}{2}}}\) on \([1,\infty)\). If we take \(n\in\mathbb{N}\) such that \(n \geq s-1\), then \[f(x)e^{\frac{x}{2}} = \frac{x^{s-1}}{e^{\frac{x}{2}}} \leq \frac{x^{n}}{e^{\frac{x}{2}}}.\] Applying L'Hôpital's rule \(n\) times, we can see that \(\lim_{x\to\infty}\frac{x^n}{e^{\frac{x}{2}}} = 0\). Hence \(\lim_{x\to\infty}f(x)e^{\frac{x}{2}} = 0\). In particular,...

As an application of integrals, we consider the length of a curve. Specifically, we consider curves in the 2-dimensional space defined parametrically. Let \(x(t)\) and \(y(t)\) be \(C^1\) functions defined on an interval containing the closed interval \([a,b]\). If \(t\) moves in \([a,b]\), the point \((x(t), y(t))\) on \(\mathbb{R}^2\) moves smoothly, drawing a curve. Let us denote this curve by \(C\). Let \(P = (x(a), y(a))\) and \(Q = (x(b), y(b))\) be the end points of the curve \(C\). We want to measure the ``length'' of the curve \(C\). But what is the length of a \emph{curve}, anyway? We do know how to calculate the length of a line segment (Pythagorean theorem). So, let us approximate the curve by line segments. Consider the partition of the closed interval \([a,b]\): \[\Delta: a = t_0 < t_1 < t_2 < \cdots < t_{n-1} < t_n = b.\] Then \(P_0 = (x(t_0), y(t_0)) = P\), \(P_1 = (x(t_1), y(t_1))\), \(P_2 = (x(t_2), y(t_2))\), \(\cdots\), \(P_{n-1} = (x(t_{n-1})...

As we have seen so far, the definite integral \(\int_{a}^{b}f(x)dx\) is defined for the (continuous) function \(f(x)\) on a bounded closed interval \([a,b]\). It is not defined on semi-open intervals such as \((a, b]\) or \([a,b)\), or on unbounded intervals such as \([a,\infty)\) or \((-\infty, \infty)\). Nevertheless, we may extend the definition of definite integrals to deal with such cases. For example, the function \(f(x)\) on \([a, b)\) is not defined on \(x = b\), but if the left limit \(\lim_{t\to b-0}\int_a^tf(x)dx\) exists, we may define it as \(\int_a^bf(x)dx\). Such an extended notion of the integral is called the improper integral . Integration on semi-open intervals For the continuous function \(f(x)\) on the semi-open interval \([a, b)\), if the limit \[\lim_{t\to b - 0}\int_a^tf(x)dx = \lim_{\varepsilon\to +0}\int_a^{b - \varepsilon}f(x)dx\] exists, we say that the improper integral \(\int_a^bf(x)dx\) converges. Similarly, for the continuous function \(f(x)\) on \([a, ...

 Recall that a rational function is a function of the form \(p(x)/q(x)\) where \(p(x)\) and \(q(x)\) are polynomial functions with real coefficients. We now consider the integration of such functions in general. A polynomial function \[f(x) = a_nx^n + a_{n-1}x^{n-1} + \cdots + a_1x + a_0 ~~ (a_0, a_1,\cdots, a_n\in\mathbb{R}),\] has an anti-derivative \[\int f(x)dx = \frac{a_n}{n+1}x^{n+1} + \frac{a_{n-1}}{n}x^{n} + \cdots + \frac{a_1}{2}x^2 + a_0x.\] Thus, the anti-derivative of a polynomial function is a polynomial function. What about rational functions? In general, anti-derivatives of a rational function may not be a rational function, but a sum of rational functions, logarithm, and inverse trigonometric functions. We use the following lemma (proof is omitted) to show this. Lemma (Partial fraction decomposition) Any rational function can be decomposed into a finite sum of rational functions of the following three forms: polynomials, \[\frac{k}{(x+a)^n}\] where \(a, k\in\ma...

 Sometimes, we may simplify integration by using the product rule of differentiation. This technique is called integration by parts. Theorem (Integration by parts) Let \(f(x)\) and \(g(x)\) be differentiable functions on an open interval \(I\). Then,  \(\int f(x)g'(x)dx = f(x)g(x) - \int f'(x)g(x)dx\); For any \(a, b \in I\), \[\int_a^bf(x)g'(x)dx = \left[f(x)g(x)\right]_a^b - \int_a^bf'(x)g(x)dx.\] Proof . By the product rule, \[[f(x)g(x)]' = f'(x)g(x) + f(x)g'(x)\] so \[f(x)g'(x) = [f(x)g(x)]' - f'(x)g(x).\] By integrating both sides, we have the desired results. ■ Example . Let us find \(\int x\cosh x dx\). \[ \begin{eqnarray*} \int x\cosh x dx &=& \int x(\sinh x)'dx \\ &=& x \sinh x - \int 1 \cdot \sinh x dx\\ &=& x \sinh x - \cosh x + C. \end{eqnarray*} \] Example (eg:recur) . Let us study how we can compute \[I_n = \int \frac{dx}{(x^2 + 1)^n}\] for \(n\in \mathbb{N}\). Note \[I_{n} = \int \fr...

 By using the chain rule, we may compute the integral of complicated (composite) functions. Theorem (Integration by substitution) Let \(f(x)\) be a continuous function. Suppose \(x\) is a differentiable function of \(t\) on an open interval \(J\), \(x = x(t)\). Then  \(\int f(x)dx = \int f(x(t))x'(t)dt\).  For any \(a,b\in J\), \(\int_{x(a)}^{x(b)}f(x)dx = \int_a^bf(x(t))x'(t)dt\). Remark . In Part 1, ``\(=\)'' means that the left-hand side is equal to the right-hand side except for a constant term. □ Proof . Let us fix \(a \in J\). For any \(t \in J\), let us define the anti-derivative \[F(x(t)) = \int_{x(a)}^{x(t)}f(x)dx.\] Up to this point, we consider \(F(x(t))\) as a function of \(x(t)\) (the independent variable is \(x(t)\), and \(t\) is just a parameter).  Now regard \(F(x(t))\) as a function of \(t\) and differentiate it with respect to \(t\) using the chain rule. We have \[\frac{d}{dt}F(x(t)) = F'(x(t))x'(t) = f(x(t))x'(t). \] Thus \(F(x(t))\)...

 So far, our discussion on the Riemann integral has been rather abstract. We know its definition and some of its properties. However, we still don't know how to calculate its value for a specific function. There are various techniques to compute integrals. We start from the most basic method based on the notion of anti-derivatives or primitive functions . Definition (Anti-derivative, primitive function) For a function \(f(x)\) on an open interval \(I\), a differentiable function \(F(x)\) on \(I\) is said to be an anti-derivative or primitive function of \(f(x)\) (on \(I\)) if \(F'(x) = f(x)\) holds. Remark .   If \(F(x)\) is an anti-derivative of \(f(x)\), then for any constant \(C\), \(F(x) + C\) is also an anti-derivative of \(f(x)\). The following lemma shows these are the only anti-derivatives. □ Lemma Let \(f(x)\) be a function on an open interval \(I\) with its anti-derivative \(F(x)\). Then any anti-derivative of \(f(x)\) is given as \(F(x) + C\) where \(C\) is ...

By definition, the definite integral is essentially the signed area. From this fact, we can derive a series of properties of the Riemann integral. We also see that continuous functions are Riemann-integrable and prove the Fundamental Theorem of Calculus. For \(a < b < c\), we have \[\int_a^bf(x) dx + \int_b^cf(x) dx = \int_a^cf(x)dx.\tag{eq:intsum}\] This means the sum of two areas is equal to the area of the combined region. If \(a < b\), we adopt the following convention: \[\int_b^af(x)dx = -\int_a^bf(x)dx.\] Then Eq. (eq:intsum) holds irrespective of the order of \(a, b\) and \(c\). Furthermore, the following proposition should be trivial from the definition: Theorem (Linearity of integral) Let \(f(x)\) and \(g(x)\) be functions that are integrable on \([a,b]\). Let \(k, l\in\mathbb{R}\) be constants. Then \(kf(x) +lg(x)\) is also integrable on \([a,b]\) and \[\int_a^b(kf(x) + lg(x))dx = k\int_a^bf(x)dx + l\int_a^bg(x)dx.\] In other words, the integral operation is linear. ...

You may have learned that the integral is the inverse operation of differentiation. Here, we define the integral as the calculation of area. This approach has an advantage that can be easily extended to higher dimensions. Definition (Partition of an interval) The partition of the closed interval \([a,b]\) is a finite sequence \(\Delta = \{x_n\}\) of the form \[a = x_0 < x_1 < x_2 < \cdots < x_{n-1} < x_{n} = b.\] Each \([x_{i},x_{i+1}]\) is called a sub-interval . The mesh or norm of a partition is defined to be the maximum length of the sub-intervals: \[\max\{(x_{i+1} - x_i) \mid {i=0, 1,\cdots, n-1}\}.\] Let \(f(x)\) be a bounded function on \([a,b]\). Let us define the following quantities: \[ \begin{eqnarray} M_i &=& \sup\{f(x)\mid x_{i} \leq x \leq x_{i+1}\},\\ m_i &=& \inf\{f(x)\mid x_{i} \leq x \leq x_{i+1}\}. \end{eqnarray} \] Remark . If \(f(x)\) is continuous on \([a,b]\), \(f(x)\) has maximum and minimum values on each \([x_i, x_{i+...

In many situations, the remainder term in the finite Taylor (Maclaurin) expansion is unimportant. To denote that some terms are not as important as others, we introduce a new notation of great convenience called the "little o."  Using this little o notation, we define the asymptotic expansion, which is (almost) the same as the finite Taylor expansion except that the remainder term is replaced by the little o term. See also : Taylor's theorem Definition (Landau's asymptotic (``little \(o\)'') notation) Let \(f(x)\) and \(g(x)\) be functions defined in the neighbor of \(x=a\). If \[\lim_{x\to a}\frac{f(x)}{g(x)} = 0,\] then, we write \[f(x) = o(g(x)) ~~ (x \to a).\] This ``\(o\)'' is called Landau's symbol} (or ``little o''), and this notation is called Landau's notation (or little-o notation). Remark . When an equation involves Landau's symbol, it does not represent exact equality. □ Example .  \(f(x) = o(1) ~ (x \to a)\) means \(\li...

The essence of differentiation is to approximate an arbitrary function by a linear function. We can extend this idea by using higher-order derivatives to obtain better approximations. Theorem (Taylor's theorem) Let \(f(x)\) be a function of class \(C^{n}\) on an open interval \(I\). Let \(a \in I\). Then for all \(x \in I\), there exists \(c_x\) between \(x\) and \(a\) such that \[ \begin{eqnarray} f(x) &=& f(a) + f'(a)(x-a) + \frac{1}{2}f''(a)(x-a)^2 + \cdots\nonumber\\ && + \frac{1}{(n-1)!}f^{(n)}(a)(x-a)^{n-1} + \frac{1}{n!}f^{(n)}(c_x)(x-a)^{n}.\nonumber\\ \label{eq:taylor} \end{eqnarray} \] Proof . If \(x=a\), then we can set \(c_x = a\) and Eq. (eq:taylor) clearly holds. Suppose \(b \in I\), \(b\neq a\). We need to show \[ \begin{eqnarray} f(b) &=& f(a) + f'(a)(b-a) + \frac{1}{2}f''(a)(b-a)^2 + \cdots\nonumber\\ && + \frac{1}{(n-1)!}f^{(n-1)}(a)(b-a)^{n-1} + \frac{1}{n!}f...

To prove L'Hôpital's rule, we first prove Cauchy's mean value theorem that generalizes the mean value theorem provided earlier. See also : Mean Value Theorem Theorem (Cauchy's Mean Value Theorem) Let \(f(x)\) and \(g(x)\) be functions that are continuous on \([a,b]\) and differentiable on \((a,b)\). Suppose that \(g'(x) \neq 0\) for all \(x \in (a,b)\).  Then, there exists a \(c\in (a,b)\) such that \[\frac{f'(c)}{g'(c)} = \frac{f(b) - f(a)}{g(b) - g(a)}.\] Proof . Note that the function \(g(x)\) satisfies the conditions of the mean value theorem. Thus, there exists a \(d\in(a,b)\) such that \[g'(d) = \frac{g(b) - g(a)}{b - a}.\] Since \(g'(d) \neq 0\) by assumption, it follows that \(g(b) - g(a) \neq 0\). Now let us define \[h(x) = f(x) - \frac{f(b) - f(a)}{g(b) - g(a)}g(x).\] Then \(h(x)\) is continuous on \([a,b]\) and differentiable on \((a,b)\). Moreover, \[h(a) = \frac{f(a)g(b) - f(b)g(a)}{g(b) - g(a)} = h(b).\] Therefore, by Rolle's theore...

We have been using the following formula without proof so far: \[\lim_{x\to 0}\frac{\sin x}{x} = 1.\] In this example, both \(\sin x\) and \(x\) converges to 0 as \(x \to 0\) so we have something like \(\frac{0}{0}\). In general, if \(\lim_{x\to a}f(x) = \lim_{x\to a}g(x) = 0\) or \(\lim_{x\to a}f(x) = \lim_{x\to a}g(x) = \pm\infty\), the limit of the form \(\lim_{x\to a}\frac{f(x)}{g(x)}\) is called an indeterminate form . L'Hôpital's rule provides a convenient way to calculate such limits (The proof will be given in another post). See also : Proof of L'Hôpital's rule Theorem (L'Hôpital's rule (1)) Let \(f(x)\) and \(g(x)\) be differentiable functions on the open interval \((a,b)\) that satisfy the following conditions. \[\lim_{x\to a+0}f(x) = \lim_{x\to a+0}g(x) = 0.\] For all \(x \in (a,b)\), \(g'(x) \neq 0\). The right limit \[\lim_{x\to a+0}\frac{f'(x)}{g'(x)}\] exists. Then, the right limit \(\lim_{x\to a+0}\frac{f(x)}{g(x)}\) exists and \[\lim...

Newton's method (the Newton-Raphson method) is a very powerful numerical method for solving nonlinear equations.  Suppose we'd like to solve a non-linear equation \(f(x) = 0\), where \(f(x)\) is a (twice) differentiable non-linear function. Newton's method generates a sequence of numbers \(c_1, c_2, c_3, \cdots\) that converges to a solution of the equation. That is, if \(\alpha\) is a solution (i.e., \(f(\alpha) = 0\)) then,  \[\lim_{n\to\infty}c_n = \alpha,\] and this sequence \(\{c_n\}\) is generated by a series of linear approximations of the function \(f(x)\). Theorem (Newton's method) Let \(f(x)\) be a function that is twice differentiable on an open interval \(I\) that contains the closed interval \([a, b]\) (i.e., \([a,b]\subset I\)) and satisfy the following conditions: \(f(a) < 0\) and \(f(b) > 0\); For all \(x\in [a, b]\), \(f'(x) > 0\) and \(f''(x) > 0\). Let us define the sequence \(\{c_n\}\) by \[ \begin{eqnarray} c_1 &...