This document provides a summary of mathematical background that will be assumed throughout the course. It is incomplete - the following references will be useful if you need to fill in the gaps.
These tools will provide the machinery we need to talk about stability and prove properties of machine learning architectures. If this is new to you, it's OK to be a little overwhelmed. We will work examples and get plenty of practice using these over and over throughout the semester. If you need help filling in gaps in your math background, be proactive and come to office hours early. Mastering these kinds of manipulations is like learning a foreign language or a musical instrument - it takes practice, and if you wait until the day before assignments/tests you won't have enough time to get these into your mental muscle memory.
\(\ell^p\) norms: For \(p \geq 1\),
$$\|\mathbf{x}\|_p = \left(\sum_{i=1}^n |x_i|^p\right)^{1/p}$$Important special cases:
Every inner product induces a norm via:
$$\|\mathbf{x}\| = \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle}$$This norm satisfies all three norm axioms. The \(\ell^2\) norm is precisely the norm induced by the standard inner product:
$$\|\mathbf{x}\|_2 = \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle} = \sqrt{\sum_{i=1}^n x_i^2}$$For a weighted inner product \(\langle \mathbf{x}, \mathbf{y} \rangle_M = \mathbf{x}^\top M \mathbf{y}\) with \(M\) symmetric positive definite, the induced norm is:
$$\|\mathbf{x}\|_M = \sqrt{\mathbf{x}^\top M \mathbf{x}}$$This is often called an energy norm or \(M\)-norm.
For example, the \(\ell^1\) and \(\ell^\infty\) norms are not induced by any inner product.
In finite dimensions, all norms are equivalent. Specifically, for any two norms \(\|\cdot\|_a\) and \(\|\cdot\|_b\) on \(\mathbb{R}^n\), there exist constants \(c, C > 0\) such that
$$c \|\mathbf{x}\|_a \leq \|\mathbf{x}\|_b \leq C \|\mathbf{x}\|_a \quad \text{for all } \mathbf{x} \in \mathbb{R}^n$$Concrete examples:
$$\|\mathbf{x}\|_\infty \leq \|\mathbf{x}\|_2 \leq \sqrt{n} \|\mathbf{x}\|_\infty$$ $$\|\mathbf{x}\|_2 \leq \|\mathbf{x}\|_1 \leq \sqrt{n} \|\mathbf{x}\|_2$$The standard inner product (or Euclidean inner product) on \(\mathbb{R}^n\) is:
$$\langle \mathbf{x}, \mathbf{y} \rangle = \sum_{i=1}^n x_i y_i = \mathbf{x}^\top \mathbf{y}$$This can also be written in Einstein notation as \(\langle \mathbf{x}, \mathbf{y} \rangle = x_i y_i\).
More generally, given a symmetric positive definite matrix \(M \in \mathbb{R}^{n \times n}\), we can define a weighted inner product:
$$\langle \mathbf{x}, \mathbf{y} \rangle_M = \mathbf{x}^\top M \mathbf{y} = \sum_{i,j=1}^n x_i M_{ij} y_j$$The standard inner product corresponds to \(M = I\) (the identity matrix).
Key properties of inner products:
Cauchy-Schwarz inequality:
$$|\langle \mathbf{x}, \mathbf{y} \rangle| \leq \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle} \sqrt{\langle \mathbf{y}, \mathbf{y} \rangle}$$Polarization identity:
$$\langle \mathbf{x}, \mathbf{y} \rangle = \frac{1}{4}\left(\|\mathbf{x} + \mathbf{y}\|^2 - \|\mathbf{x} - \mathbf{y}\|^2\right)$$where \(\|\mathbf{x}\| = \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle}\) is the induced norm.
Parallelogram law:
$$\|\mathbf{x} + \mathbf{y}\|^2 + \|\mathbf{x} - \mathbf{y}\|^2 = 2\|\mathbf{x}\|^2 + 2\|\mathbf{y}\|^2$$For \(A \in \mathbb{R}^{m \times n}\), an induced (operator) norm is defined as
$$\|A\| = \sup_{\mathbf{x} \neq \mathbf{0}} \frac{\|A\mathbf{x}\|}{\|\mathbf{x}\|} = \sup_{\|\mathbf{x}\| = 1} \|A\mathbf{x}\|$$For any induced matrix norm:
Additional useful properties:
The Einstein summation convention automatically sums over repeated indices that appear once as a subscript and once as a superscript (or twice as subscripts in Cartesian coordinates). This eliminates the need for explicit summation symbols.
Basic rules:
Inner product of vectors \(\mathbf{u}, \mathbf{v} \in \mathbb{R}^n\):
$$\mathbf{u} \cdot \mathbf{v} = u_i v_i = \sum_{i=1}^n u_i v_i$$Matrix-vector multiplication \((A\mathbf{x})_i\):
$$y_i = A_{ij} x_j = \sum_{j=1}^n A_{ij} x_j$$Matrix-matrix multiplication \((AB)_{ik}\):
$$C_{ik} = A_{ij} B_{jk} = \sum_{j=1}^n A_{ij} B_{jk}$$Trace of a matrix:
$$\text{tr}(A) = A_{ii} = \sum_{i=1}^n A_{ii}$$Frobenius norm squared:
$$\|A\|_F^2 = A_{ij} A_{ij} = \sum_{i,j} A_{ij}^2$$In matrix form, \([\delta_{ij}] = I\) (the identity matrix).
Properties:
Even permutations: \((1,2,3), (2,3,1), (3,1,2)\) have \(\epsilon = +1\)
Odd permutations: \((1,3,2), (3,2,1), (2,1,3)\) have \(\epsilon = -1\)
Applications:
Cross product in \(\mathbb{R}^3\):
$$(\mathbf{u} \times \mathbf{v})_i = \epsilon_{ijk} u_j v_k$$Explicitly:
$$\mathbf{u} \times \mathbf{v} = \begin{pmatrix} u_2 v_3 - u_3 v_2 \\ u_3 v_1 - u_1 v_3 \\ u_1 v_2 - u_2 v_1 \end{pmatrix}$$Determinant of a \(3 \times 3\) matrix:
$$\det(A) = \epsilon_{ijk} A_{1i} A_{2j} A_{3k}$$Contraction identity (very useful):
$$\epsilon_{ijk} \epsilon_{lmk} = \delta_{il} \delta_{jm} - \delta_{im} \delta_{jl}$$Vector identity:
$$(\mathbf{u} \times \mathbf{v}) \cdot (\mathbf{w} \times \mathbf{z}) = (\mathbf{u} \cdot \mathbf{w})(\mathbf{v} \cdot \mathbf{z}) - (\mathbf{u} \cdot \mathbf{z})(\mathbf{v} \cdot \mathbf{w})$$which can be proven using: \(\epsilon_{ijk} \epsilon_{imn} = \delta_{jm}\delta_{kn} - \delta_{jn}\delta_{km}\)
For \(\mathbf{x}, \mathbf{y} \in \mathbb{R}^n\):
$$|\mathbf{x}^\top \mathbf{y}| \leq \|\mathbf{x}\|_2 \|\mathbf{y}\|_2$$Equality holds if and only if \(\mathbf{x}\) and \(\mathbf{y}\) are linearly dependent.
Generalization: For any inner product space,
$$|\langle \mathbf{x}, \mathbf{y} \rangle| \leq \|\mathbf{x}\| \|\mathbf{y}\|$$For any norm:
$$\|\mathbf{x} + \mathbf{y}\| \leq \|\mathbf{x}\| + \|\mathbf{y}\|$$Reverse triangle inequality:
$$\big| \|\mathbf{x}\| - \|\mathbf{y}\| \big| \leq \|\mathbf{x} - \mathbf{y}\|$$For \(p, q \geq 1\) with \(\frac{1}{p} + \frac{1}{q} = 1\) (conjugate exponents):
$$\sum_{i=1}^n |x_i y_i| \leq \|\mathbf{x}\|_p \|\mathbf{y}\|_q$$Special case (\(p = q = 2\)): Cauchy-Schwarz inequality.
For \(p \geq 1\):
$$\|\mathbf{x} + \mathbf{y}\|_p \leq \|\mathbf{x}\|_p + \|\mathbf{y}\|_p$$This is the triangle inequality for \(\ell^p\) norms.
For \(a, b \geq 0\) and conjugate exponents \(p, q > 1\) with \(\frac{1}{p} + \frac{1}{q} = 1\):
$$ab \leq \frac{a^p}{p} + \frac{b^q}{q}$$Special case (\(p = q = 2\)):
$$ab \leq \frac{a^2}{2} + \frac{b^2}{2}$$More generally, for any \(\epsilon > 0\):
$$ab \leq \frac{\epsilon a^2}{2} + \frac{b^2}{2\epsilon}$$This is particularly useful for absorbing terms in energy estimates.
For a function \(f: \mathbb{R} \to \mathbb{R}\) that is \((n+1)\)-times continuously differentiable on an interval containing \(a\) and \(x\), Taylor's theorem states:
$$f(x) = \sum_{k=0}^{n} \frac{f^{(k)}(a)}{k!}(x-a)^k + R_n(x)$$where \(R_n(x)\) is the remainder term (or truncation error).
Lagrange form (most common for error analysis):
$$R_n(x) = \frac{f^{(n+1)}(\xi)}{(n+1)!}(x-a)^{n+1}$$for some \(\xi\) between \(a\) and \(x\). This follows from the mean value theorem applied repeatedly.
Integral form:
$$R_n(x) = \frac{1}{n!}\int_a^x (x-t)^n f^{(n+1)}(t) \, dt$$Big-O notation:
$$R_n(x) = O(|x-a|^{n+1}) \quad \text{as } x \to a$$Expanding around \(a = 0\) (Maclaurin series):
Exponential:
$$e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots + \frac{x^n}{n!} + \frac{e^\xi}{(n+1)!}x^{n+1}$$Sine:
$$\sin(x) = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \cdots + R_n(x)$$Cosine:
$$\cos(x) = 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \cdots + R_n(x)$$Natural logarithm:
$$\ln(1+x) = x - \frac{x^2}{2} + \frac{x^3}{3} - \cdots + R_n(x), \quad |x| < 1$$For a smooth function \(u(x)\) with \(|u''(x)| \leq M\), expand around \(x\):
$$u(x+h) = u(x) + h u'(x) + \frac{h^2}{2} u''(\xi)$$for some \(\xi \in (x, x+h)\). Rearranging:
$$u'(x) = \frac{u(x+h) - u(x)}{h} - \frac{h}{2} u''(\xi)$$The truncation error is:
$$\left|u'(x) - \frac{u(x+h) - u(x)}{h}\right| = \frac{h}{2}|u''(\xi)| \leq \frac{Mh}{2}$$Thus the forward difference is first-order accurate: error \(= O(h)\).
For the second derivative, expand both forward and backward:
$$u(x+h) = u(x) + h u'(x) + \frac{h^2}{2} u''(x) + \frac{h^3}{6} u'''(x) + \frac{h^4}{24} u^{(4)}(\xi_+)$$ $$u(x-h) = u(x) - h u'(x) + \frac{h^2}{2} u''(x) - \frac{h^3}{6} u'''(x) + \frac{h^4}{24} u^{(4)}(\xi_-)$$Adding these equations:
$$u(x+h) + u(x-h) = 2u(x) + h^2 u''(x) + \frac{h^4}{24}\left[u^{(4)}(\xi_+) + u^{(4)}(\xi_-)\right]$$Solving for \(u''(x)\):
$$u''(x) = \frac{u(x-h) - 2u(x) + u(x+h)}{h^2} - \frac{h^2}{24}\left[u^{(4)}(\xi_+) + u^{(4)}(\xi_-)\right]$$The truncation error is:
$$\left|u''(x) - \frac{u(x-h) - 2u(x) + u(x+h)}{h^2}\right| \leq \frac{h^2}{12}\max|u^{(4)}|$$Thus the centered difference for the second derivative is second-order accurate: error \(= O(h^2)\).
Problem: Approximate \(f(x) = \sin(x)\) using a linear function on the interval \([x_0, x_0 + h]\).
Solution: Use the first-order Taylor expansion:
$$\sin(x_0 + h) = \sin(x_0) + h\cos(x_0) + \frac{h^2}{2}(-\sin(\xi))$$for some \(\xi \in (x_0, x_0 + h)\).
The linear approximation is:
$$p(x) = \sin(x_0) + (x - x_0)\cos(x_0)$$The maximum error on the interval is:
$$\max_{x \in [x_0, x_0+h]} |f(x) - p(x)| \leq \frac{h^2}{2}\max_{\xi \in [x_0, x_0+h]} |\sin(\xi)| \leq \frac{h^2}{2}$$Numerical example: For \(x_0 = 0\) and \(h = 0.1\):
The actual error is much smaller than the bound because \(\sin(\xi) \ll 1\) for small \(\xi\).
Key insight: The truncation error scales as \(h^{n+1}\) where \(n\) is the degree of the approximating polynomial. This is fundamental to understanding convergence rates in numerical methods.
For scalar \(a, b\) and \(\epsilon > 0\):
$$2ab = \epsilon a^2 + \frac{b^2}{\epsilon} - \left(\sqrt{\epsilon} a - \frac{b}{\sqrt{\epsilon}}\right)^2 \leq \epsilon a^2 + \frac{b^2}{\epsilon}$$For vectors with respect to a positive definite matrix \(M\):
$$2\mathbf{x}^\top M \mathbf{y} \leq \epsilon \mathbf{x}^\top M \mathbf{x} + \frac{1}{\epsilon} \mathbf{y}^\top M \mathbf{y}$$If \(u(t) \geq 0\) satisfies
$$u(t) \leq C + \int_0^t \alpha(s) u(s) \, ds$$for constants \(C \geq 0\) and \(\alpha(s) \geq 0\), then
$$u(t) \leq C \exp\left(\int_0^t \alpha(s) \, ds\right)$$Discrete version: If \(u_n \geq 0\) satisfies \(u_n \leq C + \sum_{k=0}^{n-1} \alpha_k u_k\) with \(\alpha_k \geq 0\), then
$$u_n \leq C \prod_{k=0}^{n-1} (1 + \alpha_k)$$For a symmetric matrix \(A \in \mathbb{R}^{n \times n}\):
Rayleigh quotient:
$$\lambda_{\min}(A) \leq \frac{\mathbf{x}^\top A \mathbf{x}}{\mathbf{x}^\top \mathbf{x}} \leq \lambda_{\max}(A)$$Weyl's inequality: For symmetric \(A, B\):
$$\lambda_{\min}(A) + \lambda_{\min}(B) \leq \lambda_{\min}(A + B) \leq \lambda_{\min}(A) + \lambda_{\max}(B)$$Energy inequality: If \(A\) is positive definite with \(\lambda_{\min}(A) = \lambda > 0\):
$$\lambda \|\mathbf{x}\|_2^2 \leq \mathbf{x}^\top A \mathbf{x} \leq \lambda_{\max}(A) \|\mathbf{x}\|_2^2$$These tools form the foundation for deriving stability estimates, convergence rates, and error bounds throughout the course.