1.0 - Formal Definition of Asymptotic Bounds

1.1 - Upper Bounds

🌱 $O(g)$ is the set of all functions that (1) have (exist) constants $c, n_0 > 0$ that is greater than or equal to $f(n)$ for all $n \ge n_0$.

$$O(g)=\{f | \exists c, n_0\gt 0 \bullet \forall n\ge n_0 \bullet 0 \le f(n) \le c\cdot g(n) \}$$

• That is, the set of functions (where $c, n_0>0$) exist and for all values of $n\ge n_0$, the function $f(n)$ is bounded above by $c\cdot g(n)$.

Using the formal definition above, prove that $2n^3\in O(n^3)$

• To do this, we need to find constants $c,n_0 > 0$ such that $(\forall n \ge n_0 \bullet 0 \le2n^3\le cn^3)$

• Choosing $c=2$ and $n_0=1$ will suffice since

  $$\begin{aligned}&\forall n\ge1 \bullet 0 \le2n^3\le 2n^3\\ \equiv &\text{ true} \end{aligned}$$

Using the formal definition above, prove that $2n^2 \in O(n^3-n^2)$

• To do this, we need to find constants $c,n_0 > 0$ such that $(\forall n \ge n_0 \bullet 0 \le2n^3\le cn^3)$

• Trivially, we know that:

  $$\begin{aligned} &0\le 2n^2\\ \equiv & \text{ true}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{if } n \ge 0 \end{aligned}$$

• Additionally, we have:

  $$\begin{aligned} &2n^2\le c(n^3-n^2)\\ \equiv\ &2\le c(n-1)\ \ \ \ \ \ \ \ \text{if } n > 0, by (\div n^2)\\ \equiv\ &2+c\le cn\\ \equiv\ &2+2\le 2n\ \ \ \ \ \ \ \ \ \ \ \text{if } c=2\\ \equiv\ &2 \le n \end{aligned}$$

• Therefore, $c=2$ and $n_0=2$ will therefore suffice.

1.2 - Lower Bounds

🌱 $\Omega(g)$ is the set of all functions that (1) have (exist) constants $c, n_0 > 0$ that is less than or equal to $c\cdot g(n)$ for all $n \ge n_0$.

$$\Omega(g)=\{f|\exists c, n_0> 0\bullet \forall n\ge n_0 \bullet, 0 \le c\cdot g(n)\le f(n) \}$$

Using the above definition, prove that $2n^3\in \Omega(n^3)$

• We need to find constants $c, n_0>0$ such that $(\forall n \ge n_0\bullet 0\le cn^3\le2n^3)$

• Choosing $c=2$ and $n_0$ will suffice, since:

  $$\begin{aligned} &\forall n \ge1\bullet 0 \le 2n^3\le 2n^3\\ \equiv\ &\text{true} \end{aligned}$$

1.3 - Tight Bounds

🌱 $\Theta(g)$ is the set of all functions that (1) have (exist) constants $c_1, c_2, n_0>0$ that bounds the function $f(n)$ on the top and bottom to within a constant factor.

$$\Theta(g)=\{f | \exists c_1, c_2, n_0 > 0\bullet \forall n\ge n_0 \bullet 0\le c_1\cdot g(n) \le f(n)\le c_2\cdot g(n)\}$$

Using the above definition, prove that $2n^3\in\Theta (n^3)$

• We need to find constants $c_1, c_2, n_0>0$ such that $(\forall n\ge n_0, \bullet 0 \le c_1n^3\le2n^3\le c_2n^3)$

• Choosing $c_1=c_2=2$ and $n_0=1$ will suffice since:

  $$\begin{aligned} &\forall n\ge 1 \bullet 0 \le 2n^3\le 2n^3 \le 2n^3\\ \equiv\ &\text{true} \end{aligned}$$

Using the above definition, prove that $\frac{n^2}2\in\Theta(n^2)$

• We need to find constants $c_1,c_2,n_0>0$ such that

  $$(\forall n\ge n_0\bullet 0\le c_1 n^2 \le\frac{n^2}2)-2\le c_2n^2$$

• Trivially, we know that:

  $$\begin{aligned} &0\le c_1 n^2\\ \equiv\ &\text{true if } n \ge 0 \end{aligned}$$

• Additionally:

  $$\begin{aligned} &c_1 n^2\le \frac{n^2}2 -2n\\ \equiv\ &2\le n(\frac12 - c_1) \ \ \ \ \text{if } n>0\\ \equiv\ & 8\le n \ \ \ \ \text{if } c_1=4\\ \end{aligned}$$

• Finally:

  $$\begin{aligned} &\frac{n^2}{2}-2n\le c_2n^2\\ \equiv\ &\text{true\ \ \ \ \ \ \ \ \ \ \ if }n\ge0 \text{ and } c_2\le\frac12 \end{aligned}$$

• Therefore, choosing $c_1=\frac14, c_2=\frac12$ and $n_0=8$ will therefore suffice

Using the above definition, prove that $\frac{n^2}2 -2n\in \Theta(n^2)$

• We need to find constants $c_1, c_2, n_0>0$ such that:

  $$(\forall n\ge n_0, 0 \le c_1 n^2 \le \frac{n^2}2-2n\le c_2 n^2)$$

• Firstly, we know that $0 \le c_1 n^2$ is true for all $n \ge 0$

• Additionally, we prove that:

  $$\begin{aligned} &c_1 n^2\le \frac{n^2}{2}-2n\\ \equiv\ &2\le n(\frac 12-c_1) \ \ \ \ \ \text{if } n > 0\\ \equiv\ & 8 \le n \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{if } c_1 =\frac14\\ \end{aligned}$$

• Finally, we prove that:

  $$\begin{aligned} &\frac{n^2}2-2n\le c_2 n^2\\ \equiv\ &\text{true}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{if } n \ge 0 \text{ and } c_2\ge \frac12 \end{aligned}$$

• Therefore, choosing $c_1=\frac14, c_2=\frac12$ and $n_0=8$ will therefore suffice

1.4 - Properties of Asymptotic Notation

🌱 Theorem:

$$f\in O(g)\Longrightarrow g+f\in \Theta(g)$$

For example, $n\in O(n^2)\Longrightarrow n^2+n\in\Theta(n^2)$

🌱 Theorem: For $k>0$

$$k\cdot n^a\in \Theta(n^a)$$

🌱 Theorem: For $k>0$ and $0\le a\le b$

$$k\cdot n^a\in O(n^b)$$

🌱 Theorem: For any functions $f$ and $g$

$$f\in O(g)\Longleftrightarrow g\in\Omega(f)$$

🌱 Theorem: For any functions $f$ and $g$

$$f\in\Theta(g)\Longleftrightarrow f\in O(g)\wedge f\in\Omega (g)$$

2.0 - Comparing Functions

🌱 A function $f$ is asymptotically non-negative if:

$$\exists n_0\bullet (\forall n \ge n_0 \bullet f(n)\ge 0)$$

🌱 If $f$ and $g$ are asymptotically non-negative functions, and

$$\begin{aligned} \lim_{n\rightarrow\infty} \frac{f(n)}{g(n)}&=c\ge0 \text{ then } f\in O(g)\\ &=c>0 \text{ then } f\in\Theta(g)\\ &=\infty\text{ then } g\in O(f) \end{aligned}$$

where $c$ is a real-valued constant

• However, we cannot compare all functions using the asymptotic notation technique.

• For example, the following functions cannot be compared using asymptotic notation

  $$n \text{ and } n^{1+\sin n}$$

Plot of $f(n)=n$ (red) and $g(n)=n^ {1+\sin n}$

3.0 - Polynomials

$$p(n)=\sum^d_{k=0} a_k n^k=a_0n^0+a_1n^1+\cdots+a_dn^d$$

🌱 Theorem: If $a_d>0$,

$$p(n)\in\Theta(n^d)$$

$$\small\begin{aligned} &\lim_{n\rightarrow\infty} \frac{p(n)}{n^d}\\ =& \lim_{n\rightarrow\infty} \sum^d_{k=0} \frac{a_k n^k}{n_d}\\ =& \lim_{n\rightarrow\infty} \frac{a_dn^d}{n^d} + \sum^{d-1}_{k=0} \frac{a_k n^k}{n^d}\\ =& \lim_{n\rightarrow\infty} a_d + \sum^{d-1}_{k=0}\frac{a_k}{n^{d-k}}\\ =& a_d \end{aligned}$$

• Since the limit gives $a_d$, which the theorem assumes is greater than zero, we know that $p(n)\in \Theta(n^d)$
• Additionally,

🌱 For $a>1$ and $n\ge0,$

$${\color{lightblue}n^d}\in O({\color{lightgreen}a^n})$$

$$\begin{aligned} &\lim_{n\rightarrow\infty} \frac{\color{lightblue}n^d}{\color{lightgreen}a^d}\\ &\lim_{n\rightarrow\infty} \frac{dn^{d-1}}{a^n \ln a}\\ &\lim_{n\rightarrow\infty} \frac{d(d-1)n^{d-2}}{a^n(\ln a)^2} \ \ \ \ \ \ \ \ \ \ \ \text{By L'Hopital's}\\ &\lim_{n\rightarrow\infty} \frac{d(d-1)(d-2)\cdots 1 n^0}{a^n(\ln a)^d}\\ &\lim_{n\rightarrow\infty} \frac{d!}{a^n (\ln a)^d} \end{aligned}$$

As $d!$ and $(\ln a)^d$ are constants, and $a^n\rightarrow\infty$, the limit is 0. Therefore, this has a tight-o bound.

4.0 - Mathematical Background

4.1 - Finite Summations

🌱 Given a sequence $\langle a_1, a_2, \cdots, a_n\rangle$, the finite summation is given as:

$$\sum^n_{k=1} a_k=a_1+a_2+\cdots+a_n\tag{1}$$

• Additionally, we have the property that the value of a summation over an empty range is 0

  $$\sum^0_{k=1}a_k=0\text{\ \ \ \ \ (by definition)} \tag{2}$$

• Finite sums may be added in any order (commutativity)

• For $n$ nonintegral, could use floor $\lfloor n \rfloor$ or ceiling $\lceil n\rceil$.

  • Nonintegral Not of, being, or related to a mathematical integer

4.2 - Infinite Summations

🌱 The infinite summation $a_1+a_2+\cdots$ is represented as the limit of a finite summation:

$$\sum^\infty_{k=1}=\lim_{n\rightarrow\infty} \sum^n_{k=1} a_k\tag{3}$$

• The limit does not need to be well-=behaved
  • Diverges (limit does not exist) $\sum^\infty_{k=1} (-1)^n$
  • Converges
  • Absolutely convergent (any order) - converges if you take the absolute value of each value in the sum.
    • That is $\sum^\infty_{k=1} |a_k |$ converges.

4.3 - Linearity

$$\sum^n_{k=1} (ca_k+b_k)=c\sum^n_{k=1}a_k+\sum^n_{k=1} b_k \tag{4}$$

• This also holds true for infinite sums, $\sum^\infty_{k=1}$

4.4 - Arithmetic Series

$$\sum^n_{k=1}k=\frac{n(n+1)}{2}\tag{5}$$

4.5 - Geometric Series

4.5.1 - Finite Geometric Series

$$\begin{aligned} \sum^n_{k=0}x^k&=1+x+x^2+x^3+\cdots+x^n \tag{6}\\ &=\frac{x^{n+1}-1}{x-1} \ \ \ \ \ \ \text{for } x \ne 1 \end{aligned}$$

4.5.2 - Infinite Geometric Series

• Derived from the above formula, but as $n\rightarrow 0$, so all that’s left is the $1$ in the numerator

$$\sum^\infty_{k=0}x^k=\frac{1}{1-x}\ \ \ \ \ \ \ \text{for } |x|<1\tag{7}$$

4.6 - Harmonic Series

$$\begin{aligned} H_n&=1 + \frac{1}{2}+\frac13+\cdots+\frac1n &=\sum^n_{k=1} \frac1k \tag{8} \end{aligned}$$

$H_n=\ln n + \gamma\Rightarrow \gamma=0.577...$

4.7 - Tight Upper Bound

🌱 Little-o notation represents the tight asymptotic bound on a function.

$$o(g(n))=\{f(n):\forall c >0, \exists n_0>0 | 0\le f(n)<c\cdot g(n), \forall n \ge 0\}$$

• Note that this function requires both functions $f$ and $g$ to be asymptotically non-negative.

• We can use the little-o definition with limits:

  $$f(n)\in o(g(n)) \ \ \text{if}\ \ \lim_{n\rightarrow\infty}\frac{f(n)}{g(n)}=0\tag{9}$$

  and $f(n)$ is asymptotically non-negative.

4.8 - L’Hopital’s Rule

🌱 How do you evaluate the limit $\lim_{n\rightarrow\infty} \frac{f(n)}{g(n)}$ if it looks hard?

Consider $\lim_{n\rightarrow a}\frac{f(n)}{g(n)}$ when either:

$$\lim_{n\rightarrow a} f(n)=0=\lim_{n\rightarrow a}g(n)$$

or when

$$\lim_{n\rightarrow a} f(n)=\pm\infty=\lim_{n\rightarrow a} g(n)$$

We can use L’Hopital’s rule, giving:

$$\lim_{n\rightarrow a} \frac{f(n)}{g(n)}=\lim_{n\rightarrow a} \frac{\frac{d}{dx} f(n)}{\frac{d}{dx} g(n)}\tag{10}$$

4.9 - Telescoping Series

$$\begin{aligned} \sum^n_{k=1}(a_k-a_{k-1})&=(a_1-a_0) + (a_2-a_1)+\cdots+(a_n-a_{n-1})\\ &=a_n-a_0 \tag{11} \end{aligned}$$

$$\sum^{n-1}_{k=0}(a_k-a_{k+1})=a_0-a_n \tag{12}$$

Example:

$$\sum^{n-1}_{k=1}\frac{1}{k(k+1)}=\sum^{n-1}_{k=1}(\frac1k-\frac1{k+1})=1-\frac1n$$

We know this because $\frac1{k(k+1)}=\frac 1k-\frac 1{k+1}$

4.10 - Differentiation

Example: $\sum^\infty_{k=0} kx^k$ where $|x|<1$, we know:

$$\sum^\infty_{k=0}=\frac{1}{1-x}\ \ \ \ \ \text{if } |x|<1$$

Differentiation both sides with respect to $x$ gives:

$$\sum^\infty_{k=0} x^k = \frac{1}{(1-x)^2}$$

This gives:

$$\sum^\infty_{k=0} kx^k=x(\sum^\infty_{k=0} kx^{k-1})=\frac{x}{(1-x)^2}$$

4.11 - Algebraic Manipulation

Example $s=\sum^\infty_{i=1} \frac{i}{2^i}$

$$\begin{aligned} s&=\frac12+\frac{2}{2^2}+\frac{3}{2^3}+\cdots\\ 2s&=1+\frac22+\frac{3}{2^2}+\cdots\\ 2s-s&=1+\frac12+\frac{1}{2^2}+\cdots+\\ &=\sum^\infty_{k=0}\frac1{2^k}=\frac{1}{1-\frac{1}{2}}=2 \end{aligned}$$

so $s=2$

4.12 - Products

Definitions:

$$\prod^n_{k=1} a_k=a_1 a_2\cdots a_n$$

Note that by definition, products over empty ranges have a value of 1.

$$\prod^0_{k=1} a_k=1$$

Products can be converted to sums using logarithms

$$\lg \prod^n_{k=1}a_k=\sum^n_{k=1} \lg a_k$$

4.13 - Logarithms

Remember that $\log x=\log_2x$ and $\ln x=\log_e x$

4.14 - Bounding Sums - Upper Bounds

$$\sum^n_{k=1} a_k\le na_{\text{max}}$$

For example:

$$\sum^n_{k=1}k\le\sum^n_{k=1}n=n^2$$

If we know that $\frac{a_{k+1}}{a_k}\le r, \forall k \ge 0$ with $r<1$ as a constant (i.e. geometric series) then:

$$\sum^n_{k=0}\le \sum^\infty_{k=0} a_0 r^k = a_0\sum^n_{k=0} r^k\le \frac {a^0}{1-r}$$

4.14 - Bounding Sums - Lower Bounds

$$\sum^n_{k=1} a_k\le na_{\text{min}}$$

For example:

$$\sum^n_{k=1} k \ge \sum^n_{k=1}1=n$$

This is a linear bound, which is poor - Split the sum

$$\sum^n_{k=1}k=\sum^{n/2}_{k=1}k+\sum^n_{k=\frac n2+1}k\ge \sum^\frac n2_{k=1} 0+\sum^n_{k=\frac n2 + 1} \frac n2 \ge (\frac n2)^2=\frac{n^2}4$$

This is a quadratic bound - $\Theta(n^2)$

4.15 - Bounding Products - Upper and Lower Bounds

Give an upper and lower bound for the product $n!$

$$n!=1\times 2\times \cdots \times 3\times (n-1)\times n=\prod^n_{k=1}k$$

An upper bound for this product is:

$$n!=\prod^n_{k=1}k\le\prod^n_{k=1}n=n^n$$

A (trivial) lower bound for this product is:

$$n!=\prod^n_{k=1}k\ge \prod^n_{k=1}1=1$$

A tight lower bound for this product is

$$\begin{aligned} n!&=\prod^\frac n2_{k=1}k\times\prod^n_{k=\frac n2 + 1} k\\ &\ge \prod^\frac n2 _{k=1} 1 \times \prod ^n _{k=\frac n2 + 1} \frac n2 = (\frac n2)^ \frac n2 \end{aligned}$$

Note that Stirling’s approximation gives:

$$n!=\sqrt{2\pi n} (\frac ne)^n (1+\Theta(\frac 1n))$$

4.16 - Splitting Sums is a Powerful Technique

Consider using the splitting sums technique to compute the upper bound of the harmonic series:

Consider $H_n=\sum^n_{k=1} \frac 1k$

Split the range into $\lfloor \lg n\rfloor$ pieces, with each piece summing $\le 1$

$$(1)+(1/2+1/3)+(1/4+1/5+1/6+1/7)+\cdots$$

$$\sum^n_{k=1}\frac 1k\le\sum^{\lfloor \lg n\rfloor}_{i=0} (\sum^{2^i-1}_{j=0} \frac 1{2^i + j})\le\sum^{\lfloor \lg n\rfloor}_{i=0}(\sum^{2^i-1}_{j=0} \frac 1 {2^i})\le \sum^{\lfloor \lg n \rfloor}_{i=0} 1 \le 1 + \lg n$$

4.17 - Approximation by Integrals

Consider $\sum^n_{k=m} f(k)$ where $f(k)$ is monotonically increasing:

$$\int^n_{m-1} f(x)\ dx\le\sum^n_{k=m} f(k) \le \int^{n+1}_m f(x)\ dx$$

There is a similar proof for monotonically decreasing functions:

$$\int^{n+1}_m f(x)\ dx \le \sum^n_{k=m} f(k) \le \int^n_{m-1} f(x)\ dx$$

Thus $H_n\ge \int^{n+1}_1 \frac 1x dx = \ln (n + 1)$ and $\sum^n_{k=2} \frac 1k \le \int ^n_1 \frac 1x dx=\ln n$ so $H_n\le \ln n +1$