1.0 - Matrix Multiplication

🌱 We want to find an order in which to multiply the chain of matrices that has the least cost

$$M_1\cdot M_2\cdot M_3 \cdots M_n$$

which has the least cost (i.e. will be the fastest). Assume that the matrices may have different dimensions, but that the multiplication is well defined (e.g. $\#\text{cols of } M_1=\#\text{rows of }M_{i+1}$

• First consider the trivial case in which we want to multiply the three matrices $M_1\cdot M_2\cdot M_3$ - this operation can be completed in two different ways:

  $$M_1\cdot(M_2\cdot M_3)\ \ \ \ \text{or}\ \ \ \ (M_1\cdot M_2)\cdot M_3$$

• Which one has the least cost?

1.1 - Mechanics of Matrix Multiplication

• Given that we have two $2\times 2$ matrices $M_1$ and $M_2$ that we want to multiply:

  $$M_1=\begin{matrix}w & x\\y&z\end{matrix}\ \ \ \ \ \ \ \ \ \ \ M_2=\begin{matrix}\alpha&\beta\\\gamma&\delta\end{matrix}$$

• Their product is given as:

  $$M_1\cdot M_2=\begin{matrix}w\cdot\alpha + x\cdot\gamma &w\cdot\beta + x\cdot\delta\\ y\cdot\alpha + z \cdot \gamma & y \cdot\beta + z \cdot \delta\end{matrix}$$

• The matrices $M_1$ and $M_2$ can only be multiplied if their shapes are compatible - that is, # of columns of $M_1$ = # of rows of $M_2$

• Suppose $M_1$ is of size $\textbf{p}\times\textbf{q}$ and $M_2$ is of size $\textbf{q} \times \textbf{r}$ then $M_1\cdot M_2$ is of size $\textbf{p}\times \textbf{r}$

• Consider $M_1$ is a $\textbf 1\times \textbf 3$ matrix, and $M_2$ is a $\textbf 3 \times \textbf 2$ matrix:

• The resulting matrix is of size $\textbf1 \times \textbf3$

  $$M_1=\begin{matrix} a & b & c\end{matrix} \ \ \ \ \ \ \ \ \ \ M_2=\begin{matrix}d&g\\e&h\\f&i\end{matrix}\\ M_1\cdot M_2=\begin{matrix}a\cdot d+b\cdot e+c\cdot f & a\cdot g + b\cdot h+c\cdot i \end{matrix}$$

1.2 - Multiplying Matrices

• The following is a straightforward algorithm to multiply two matrices:

  • Matrix A of dimension $\textbf p\times \textbf q$
  • Matrix B of dimension $\textbf q\times \textbf r$

  mat_mult(A, B, p, q, r):
      let C be a new p x r matrix
      for i = 1 to p
          for j = 1 to r
              C[i][j] = 0
              for k = 1 to q
                  C[i][j] = C[i][j] + A[i][k] + B[k][j]
  

• The total number of multiplications to perform in the inner-loop is $p\times q\times r$

• The total time complexity is $\Theta(p\times q \times r)$

• In general, matrix multiplication is not commutative, that is,

  $$M_1\cdot M_2 \ne M_2\cdot M_1$$

• However, it is associative

  $$M_1\cdot( M_2\cdot M_3)=(M_1\cdot M_2)\cdot M_3$$

• Note that calculating the result of $M_1\cdot M_2\cdot M_3$ can result in huge differences in time factors, depending on which of the parenthesising choices are made

• Consider $M_1$ is $10\times 100$ and $M_2$ is $100\times 5$

• The cost of $M_1\cdot M_2$ is $10\times100\times5=5000$ iterations

1.3 - Example - Importance of Order

• Now consider $M_1$ is $10\times 100$ and $M_2$ is $100\times 5$ and $M_3$ is $5\times50$
• Case 1 $(M_1\cdot M_2)\cdot M_3$
  • From before, we know that computing $M_1\cdot M_2$ will take 5000 iterations
  • Given $M_1\cdot M_2$, computing $(M_1\cdot M_2)\cdot M_3$ requires an additional $10\times 5 \times 50$ iterations
  • In total, this is 7, 500 iterations
• Case 2 ($M_1\cdot(M_2\cdot M_3)$)
  • The cost of computing $M_2\cdot M_3$ is $10\times5\times50=25,000$
  • The cost of computing $M_1\cdot(M_2\cdot M_3)$ given pre-computed $M_2\cdot M_3$ is $10\times100\times50=50,000$
  • In total, this is 75, 000 iterations - 10 times more than the other solution

1.4 - Task

🌱 We want to find an order in which to multiply the chain of matrices that has the least cost

$$M_1\cdot M_2\cdot M_3 \cdots M_n$$

which has the least cost (i.e. will be the fastest), assuming that each matrix $M_i$ has dimensions $p_{i-1}\times p_i$ so that by definition, each pair of adjacent matrices is compatible.

Additionally, the cost of $M_i\cdot M_{i+1}=p_{i-1}\cdot p_i\cdot p_{i+1}$, in which the resulting matrix $M_i\cdot M_{i+1}$ is of dimension $p_{i-1}\times p_{i+1}$

1.4.1 - Naive Solution

• Why can’t we just enumerate each possible way of multiplying $n$ matrices together, and see which one is cheapest?

• There are an exponential number of possible ways to multiply these matrices together $(\Omega(2^n))$ ways.

  $$\begin{aligned} N(1)&=1&(M_1)\\ N(2)&=1\\ &=N(1)\times N(1)&(M_1)\cdot (M_2)\\ N(3)&=2\\ &=N(1)\times N(2)\ + &(M_1)\cdot(M_2\cdot M_3)\\ &\ \ \ \ \ N(2)\times N(1) &(M_1\cdot M_2)\cdot (M_3)\\ N(4)&=5\\ &=N(1)\times N(3)\ + &M_1\cdot (M_2 \cdot (M_3\cdot M_4)), M_1\cdot ((M_2\cdot M_3)\cdot M_4)\\ &= N(2)\times N(2)\ + &(M_1\cdot M_2)\cdot (M_3\cdot M_4)\\ &=N(3)\times N(1) &(M_1\cdot (M_2\cdot M_3))\cdot M_4, ((M_1\cdot M_2)\cdot M_3)\cdot M_4\\ N(n)&=\sum^{n-1}_{i=1}N(i)\times N(n-1) \end{aligned}$$

We know that the recurrence $N(n)\in \Omega(2^n)$ where

$N(1)=1$

$\begin{aligned}N(n)&=\sum^{n-1}_{i=1} N(i)\times N(n-i)\\ &=N(1)\times N(n-1)+\cdots+N(n-1)\times N(1) &\text{ for } n > 2\\ &\le N(1)\times N(n-1) + N(n-1)\times N(1)\\ &=2N(n-1) \end{aligned}$

1.4.2 - Recursive Solution

• We first identify the sub-problems

  • Let $C_{i..j}$ give the minimum cost of multiplying $M_i\cdot M_{i+1}\cdots M_{j}$.
  • Thus, the solution to our problem will be given by $C_{1..n}$

• We then identify the base cases.

  • The solution to each sub-problem $C_{i..i} =0$

• We then identify the recusive case.

  • Consider the chain of length 4:

    $$C_{1..4}=\min\begin{cases}C_{1..1}+C_{2..4} + p_o\cdot p_1\cdot p_4 & M_1\cdot(\cdots)\\ C_{1..2}+C_{3..4}+p_0\cdot p_2\cdot p_4 &(M_1\cdot M_2)\cdot(M_3\cdot M_4)\\ C_{1..3}+C_{4..4}+p_0\cdot p_3\cdot p_4&(\cdots)\cdot M_4\end{cases}$$

  • In which we have:

    $$C_{2..4}=\min\begin{cases}C_{2..3}+C_{4..4}+p_0\cdot p_1\cdot p_2&(M_2\cdot M_3)\cdot M_4\\ C_{2..2}+C_{3..4}+p_1\cdot p_2\cdot p_3&M_2\cdot(M_3\cdot M_4)\end{cases}$$

• More generally, we have:

  $$C_{i..j}=\min_{i\le k \lt j}\{C_{i..k}+C_{k+1..j} + p_{i-1}\cdot p_i\cdot p_j\}$$

• Finally, we have:

  $$C_{i..n}=\min_{1\le k \lt n} \{C_{1..k}+C_{k+1..n} + p_0\cdot p_k\cdot p_n\}$$

1.4.3 - Dynamic Solution

• We notice that each sub-problem $C_{i..j}$ depends on every sub(sub) problem $C_{i..k}$ and $C_{j+1..j}$.

• Furthermore, each subproblem may be re-calculated many times in the recursive definition

• Converting this into a Dynamic programming solution allows us to change an exponential-time problem into a polynomial-time problem $\Theta(n^3)$

• We store the solution $C_{i..j}$ at $\textbf{m[i,j]}$ in an $\textbf n \times \textbf n$ matrix, $m$.

  • Thus, $m[i,j]$ is the minimum cost paranthesisation of $M_i\cdots M_j$
  • Additionally, we are only ever interested in entries $m[i,j]$ where $i\le j$, since the cost of $C_{5..2}$ is nonsensical.

• We should calculate sub-problems in an order such that when we need it, they are already computed.

• For $1\le i \le n-1$, cases $\text{m[i, i+1]}$ depend on $\text{m[i,i]}$ and $\text{m[i+1, i+1]}$ and they should be calculated next.

• Thus, we should:

  1. Calculate all $\text{m[i,i]}$ for all $i\le i\le n$
  2. Calculate all $\text{m[i, i+1]}$ for all $1\le i \le n-1$
  3. Calculate all $\text{m[i, i+2]}$ for all $1\le i \le n-2$

matrix_chain_order(p, n)
    m = new int[n, n]
    s = new int[n, n]
    for i = 1 to n
        m[i, i] = 0
        s[i, i] = i
    for L = 2 to n
        for i = 1 to n - L + 1
            j + i + L -1
            assert i <= k
            assert k < j
            m[i, j] = min m[i, k] + m[k + 1, j] + (p_{i-1} * p_k * p_j)
            s[i, j] = k

• For $\Theta(n^2)$ sub-problems, we must find the minimum set of possibilities, which makes it an $\Theta(n^3)$ operation
• We additionally create a matrix in which we store our solution space.