x̄ - > Advanced matrix concepts like diagonalization, SVD, or matrix exponentials!

Mathematics of Transformers

1. Matrix Diagonalization

Problem:

Diagonalize the matrix:

\[ A = \begin{bmatrix} 6 & 2 \\ 2 & 3 \end{bmatrix} \]

Solution:

Step 1: Find Eigenvalues

We solve \( \det(A - \lambda I) = 0 \):

\[ A - \lambda I = \begin{bmatrix} 6 - \lambda & 2 \\ 2 & 3 - \lambda \end{bmatrix} \] \[ \det(A - \lambda I) = (6 - \lambda)(3 - \lambda) - 4 = \lambda^2 - 9\lambda + 14 = (\lambda - 7)(\lambda - 2) \]

Eigenvalues: \( \lambda_1 = 7, \lambda_2 = 2 \).

Step 2: Find Eigenvectors

For \( \lambda_1 = 7 \):

\[ (A - 7I) = \begin{bmatrix} -1 & 2 \\ 2 & -4 \end{bmatrix} \] \[ -1v_1 + 2v_2 = 0 \quad \Rightarrow \quad v_2 = \frac{1}{2}v_1 \]

Eigenvector: \( \mathbf{v}_1 = k \begin{bmatrix} 1 \\ \frac{1}{2} \end{bmatrix} \).

For \( \lambda_2 = 2 \):

\[ (A - 2I) = \begin{bmatrix} 4 & 2 \\ 2 & 1 \end{bmatrix} \] \[ 4v_1 + 2v_2 = 0 \quad \Rightarrow \quad v_2 = -2v_1 \]

Eigenvector: \( \mathbf{v}_2 = k \begin{bmatrix} 1 \\ -2 \end{bmatrix} \).

Step 3: Construct \( P \) and \( D \)

Matrix \( P \) contains the eigenvectors as columns:

\[ P = \begin{bmatrix} 1 & 1 \\ \frac{1}{2} & -2 \end{bmatrix} \]

Matrix \( D \) is diagonal with eigenvalues:

\[ D = \begin{bmatrix} 7 & 0 \\ 0 & 2 \end{bmatrix} \]

Final Answer:

\[ A = PDP^{-1} \]

2. Singular Value Decomposition (SVD)

Problem:

Compute the SVD of:

\[ A = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix} \]

Solution:

Step 1: Compute \( A^T A \)

\[ A^T A = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}^T \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 4 \end{bmatrix} \]

Step 2: Compute Eigenvalues of \( A^T A \)

\[ \lambda_1 = 4, \quad \lambda_2 = 1 \]

The singular values (\( \sigma \)) are the square roots of the eigenvalues:

\[ \sigma_1 = 2, \quad \sigma_2 = 1 \]

Step 3: Compute \( U \), \( \Sigma \), and \( V \)

• Matrix \( \Sigma \): Diagonal matrix of singular values:
\[ \Sigma = \begin{bmatrix} 2 & 0 \\ 0 & 1 \end{bmatrix} \]
• Matrix \( V \): Eigenvectors of \( A^T A \):
\[ V = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \]
• Matrix \( U \): Compute \( U = AV / \Sigma \):
\[ U = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \]

Final Answer:

\[ A = U \Sigma V^T \]

3. Matrix Exponentials

Problem:

Find \( e^A \), where:

\[ A = \begin{bmatrix} 0 & 1 \\ -1 & 0 \end{bmatrix} \]

Solution:

Step 1: Use the Series Definition

\[ e^A = I + A + \frac{A^2}{2!} + \frac{A^3}{3!} + \cdots \]

Step 2: Compute Powers of \( A \)

\[ A^2 = \begin{bmatrix} 0 & 1 \\ -1 & 0 \end{bmatrix}^2 = \begin{bmatrix} -1 & 0 \\ 0 & -1 \end{bmatrix} = -I \] \[ A^3 = A^2 \cdot A = -A, \quad A^4 = A^2 \cdot A^2 = I \]

Step 3: Substitute into the Series

\[ e^A = \cos(1)I + \sin(1)A \]

Final Answer:

\[ e^A = \begin{bmatrix} \cos(1) & \sin(1) \\ -\sin(1) & \cos(1) \end{bmatrix} \]

This work is licensed under a Creative Commons Attribution 4.0 International License.