Matrix Diagonalization and Similar Matrices
Introduction to Diagonalization
Diagonalization is a fundamental process in linear algebra that simplifies the study of matrices. It involves transforming a given square matrix into a diagonal matrix through a similarity transformation. A diagonal matrix is a matrix where all the off-diagonal elements are zero. Such matrices are exceptionally easy to work with, especially when it comes to operations like exponentiation or solving systems of linear differential equations. The core idea is to find a new basis in which the linear transformation represented by the matrix is represented by a diagonal matrix.
The process of diagonalization is closely tied to the eigenvalues and eigenvectors of a matrix. Eigenvectors are special non-zero vectors that, when multiplied by a matrix, result in a scaled version of themselves. The scaling factor is the corresponding eigenvalue. If a matrix has a full set of linearly independent eigenvectors, it can be diagonalized.
Eigenvalues and Eigenvectors
Before we can diagonalize a matrix, we must first understand its eigenvalues and eigenvectors. For a square matrix $A$, a non-zero vector $v$ is an eigenvector if $Av = \lambda v$, where $\lambda$ is a scalar known as the eigenvalue.
To find the eigenvalues, we rearrange the equation: $Av - \lambda v = 0$, which can be written as $(A - \lambda I)v = 0$, where $I$ is the identity matrix of the same dimension as $A$. For this equation to have a non-trivial solution (i.e., $v \neq 0$), the matrix $(A - \lambda I)$ must be singular. This means its determinant must be zero:
$$ \det(A - \lambda I) = 0 $$
This equation is called the characteristic equation, and its roots are the eigenvalues of the matrix $A$.
Once the eigenvalues are found, we can find the corresponding eigenvectors. For each eigenvalue $\lambda_i$, we solve the system of linear equations $(A - \lambda_i I)v = 0$ to find the non-zero vector(s) $v$ that satisfy the definition of an eigenvector. The set of all eigenvectors corresponding to a particular eigenvalue, along with the zero vector, forms a subspace called the eigenspace.
Diagonalizable Matrices
A square matrix $A$ is diagonalizable if there exists an invertible matrix $P$ and a diagonal matrix $D$ such that $A = PDP^{-1}$. The diagonal entries of $D$ are the eigenvalues of $A$, and the columns of $P$ are the corresponding eigenvectors.
A crucial theorem states that an $n \times n$ matrix $A$ is diagonalizable if and only if it has $n$ linearly independent eigenvectors. This means that the sum of the dimensions of the eigenspaces must equal $n$.
If a matrix has $n$ distinct eigenvalues, then it is guaranteed to be diagonalizable. This is because eigenvectors corresponding to distinct eigenvalues are always linearly independent. However, a matrix with repeated eigenvalues might still be diagonalizable if the geometric multiplicity (the dimension of the eigenspace) of each eigenvalue equals its algebraic multiplicity (the number of times the eigenvalue appears as a root of the characteristic equation).
Let's consider an example. Suppose we have a matrix $A$ and we find its eigenvalues $\lambda_1, \lambda_2, \dots, \lambda_n$ and their corresponding linearly independent eigenvectors $v_1, v_2, \dots, v_n$. We can form the matrix $P$ by using the eigenvectors as its columns: $P = [v_1 | v_2 | \dots | v_n]$. The diagonal matrix $D$ will have the eigenvalues on its diagonal, in the same order as their corresponding eigenvectors in $P$:
$$ D = \begin{pmatrix} \lambda_1 & 0 & \dots & 0 \\ 0 & \lambda_2 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & \lambda_n \end{pmatrix} $$
Then, the relationship $A = PDP^{-1}$ holds. This can be rewritten as $AP = PD$. Multiplying by $P^{-1}$ on the right gives $A = PDP^{-1}$.
The Process of Diagonalization
To diagonalize a matrix $A$, follow these steps:
- Find the eigenvalues of $A$ by solving the characteristic equation $\det(A - \lambda I) = 0$.
- For each eigenvalue $\lambda_i$, find the corresponding eigenvectors by solving the system $(A - \lambda_i I)v = 0$.
- Check if there are $n$ linearly independent eigenvectors. If not, the matrix cannot be diagonalized.
- If there are $n$ linearly independent eigenvectors, form the matrix $P$ whose columns are these eigenvectors.
- Form the diagonal matrix $D$ whose diagonal entries are the corresponding eigenvalues in the same order as the eigenvectors in $P$.
- The diagonalization is given by $A = PDP^{-1}$.
Calculating $P^{-1}$ can be computationally intensive. However, the purpose of diagonalization is often to understand the properties of $A$ through $D$, or to compute powers of $A$. For instance, $A^k = (PDP^{-1})^k = PD P^{-1} P D P^{-1} \dots P D P^{-1} = PD^k P^{-1}$. Calculating $D^k$ is trivial for a diagonal matrix: simply raise each diagonal element to the power of $k$.
Example: Diagonalize the matrix $A = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}$.
Step 1: Find eigenvalues. The characteristic equation is $\det(A - \lambda I) = 0$. $$ A - \lambda I = \begin{pmatrix} 4-\lambda & -2 \\ 1 & 1-\lambda \end{pmatrix} $$ $$ \det(A - \lambda I) = (4-\lambda)(1-\lambda) - (-2)(1) = 4 - 4\lambda - \lambda + \lambda^2 + 2 = \lambda^2 - 5\lambda + 6 $$ Setting the determinant to zero: $\lambda^2 - 5\lambda + 6 = 0$. Factoring gives $(\lambda - 2)(\lambda - 3) = 0$. The eigenvalues are $\lambda_1 = 2$ and $\lambda_2 = 3$.
Step 2: Find eigenvectors. For $\lambda_1 = 2$: Solve $(A - 2I)v = 0$. $$ \begin{pmatrix} 4-2 & -2 \\ 1 & 1-2 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 2 & -2 \\ 1 & -1 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix} $$ Both rows give the equation $x - y = 0$, or $x = y$. Let $y = t$. Then $x = t$. The eigenvectors are of the form $t \begin{pmatrix} 1 \\ 1 \end{pmatrix}$. A corresponding eigenvector is $v_1 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}$.
For $\lambda_2 = 3$: Solve $(A - 3I)v = 0$. $$ \begin{pmatrix} 4-3 & -2 \\ 1 & 1-3 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 1 & -2 \\ 1 & -2 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix} $$ Both rows give the equation $x - 2y = 0$, or $x = 2y$. Let $y = t$. Then $x = 2t$. The eigenvectors are of the form $t \begin{pmatrix} 2 \\ 1 \end{pmatrix}$. A corresponding eigenvector is $v_2 = \begin{pmatrix} 2 \\ 1 \end{pmatrix}$.
Step 3: Check for linear independence. We have two distinct eigenvalues, so the eigenvectors $v_1 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}$ and $v_2 = \begin{pmatrix} 2 \\ 1 \end{pmatrix}$ are guaranteed to be linearly independent. The matrix $A$ is $2 \times 2$, and we have 2 linearly independent eigenvectors, so $A$ is diagonalizable.
Step 4: Form matrix $P$. $$ P = [v_1 | v_2] = \begin{pmatrix} 1 & 2 \\ 1 & 1 \end{pmatrix} $$
Step 5: Form matrix $D$. The eigenvalues are $\lambda_1 = 2$ and $\lambda_2 = 3$, corresponding to $v_1$ and $v_2$ respectively. $$ D = \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} $$
Step 6: The diagonalization is $A = PDP^{-1}$. We can verify this by calculating $P^{-1}$. The determinant of $P$ is $(1)(1) - (2)(1) = 1 - 2 = -1$. $$ P^{-1} = \frac{1}{-1} \begin{pmatrix} 1 & -2 \\ -1 & 1 \end{pmatrix} = \begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix} $$ Now, calculate $PDP^{-1}$: $$ PDP^{-1} = \begin{pmatrix} 1 & 2 \\ 1 & 1 \end{pmatrix} \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} \begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix} $$ $$ = \begin{pmatrix} 1 \cdot 2 + 2 \cdot 0 & 1 \cdot 0 + 2 \cdot 3 \\ 1 \cdot 2 + 1 \cdot 0 & 1 \cdot 0 + 1 \cdot 3 \end{pmatrix} \begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 2 & 6 \\ 2 & 3 \end{pmatrix} \begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix} $$ $$ = \begin{pmatrix} 2(-1) + 6(1) & 2(2) + 6(-1) \\ 2(-1) + 3(1) & 2(2) + 3(-1) \end{pmatrix} = \begin{pmatrix} -2 + 6 & 4 - 6 \\ -2 + 3 & 4 - 3 \end{pmatrix} = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix} $$ This is indeed the original matrix $A$.
Similar Matrices
Two square matrices $A$ and $B$ of the same size are called similar if there exists an invertible matrix $P$ such that $B = P^{-1}AP$. The transformation $A \mapsto P^{-1}AP$ is called a similarity transformation.
The concept of similarity is central to diagonalization. If a matrix $A$ is diagonalizable, then it is similar to a diagonal matrix $D$. Specifically, if $A = PDP^{-1}$, then $D = P^{-1}AP$. Thus, $A$ and $D$ are similar matrices.
Similar matrices share many important properties. This is why similarity is such a useful concept. If $A$ and $B$ are similar, then:
- They have the same determinant: $\det(B) = \det(P^{-1}AP) = \det(P^{-1})\det(A)\det(P) = (\det(P))^{-1}\det(A)\det(P) = \det(A)$.
- They have the same trace: $\operatorname{tr}(B) = \operatorname{tr}(P^{-1}AP) = \operatorname{tr}(APP^{-1}) = \operatorname{tr}(A)$. (The trace is the sum of the diagonal elements).
- They have the same characteristic polynomial, and thus the same eigenvalues.
- They have the same rank.
- If $A$ is invertible, then $B$ is also invertible and $B^{-1} = (P^{-1}AP)^{-1} = P^{-1}A^{-1}(P^{-1})^{-1} = P^{-1}A^{-1}P$.
- For any positive integer $k$, $A^k$ and $B^k$ are also similar: $B^k = (P^{-1}AP)^k = P^{-1}A^k P$.
The fact that similar matrices have the same eigenvalues is particularly important. If we can find a matrix $P$ such that $P^{-1}AP = D$ (a diagonal matrix), then the eigenvalues of $A$ are simply the diagonal entries of $D$. This provides a straightforward way to find eigenvalues if we can find the similarity transformation.
Conditions for Diagonalizability
An $n \times n$ matrix $A$ is diagonalizable if and only if the sum of the dimensions of its eigenspaces equals $n$. This is equivalent to saying that for each eigenvalue $\lambda$, its algebraic multiplicity (the multiplicity of $\lambda$ as a root of the characteristic polynomial) is equal to its geometric multiplicity (the dimension of the eigenspace corresponding to $\lambda$).
Let $m_a(\lambda)$ be the algebraic multiplicity of eigenvalue $\lambda$, and $m_g(\lambda)$ be the geometric multiplicity. For an $n \times n$ matrix $A$, $A$ is diagonalizable if and only if:
- The characteristic polynomial splits completely into linear factors (i.e., the sum of algebraic multiplicities of all eigenvalues is $n$). This is always true if we allow complex eigenvalues.
- For every eigenvalue $\lambda$, $m_a(\lambda) = m_g(\lambda)$.
We know that for any eigenvalue $\lambda$, $1 \le m_g(\lambda) \le m_a(\lambda)$. Therefore, the condition simplifies to checking if $m_g(\lambda) = m_a(\lambda)$ for all eigenvalues.
If an $n \times n$ matrix $A$ has $n$ distinct eigenvalues, then it is always diagonalizable. This is because for distinct eigenvalues, the geometric multiplicity is always 1, and the algebraic multiplicity is also 1.
Consider a matrix that is NOT diagonalizable. This happens when $m_g(\lambda) < m_a(\lambda)$ for at least one eigenvalue $\lambda$.
Example of a non-diagonalizable matrix: Let $A = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$.
Step 1: Find eigenvalues. $$ \det(A - \lambda I) = \det \begin{pmatrix} 1-\lambda & 1 \\ 0 & 1-\lambda \end{pmatrix} = (1-\lambda)^2 = 0 $$ The only eigenvalue is $\lambda = 1$, with algebraic multiplicity $m_a(1) = 2$.
Step 2: Find eigenvectors. Solve $(A - 1I)v = 0$. $$ \begin{pmatrix} 1-1 & 1 \\ 0 & 1-1 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix} $$ This gives the equation $0x + 1y = 0$, so $y = 0$. The variable $x$ can be any real number. The eigenvectors are of the form $\begin{pmatrix} x \\ 0 \end{pmatrix} = x \begin{pmatrix} 1 \\ 0 \end{pmatrix}$. The eigenspace is spanned by the single vector $\begin{pmatrix} 1 \\ 0 \end{pmatrix}$. The geometric multiplicity of $\lambda = 1$ is $m_g(1) = 1$.
Since $m_g(1) = 1 < m_a(1) = 2$, the matrix $A$ is not diagonalizable. It does not have 2 linearly independent eigenvectors.
Applications of Diagonalization
Diagonalization has numerous applications in various fields of science and engineering.
1. Matrix Powers: As mentioned earlier, computing $A^k$ is greatly simplified. If $A = PDP^{-1}$, then $A^k = PD^k P^{-1}$. This is particularly useful for large values of $k$.
2. Systems of Linear Differential Equations: Consider a system of first-order linear differential equations $\frac{d\mathbf{x}}{dt} = A\mathbf{x}$. If $A$ is diagonalizable, $A = PDP^{-1}$, we can transform the system into an uncoupled system. Let $\mathbf{y} = P^{-1}\mathbf{x}$. Then $\mathbf{x} = P\mathbf{y}$, and $\frac{d\mathbf{x}}{dt} = P\frac{d\mathbf{y}}{dt}$. Substituting into the original equation: $P\frac{d\mathbf{y}}{dt} = A(P\mathbf{y}) = (PDP^{-1})P\mathbf{y} = PD\mathbf{y}$. Multiplying by $P^{-1}$ on the left: $\frac{d\mathbf{y}}{dt} = D\mathbf{y}$. If $D = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$, this becomes a system of $n$ independent equations: $\frac{dy_i}{dt} = \lambda_i y_i$. The solutions are $y_i(t) = c_i e^{\lambda_i t}$. We can then transform back to find the solution for $\mathbf{x}(t)$ using $\mathbf{x} = P\mathbf{y}$.
3. Markov Chains: In probability theory, the long-term behavior of Markov chains is often analyzed using matrix diagonalization. The transition matrix of a Markov chain can be diagonalized to find the steady-state distribution.
4. Principal Component Analysis (PCA): In statistics and machine learning, PCA is a technique used for dimensionality reduction. It relies on finding the eigenvectors and eigenvalues of the covariance matrix of the data. Diagonalization is implicitly used to transform the data into a new coordinate system where the components are uncorrelated.
5. Quadratic Forms: Diagonalization is used to simplify quadratic forms $Q(\mathbf{x}) = \mathbf{x}^T A \mathbf{x}$. By changing to a basis of eigenvectors, the quadratic form can be expressed as a sum of squares with coefficients being the eigenvalues.
Summary of Key Concepts
Matrix diagonalization is the process of finding an invertible matrix $P$ and a diagonal matrix $D$ such that $A = PDP^{-1}$. This is possible if and only if the matrix $A$ has a full set of linearly independent eigenvectors.
Similar matrices are matrices $A$ and $B$ such that $B = P^{-1}AP$ for some invertible matrix $P$. Similar matrices share many properties, including determinant, trace, and eigenvalues. Diagonalization shows that a diagonalizable matrix is similar to a diagonal matrix.
The process involves finding eigenvalues and eigenvectors. For an $n \times n$ matrix to be diagonalizable, the sum of the dimensions of its eigenspaces must equal $n$, or equivalently, the geometric multiplicity of each eigenvalue must equal its algebraic multiplicity.