Vector Spaces

A vector space is a fundamental concept in linear algebra. It is a collection of objects called vectors, which can be added together and multiplied by scalars (numbers). These operations must satisfy certain axioms.

Formally, a vector space $V$ over a field $F$ (usually the real numbers $\mathbb{R}$ or complex numbers $\mathbb{C}$) is a set equipped with two operations: vector addition ($+$) and scalar multiplication ($\cdot$), such that for any vectors $\mathbf{u}, \mathbf{v}, \mathbf{w} \in V$ and any scalars $a, b \in F$:

1. Closure under addition: $\mathbf{u} + \mathbf{v} \in V$.
2. Associativity of addition: $(\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})$.
3. Commutativity of addition: $\mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}$.
4. Existence of additive identity: There exists a zero vector $\mathbf{0} \in V$ such that $\mathbf{v} + \mathbf{0} = \mathbf{v}$ for all $\mathbf{v} \in V$.
5. Existence of additive inverses: For every $\mathbf{v} \in V$, there exists $-\mathbf{v} \in V$ such that $\mathbf{v} + (-\mathbf{v}) = \mathbf{0}$.
6. Closure under scalar multiplication: $a \cdot \mathbf{v} \in V$.
7. Distributivity of scalar multiplication over vector addition: $a \cdot (\mathbf{u} + \mathbf{v}) = a \cdot \mathbf{u} + a \cdot \mathbf{v}$.
8. Distributivity of scalar multiplication over field addition: $(a+b) \cdot \mathbf{v} = a \cdot \mathbf{v} + b \cdot \mathbf{v}$.
9. Associativity of scalar multiplication: $(ab) \cdot \mathbf{v} = a \cdot (b \cdot \mathbf{v})$.
10. Existence of multiplicative identity: $1 \cdot \mathbf{v} = \mathbf{v}$, where $1$ is the multiplicative identity in $F$.

Examples of Vector Spaces

• The set of all $n$-tuples of real numbers, $\mathbb{R}^n$, with component-wise addition and scalar multiplication.
• The set of all $m \times n$ matrices with real entries, $M_{m \times n}(\mathbb{R})$.
• The set of all polynomials of degree at most $n$, $P_n(\mathbb{R})$.
• The set of all continuous real-valued functions on an interval $[a, b]$, $C[a, b]$.

Subspaces

A subset $W$ of a vector space $V$ is called a subspace if $W$ itself is a vector space under the same operations of vector addition and scalar multiplication inherited from $V$. A non-empty subset $W$ of $V$ is a subspace if and only if for any $\mathbf{u}, \mathbf{v} \in W$ and any scalar $c \in F$, $c\mathbf{u} + \mathbf{v} \in W$.

Every vector space has at least two trivial subspaces: the zero subspace $\{\mathbf{0}\}$ and the space $V$ itself.

Linear Independence

A set of vectors $\{\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_k\}$ in a vector space $V$ is said to be linearly independent if the only solution to the vector equation $c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + \cdots + c_k \mathbf{v}_k = \mathbf{0}$ is the trivial solution $c_1 = c_2 = \cdots = c_k = 0$.

If there exists a non-trivial solution (i.e., at least one $c_i \neq 0$), then the set of vectors is linearly dependent.

Intuitively, a set of vectors is linearly independent if none of the vectors can be expressed as a linear combination of the others. If a set is linearly dependent, at least one vector is redundant in the sense that it can be formed by scaling and adding the other vectors.

Example of Linear Independence

Consider the vectors $\mathbf{v}_1 = (1, 0)$ and $\mathbf{v}_2 = (0, 1)$ in $\mathbb{R}^2$. We want to solve $c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 = \mathbf{0}$. This gives $c_1 (1, 0) + c_2 (0, 1) = (0, 0)$, which simplifies to $(c_1, c_2) = (0, 0)$. The only solution is $c_1 = 0$ and $c_2 = 0$. Thus, $\{\mathbf{v}_1, \mathbf{v}_2\}$ is linearly independent.

Now consider $\mathbf{v}_1 = (1, 2)$, $\mathbf{v}_2 = (2, 4)$, and $\mathbf{v}_3 = (3, 1)$ in $\mathbb{R}^2$. We look for $c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + c_3 \mathbf{v}_3 = \mathbf{0}$. Notice that $\mathbf{v}_2 = 2\mathbf{v}_1$. So, we can write $2\mathbf{v}_1 - \mathbf{v}_2 + 0\mathbf{v}_3 = \mathbf{0}$. Here, $c_1=2, c_2=-1, c_3=0$ is a non-trivial solution. Thus, the set is linearly dependent.

Bases

A basis for a vector space $V$ is a set of vectors $B = \{\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n\}$ such that:

1. The set $B$ is linearly independent.
2. The set $B$ spans $V$. This means that every vector $\mathbf{v} \in V$ can be expressed as a linear combination of the vectors in $B$: $\mathbf{v} = c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + \cdots + c_n \mathbf{v}_n$ for some scalars $c_1, c_2, \ldots, c_n$.

If a vector space $V$ has a basis with $n$ vectors, then every basis for $V$ has exactly $n$ vectors. This number $n$ is called the dimension of the vector space $V$, denoted as $\dim(V)$.

A vector space that has a finite basis is called a finite-dimensional vector space. Otherwise, it is infinite-dimensional.

Examples of Bases and Dimensions

• The standard basis for $\mathbb{R}^n$ is $E = \{\mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_n\}$, where $\mathbf{e}_i$ is the vector with a 1 in the $i$-th position and 0s elsewhere. For $\mathbb{R}^3$, the standard basis is $\{(1, 0, 0), (0, 1, 0), (0, 0, 1)\}$. The dimension of $\mathbb{R}^n$ is $n$.
• The set $\{1, x, x^2, \ldots, x^n\}$ is a basis for the vector space of polynomials $P_n(x)$. The dimension is $n+1$.
• The zero vector space $\{\mathbf{0}\}$ has dimension 0. By convention, it has an empty basis.

Dual Spaces

Given a vector space $V$ over a field $F$, the dual space of $V$, denoted by $V^*$ or $V'$, is the vector space of all linear functionals on $V$. A linear functional is a linear map from $V$ to the field $F$.

Formally, $V^* = \{f: V \to F \mid f \text{ is a linear transformation}\}$.

If $V$ is finite-dimensional with dimension $n$, then its dual space $V^*$ is also finite-dimensional and has the same dimension, $\dim(V^*) = \dim(V) = n$.

Basis of a Dual Space

Let $B = \{\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n\}$ be a basis for a finite-dimensional vector space $V$. The dual basis for $V^*$ is a set of linear functionals $\{f_1, f_2, \ldots, f_n\}$ such that $f_i(\mathbf{v}_j) = \delta_{ij}$, where $\delta_{ij}$ is the Kronecker delta ($\delta_{ij} = 1$ if $i=j$, and $\delta_{ij} = 0$ if $i \neq j$).

For any vector $\mathbf{v} \in V$, if $\mathbf{v} = c_1 \mathbf{v}_1 + \cdots + c_n \mathbf{v}_n$, then the action of $f_i$ on $\mathbf{v}$ is $f_i(\mathbf{v}) = f_i(c_1 \mathbf{v}_1 + \cdots + c_n \mathbf{v}_n) = c_1 f_i(\mathbf{v}_1) + \cdots + c_n f_i(\mathbf{v}_n) = c_1 \cdot 0 + \cdots + c_i \cdot 1 + \cdots + c_n \cdot 0 = c_i$.

This means that the dual basis vectors $f_i$ extract the coefficients of the original basis vectors in the expansion of $\mathbf{v}$.

Example of Dual Space

Consider $V = \mathbb{R}^2$ with the standard basis $B = \{\mathbf{e}_1, \mathbf{e}_2\}$, where $\mathbf{e}_1 = (1, 0)$ and $\mathbf{e}_2 = (0, 1)$. The dual space $V^*$ also has dimension 2. Let the dual basis be $\{f_1, f_2\}$.

We require: $f_1(\mathbf{e}_1) = 1, f_1(\mathbf{e}_2) = 0$ $f_2(\mathbf{e}_1) = 0, f_2(\mathbf{e}_2) = 1$

A linear functional $f$ on $\mathbb{R}^2$ can be represented by a row vector $(a, b)$, so $f(\mathbf{x}) = (a, b) \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = ax_1 + bx_2$.

For $f_1$, we have $f_1(1, 0) = a(1) + b(0) = a = 1$, and $f_1(0, 1) = a(0) + b(1) = b = 0$. So $f_1$ corresponds to the row vector $(1, 0)$, and $f_1(x_1, x_2) = x_1$.

For $f_2$, we have $f_2(1, 0) = a(1) + b(0) = a = 0$, and $f_2(0, 1) = a(0) + b(1) = b = 1$. So $f_2$ corresponds to the row vector $(0, 1)$, and $f_2(x_1, x_2) = x_2$.

Thus, the dual basis is $\{f_1, f_2\}$, where $f_1$ is the projection onto the first coordinate and $f_2$ is the projection onto the second coordinate.

Inner Product Spaces

An inner product space is a vector space $V$ over the field of real or complex numbers, equipped with an operation called an inner product. The inner product takes two vectors and returns a scalar. It is typically denoted by $\langle \mathbf{u}, \mathbf{v} \rangle$.

The inner product must satisfy the following axioms for all vectors $\mathbf{u}, \mathbf{v}, \mathbf{w} \in V$ and scalars $a \in F$:

1. Linearity in the first argument (for complex spaces): $\langle a\mathbf{u} + b\mathbf{v}, \mathbf{w} \rangle = a\langle \mathbf{u}, \mathbf{w} \rangle + b\langle \mathbf{v}, \mathbf{w} \rangle$. (For real spaces, it's just linearity: $\langle \mathbf{u} + \mathbf{v}, \mathbf{w} \rangle = \langle \mathbf{u}, \mathbf{w} \rangle + \langle \mathbf{v}, \mathbf{w} \rangle$ and $\langle c\mathbf{u}, \mathbf{v} \rangle = c\langle \mathbf{u}, \mathbf{v} \rangle$.)
2. Conjugate symmetry: $\langle \mathbf{u}, \mathbf{v} \rangle = \overline{\langle \mathbf{v}, \mathbf{u} \rangle}$. (For real spaces, this simplifies to symmetry: $\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle$.)
3. Positive-definiteness: $\langle \mathbf{v}, \mathbf{v} \rangle \ge 0$, and $\langle \mathbf{v}, \mathbf{v} \rangle = 0$ if and only if $\mathbf{v} = \mathbf{0}$.

The inner product allows us to define geometric concepts like length (norm) and angle between vectors.

• The norm (or length) of a vector $\mathbf{v}$ is defined as $\|\mathbf{v}\| = \sqrt{\langle \mathbf{v}, \mathbf{v} \rangle}$.
• Two vectors $\mathbf{u}$ and $\mathbf{v}$ are orthogonal if their inner product is zero: $\langle \mathbf{u}, \mathbf{v} \rangle = 0$.

Examples of Inner Product Spaces

• $\mathbb{R}^n$ with the standard dot product: $\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u} \cdot \mathbf{v} = u_1 v_1 + u_2 v_2 + \cdots + u_n v_n$.
• $\mathbb{C}^n$ with the standard complex inner product: $\langle \mathbf{u}, \mathbf{v} \rangle = u_1 \overline{v_1} + u_2 \overline{v_2} + \cdots + u_n \overline{v_n}$.
• The space of continuous functions $C[a, b]$ with the inner product $\langle f, g \rangle = \int_a^b f(x) g(x) dx$.

Linear Transformations

A linear transformation (or linear map, or homomorphism) between two vector spaces $V$ and $W$ over the same field $F$ is a function $T: V \to W$ that preserves the operations of vector addition and scalar multiplication.

That is, for any vectors $\mathbf{u}, \mathbf{v} \in V$ and any scalar $c \in F$:

1. $T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})$ (preserves addition)
2. $T(c\mathbf{u}) = cT(\mathbf{u})$ (preserves scalar multiplication)

If $T$ satisfies these two conditions, it automatically satisfies $T(\mathbf{0}_V) = \mathbf{0}_W$ and $T(c_1 \mathbf{v}_1 + \cdots + c_k \mathbf{v}_k) = c_1 T(\mathbf{v}_1) + \cdots + c_k T(\mathbf{v}_k)$.

Examples of Linear Transformations

• The zero map: $T(\mathbf{v}) = \mathbf{0}_W$ for all $\mathbf{v} \in V$.
• The identity map: $T(\mathbf{v}) = \mathbf{v}$ for all $\mathbf{v} \in V$.
• Rotation of vectors in $\mathbb{R}^2$ by an angle $\theta$. The transformation matrix is $\begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$.
• Projection onto a subspace.
• Differentiation: $D: P_n(\mathbb{R}) \to P_{n-1}(\mathbb{R})$, where $D(p(x)) = p'(x)$.
• Integration: $I: C[a, b] \to C[a, b]$, where $I(f)(x) = \int_a^x f(t) dt$.

Matrix Representation of Linear Transformations

If $V$ and $W$ are finite-dimensional vector spaces, and we choose bases for both $V$ and $W$, then any linear transformation $T: V \to W$ can be represented by a matrix.

Let $\dim(V) = n$ and $\dim(W) = m$. Let $B_V = \{\mathbf{v}_1, \ldots, \mathbf{v}_n\}$ be a basis for $V$ and $B_W = \{\mathbf{w}_1, \ldots, \mathbf{w}_m\}$ be a basis for $W$.

To find the matrix representation of $T$, we apply $T$ to each basis vector of $V$. Each result $T(\mathbf{v}_j)$ is a vector in $W$, so it can be expressed as a linear combination of the basis vectors of $W$: $T(\mathbf{v}_j) = a_{1j} \mathbf{w}_1 + a_{2j} \mathbf{w}_2 + \cdots + a_{mj} \mathbf{w}_m$.

The $j$-th column of the matrix representation of $T$ (with respect to bases $B_V$ and $B_W$) is the column vector of coefficients $(a_{1j}, a_{2j}, \ldots, a_{mj})^T$. The resulting matrix $A$ will be an $m \times n$ matrix.

If $\mathbf{x}$ is a vector in $V$ with coordinate vector $[\mathbf{x}]_{B_V}$ relative to $B_V$, then the coordinate vector of $T(\mathbf{x})$ relative to $B_W$ is given by $[T(\mathbf{x})]_{B_W} = A [\mathbf{x}]_{B_V}$.

Example of Matrix Representation

Let $T: \mathbb{R}^2 \to \mathbb{R}^3$ be a linear transformation. Let the standard basis for $\mathbb{R}^2$ be $B_V = \{\mathbf{e}_1=(1,0), \mathbf{e}_2=(0,1)\}$ and the standard basis for $\mathbb{R}^3$ be $B_W = \{\mathbf{f}_1=(1,0,0), \mathbf{f}_2=(0,1,0), \mathbf{f}_3=(0,0,1)\}$.

Suppose $T(1,0) = (1, 2, 3)$ and $T(0,1) = (4, 0, -1)$.

We express $T(\mathbf{e}_1)$ and $T(\mathbf{e}_2)$ in terms of $B_W$: $T(\mathbf{e}_1) = (1, 2, 3) = 1\mathbf{f}_1 + 2\mathbf{f}_2 + 3\mathbf{f}_3$. The coefficients are $(1, 2, 3)^T$. $T(\mathbf{e}_2) = (4, 0, -1) = 4\mathbf{f}_1 + 0\mathbf{f}_2 + (-1)\mathbf{f}_3$. The coefficients are $(4, 0, -1)^T$.

The matrix representation $A$ of $T$ (with respect to the standard bases) has these coefficient vectors as its columns: $A = \begin{pmatrix} 1 & 4 \\ 2 & 0 \\ 3 & -1 \end{pmatrix}$.

To find $T(2, 5)$, we first find its coordinate vector in $B_V$, which is $\begin{pmatrix} 2 \\ 5 \end{pmatrix}$. Then, $[T(2, 5)]_{B_W} = A \begin{pmatrix} 2 \\ 5 \end{pmatrix} = \begin{pmatrix} 1 & 4 \\ 2 & 0 \\ 3 & -1 \end{pmatrix} \begin{pmatrix} 2 \\ 5 \end{pmatrix} = \begin{pmatrix} 1(2) + 4(5) \\ 2(2) + 0(5) \\ 3(2) + (-1)(5) \end{pmatrix} = \begin{pmatrix} 22 \\ 4 \\ 1 \end{pmatrix}$.

So, $T(2, 5) = 22\mathbf{f}_1 + 4\mathbf{f}_2 + 1\mathbf{f}_3 = (22, 4, 1)$.

Rank of a Linear Transformation

The rank of a linear transformation $T: V \to W$ is the dimension of its image (or range). The image of $T$, denoted $\text{Im}(T)$ or $R(T)$, is the set of all vectors in $W$ that are the result of applying $T$ to some vector in $V$: $\text{Im}(T) = \{ T(\mathbf{v}) \mid \mathbf{v} \in V \} = \{ \mathbf{w} \in W \mid \mathbf{w} = T(\mathbf{v}) \text{ for some } \mathbf{v} \in V \}$.

The image of $T$ is a subspace of $W$.

The rank-nullity theorem states that for a linear transformation $T: V \to W$ where $V$ is finite-dimensional: $\dim(V) = \text{rank}(T) + \text{nullity}(T)$ where $\text{rank}(T) = \dim(\text{Im}(T))$ and $\text{nullity}(T) = \dim(\text{Ker}(T))$.

The kernel (or null space) of $T$, denoted $\text{Ker}(T)$ or $N(T)$, is the set of all vectors in $V$ that are mapped to the zero vector in $W$: $\text{Ker}(T) = \{ \mathbf{v} \in V \mid T(\mathbf{v}) = \mathbf{0}_W \}$.

The kernel of $T$ is a subspace of $V$.

Rank of a Matrix

If a linear transformation $T: V \to W$ is represented by an $m \times n$ matrix $A$ (with respect to some bases), then the rank of the linear transformation $T$ is equal to the rank of the matrix $A$.

The rank of a matrix $A$ is defined as:

• The dimension of the column space of $A$ (the subspace spanned by the columns of $A$).
• The dimension of the row space of $A$ (the subspace spanned by the rows of $A$).
• The maximum number of linearly independent columns (or rows) of $A$.
• The size of the largest non-zero minor (determinant of a square submatrix) of $A$.

The rank of an $m \times n$ matrix $A$ is always less than or equal to $\min(m, n)$.

Example of Rank

Consider the linear transformation $T: \mathbb{R}^2 \to \mathbb{R}^3$ represented by the matrix $A = \begin{pmatrix} 1 & 4 \\ 2 & 0 \\ 3 & -1 \end{pmatrix}$.

The column space is spanned by the vectors $(1, 2, 3)^T$ and $(4, 0, -1)^T$. These two vectors are linearly independent (neither is a scalar multiple of the other). Therefore, the dimension of the column space is 2.

The rank of the matrix $A$ is 2.

Since $V = \mathbb{R}^2$, $\dim(V) = 2$. By the rank-nullity theorem, $\dim(V) = \text{rank}(T) + \text{nullity}(T)$. So, $2 = 2 + \text{nullity}(T)$, which implies $\text{nullity}(T) = 0$.

This means the kernel of $T$ contains only the zero vector, $\text{Ker}(T) = \{\mathbf{0}\}$. The nullity is 0.

The image of $T$ is a subspace of $\mathbb{R}^3$ with dimension 2. It is the plane in $\mathbb{R}^3$ spanned by $(1, 2, 3)^T$ and $(4, 0, -1)^T$.