Essence of Linear Algebra

1 English-Chinese Glossary

English	中文
cross production	叉积
determinantion	行列式
eigenvalue	特征值

2 Vector

The introduction of numbers as coordinates is an act of violence.

AND on the flip side, it gives people a language to describe space and the manipulation of space using numbers that can be crunched and run through a computer. Bold statement: Linear algebra allows programmers to manipulate space.

$i$ and $j$ are basis vectors, and any vector can be considered a linear combination of them.

Collinear vectors are linearly dependent, and the space they span is just a line (or the origin); Non-collinear vectors are linearly independent, and the space they span is the set of all vectors;

3 Matrix

kind of Linear transformation

Fortunately, linear algebra only involves linear transformations;

Matrices can be understood as transformations; ac represents the position after a basis transformation, bd as well, so 1001 is equivalent to no transformation; Knowing how the basis transforms tells you how all vectors transform; Orthogonal transformations are those where basis vectors maintain their length and remain orthogonal (rigid body motion);

4 Matrix Multiply

A single matrix is a linear transformation, so matrix multiplication is a composite transformation. The collective meaning of matrix multiplication is to apply one linear transformation and then another. Non-square matrices represent transformations across dimensions;

5 Determinant

This is the determinant! 👆 The area of a 1x1 small square after matrix transformation equals the value of the corresponding determinant.

If the determinant is 0, everything is flattened, which is a non-invertible transformation, and the matrix is also non-invertible. The value of the determinant can be negative. Can your area be negative too? The area equals the absolute value, and a negative number indicates a change in spatial orientation (a piece of paper flipped over);

However, area cannot explain everything; in higher dimensions, it is something else;

6 Inverse matrices & Column space & Rank

Matrices not only manipulate space but can also be used to solve systems of equations.

Convert the system of equations into matrix multiplication, naturally returning to the traditional skill of manipulating space; $\vec{x}$ transforms into $\vec{v}$ under the action of matrix $A$, and using the inverse transformation $A^{-1}$ to find the original $\vec{x}$ is the process of solving the system of equations; When the determinant is not zero, finding the inverse matrix can solve the system of equations;

When the determinant is zero, the system of equations may still have a solution, provided that $\vec{v}$ survives in the compressed space (column space);

For an explanation of rank, video around 8 minutes is really exquisite. Rank represents the dimension of the transformed (column) space; (in a system of equations, the rank of the matrix is exactly the number of constraints) The set of all possible $A\vec{v}$ is the column space; The set of vectors that transform to the origin is the null space or kernel; Kernel methods in SVM?

7 Duality of Dot Product

The traditional way of understanding the dot product of vectors is projection, but to understand duality, forget it first. Duality refers to a natural yet unexpected correspondence. Vector is the physical embodiment of a linear transformation. Understanding duality is crucial for understanding Hilbert space, PCA, IDA.

There is a wonderful relationship between vectors and the corresponding 1×n matrices. The transformation represented by a 1×n matrix is equivalent to performing a dot product with an n×1 vector; every vector is an incarnation of some matrix; every matrix corresponds to some vector;

8 Cross Product

The traditional explanation is as shown in the image above.

9 Change of Basis

How do two people in different coordinate systems communicate? By placing the other person’s basis vectors into their own coordinate system to obtain the transformation matrix.

For a vector $\vec{v}$ in another coordinate system, first use a transformation (left three) to convert it into a vector in our own coordinate system, then perform the transformation in our own coordinate system (left middle), and finally convert the transformation result into their coordinates (left one); The expression $A^{-1}MA$ represents a kind of transfer action, and the product of such matrices is still a transformation, for others.

10 Eigenvectors & Eigenvalues

Function: Eigenvectors with eigenvalue 1 are the axis of rotation. Calculation: $A\vec{v}=\lambda \vec{v}$, after rearranging, $det(A-\lambda I)=0$, i.e., find a vector that can compress space. Rotational transformation eigenvectors are in the complex vector space, shear transformation has only one eigenvector; There are also cases where the eigenvalue is unique, and the eigenvectors are not collinear (e.g., stretching all vectors twice).

Diagonal matrices, all basis vectors are eigenvectors, and the values on the diagonal are the corresponding eigenvalues.

If one day, you want to use two non-collinear eigenvectors [1 0] [0 1] as the basis of a new coordinate system, this process of basis transformation is similar diagonalization; the resulting matrix must be a diagonal matrix, and the values are eigenvalues. Such eigenvectors are also called eigenbasis; Why go to great lengths to perform eigenbasis transformation? For example, the above $\begin{bmatrix}3 &1 \ 0 & 2\end{bmatrix}$, calculating such a transformation 100 times would be very complex, but after transformation, you can quickly get the result $\begin{bmatrix}3^{100} &1 \ 0 & 2^{100}\end{bmatrix}$, and then transform it back.

11 Vector Spaces

Determinants, eigenvectors, etc., are independent of the chosen coordinate system… Functions can also be differentiated using matrices… So what exactly is a vector?

Vectors are not a thing. Axioms are not natural laws; they are rules defined by mathematicians, connecting mathematicians with those who use mathematical tools; vectors can be anything, points, arrows, functions, strange creatures…, as long as they satisfy the rules defined by these axioms. Asking what a vector is is equivalent to asking what “1” is, which is meaningless.

12 Cramer Rule

For solving determinants, computers use Cramer’s rule, while humans use Gaussian elimination; but Cramer’s rule is much more interesting.

A unique way to represent coordinates: y = area/1, x = area / 1;

After such a transformation, y is still the area of the quadrilateral based on the green basis, which exactly fits the geometric meaning of the determinant; At this time, the area of the quadrilateral, with the green basis unchanged (the first column of the determinant), the height becomes the transformed 42. This is the geometric meaning of Cramer’s rule.