02. Linear Algebra(1): Basic Elements of Linear Algebra

Linear Algebra is an important mathematical component
in data science.

 

 

 

2-1. Basic Elements of Linear Algebra

 

 

 

The goal is to understand,

• Basic elements of linear algebra
• Matrix notations and operations
• Linear system
• Solving a linear system

 

 

 

Scalar, Vector, and Matrix

 

 

 

 

Column Vector and Row Vector

 

 

 

 

Matrix Notations

 

 

 

 

Vector/Matrix Additions and Multiplications

 

 

 

 

Properties of Matrix Addition and Multiplication

 

1) Matrix multiplication is NOT commutative

 

AB ≠ BA

 

 

 

2) Other properties

 

 

 

 

Linear Equation

 

 

 

 

Linear System: Set of Equations

 

A system of linear equations (or a linear system) is a collection of one or more linear equations involving the same variables - say, x1, ... , xn

 

 

ex)

 

 

Once we solve for x1, x2, and x3,

given a new person with his or her weight, height, and is_smoking,

we can predict his or her life-span.

 

This process of solving a linear system and obtaining a prediction model from existing data is a key problem in data science.

 

 

 

From Multiple Equations to a Single Matrix Equation

 

The essential information of a linear system can be written compactly using a matrix.

 

 

 

 

How to solve for x

 

* How to solve for unknown variable x given the linear system

 

We first discuss the definition of an identity matrix and an inverse matrix.

 

 

1) Identity Matrix

 

An identity matrix is a square matrix whose diagonal entries are all 1's,

and all the other entries(off diagonal elements) are zeros.

 

Identity matrix has an important property that it preserves any vector X after multiplying it with itself.

 

 

 

2) Inverse Matrix

 

 

 

We can now solve Ax = b, and a linear system via inverse matrix as follows:

 

 

 

Back to the previous example,

 

 

Now the life-span can be written as:

(life-span) = -0.4 x (weight) + 20 x (height) - 20 x (is_smoking)

 

 

 

Problematic Case (1): Non-Invertible Matrix A for Ax = b

 

If Matrix A is invertible, the solution is uniquely obatined as:

 

 

 

In case A is non-invertible, i.e., the inverse matrix does not exist,

 

 

either no solution or infinitely many solutions(not unique) exist for Ax = b.

 

Both cases cause a big problem in practice.

 

In out life span example,

in case out linear system has no solution,

we cannot obtain the lifespan prediction model using our input.

 

On the other hand, if our linear system has multiple solutions that are not unique,

our prediction model cannot be uniquely determined,

resulting in one model giving the predicted lifespan as 60 for example,

while another model is giving it as 85, which is definitely confusing.

 

 

 

Problematic Case (2): Rectangular Matrix A in Ax = b

 

The inverse matrix is only defined for a square matrix.

What if matrix A has different number of rows and columns?

 

 

* m = #equations(number of equations), n = #variables

 

 

1) m < n : more variables than equations

 

We have lots of variables that we can freely choose,

but there are only a small number of equations that we have to satisfy.

Thereby in this case, usually infinitely many solutions exist (under-determined system).

 

 

2) m > n : more equations than variables

 

Usually no solution exists (over-determined system),

since we have to satisfy too many equations using only a small number of variables.

 

 

We have some advanced techniques that can handle these situations, which will be covered in later classes.