📘 Section 12.1.2 – Properties of Least Squares Estimators

🎯 Why This Matters

Understanding the properties of least squares estimators is crucial for interpreting regression results. It allows students to assess the reliability and accuracy of predictive models for outcomes like GPA, job offers, or salaries.

🔢 Model Setup

The general multiple linear regression model in matrix form is:

Y = Xβ + ε

Y: n×1 vector of responses (e.g., GPA)
X: n×(k+1) matrix of predictors (e.g., study time, sleep, attendance)
β: (k+1)×1 vector of unknown coefficients
ε: vector of random errors, E(ε) = 0 and Var(ε) = σ²I

📐 Least Squares Estimator

The least squares estimator minimizes the sum of squared residuals:

β̂ = (XᵀX)⁻¹XᵀY

🧠 Key Properties

Linearity: β̂ is a linear function of Y
Unbiasedness: E(β̂) = β
Variance: Var(β̂) = σ²(XᵀX)⁻¹

📊 Estimating σ² (Error Variance)

The error variance is estimated by:

σ̂² = SSE / (n − k − 1)

Where SSE = ∑(Yᵢ − Ŷᵢ)². This is also called the Mean Squared Error (MSE).

📈 Covariance Matrix of β̂

The covariance matrix for the estimated coefficients is:

Cov(β̂) = σ̂² (XᵀX)⁻¹

This helps assess the variability and potential correlation between coefficient estimates.

🎓 Student-Based Example

Suppose we want to model a student's GPA based on:

x₁ = Study hours per week
x₂ = Sleep hours per night
x₃ = Number of classes attended per week

The model is:

GPA = β₀ + β₁x₁ + β₂x₂ + β₃x₃ + ε

🧮 Least Squares Estimation Example

Step 1: Least Squares Function

Given n = 5 observations, the least squares objective is:

L = ∑_i=1⁵ [y_i − (β₀ + β₁x_i1 + β₂x_i2 + β₃x_i3)]²

Step 2: Matrix Form

X = [ [1, 15, 7.0, 40],
[1, 20, 8.0, 42],
[1, 10, 6.0, 35],
[1, 12, 6.5, 38],
[1, 18, 7.5, 41]
]
Y = [3.5, 3.8, 2.9, 3.2, 3.7]^T

Step 3: Estimate Coefficients

Apply the formula:

β̂ = (X^TX)⁻¹X^TY

Steps involved:

Compute X^TX (4×4 matrix)
Compute X^TY (4×1 vector)
Multiply the inverse of X^TX with X^TY to get β̂