ECE 833 Individual Topics in Electrical Engineering: Statistical Foundations for Machine Learning

Time & Location: MW 2:30-3:45 PM, Online Synchronous

Instructor: Dr. Chau-Wai Wong, chauwai dot wong at ncsu.edu

Objective: (1) To prepare students with statistical foundations for machine learning (ML) research, and (2) to engage students in real-world machine learning tasks.

Prerequisites: Undergrad signal processing and probability/statistics. Basic programming skills. Talk to the instructor if the prerequisites are not met.

Followup ECE courses: ECE 542 Neural Networks (applying ML to neural nets), ECE 759 Pattern Recognition and Machine Learning (more ML algorithms), ECE 751 Detection and Estimation Theory (in-depth treatment of statistics)

Workload & Grading: There will be 4 homework assignments (30%), 1 term project (50%), and one midterm exam (20%).

Textbooks:

T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edition, Springer, 2009. [Online]
G. James, D. Witten, T. Hastie and R. Tibshirani, Introduction to Statistical Learning with Applications in R, Springer, 2013. [Online]
K. P. Murphy, Machine Learning: A Probabilistic Perspective, 2012. (Unity ID login required.)

Reference Books:

I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016. [Online]
J. L. Devore, Probability and Statistics for Engineering and the Sciences, Ed. 9.
H. Scheffe, The Analysis of Variance, Wiley, 1959.
J. J. Faraway, Linear Models with R, Taylor & Francis, 2005. [Online]

Some scanned sections/chapters of the books listed above can be found at NC State Course Reserves.

Topics:

I. Review in linear algebra and probability theory.

II. Statistics fundamentals

statistic, point estimate, method of moments, maximum likelihood (MLE), maximum a posteriori (MAP).
linear model, least-squares (LS), orthogonality principle, normal equations, variance–bias tradeoff.

III. Foundations of machine learning

curse of dimensionality, training vs. generalization errors, interpretability vs. prediction accuracy
regression vs. classification, generalized linear models, exponential family.
model selection: cross-validation (CV), analytical methods (AIC, BIC, MDL).
bootstrap, bagging, regularization.

Class #	Date	Topic	Lecture notes	Videos	Meeting time	Reading Assignment	HW Assignment
1	8/29	Review on probability	Notes Set 1	[V1] [V2]	2:30 pm	Your UG prob textbook	HW0 optional & HW1 (due in 2 weeks)
2	8/31	Review on probability (cont'd); Statistics fundamentals	Notes Set 1 Notes Set 2	[V2] [V3a]	2:30 pm
3	9/7	Statistics fundamentals (cont'd)	Notes Set 2	[V3b] [V3c]	2:30 pm	Devore 5.3–5, 6.1–2
4	9/12	Deep learning intro		Link	2:30 pm		Project planning
5	9/14	Linear models	Notes Set 3	Link	2:30 pm	Scheffe 1.1–2
6	9/19	Linear algebra review; Least-squares	Notes Set 3	Link	2:30 pm	Hayes 2.3.1–6, Scheffe App I–II; Scheffe 1.3	HW2 (due in 2 weeks)
7	9/21	Geometric interpretation	Notes Set 3	Link	2:30 pm		Project proposal due
Statistical Machine Learning
8	9/26	Regression function perspective	Slides^GoogleDrive (Annotated)	Link	2:30 pm	ISLR Ch2
9	9/28	Curse of dimensionality; Training vs. generalization errors	Slides^GoogleDrive (Annotated)	Link	2:30 pm	ESL Ch2	HW3 (due in 2 weeks) main_pca_visualization.m LoadImgData.m PcaViaKlt.m yalefaces.zip
10	10/3	Bias-variance tradeoff	Notes Set 4	Link	2:30 pm	ISLR 2.2.2 ESL 2.9, 7.3 Yang et al.
11	10/5	Eigenanalysis, PCA/KLT; Regularization	Notes Set 4	Link	2:30 pm	ESL 14.5.1; ISLR 6.2, ESL 3.4
12	10/12	Regularization	Slides^GoogleDrive (Annotated)	Link	2:30 pm	ISLR 6.2, ESL 3.4
13	10/17	Bayes classifier, Error metrics	Notes Set 6	Link	2:30 pm	Murphy 5.7.1–2 ESL 2.4	HW4 (due in 2 weeks)
14	10/19	Logistic regression	Notes Set 6	Link	2:30 pm	ISLR 4.3, ESL 4.4, Murphy 8.1–3
15	10/24	Generalized models	Notes Set 6	Link	2:30 pm	McCulloch 5.1–4
16	10/26	Model selection: cross-validation (CV) Analytical methods (AIC, BIC, MDL)	Slides (Annotated)	Link	2:30 pm	ESL 7.4–5, 7.10, ISLR 5.1
17	10/31	Bootstrap, Bagging	Slides^GoogleDrive (Annotated)	Link	2:30 pm	ISLR 5.2, ESL 7.11
18	11/2	Midterm		Link	2:30 pm