ECE492-45 Special Topics: Introduction to Machine Learning

Time & Location: MW4:30–5:45 pm @ 1226 Engineering Building II

Instructor: Dr. Chau-Wai Wong, chauwai.wong [ ] ncsu [ ] edu

Office hour: Tu4:30–5:45 pm or by appointment

Teaching Assistant: Runze Liu, rliu10 [ ] ncsu [ ] edu

Office hour: Th3:15–4:15 pm @ 2091 EB2

Objective: To introduce fundamental concepts in machine learning, and to provide students with hands-on experiences in machine learning applications.

Prerequisites: ST 300-level or above, and ECE 301/CSC 316. Talk to the instructor if not in ECE/CSC.

Workload & Grading: There will be weekly homework assignments that contains both written problems and programming problems (40%), two midterm exams (20%×2), and one final exam (20%). Programming will be in Python, R, and Matlab.

Discussions: ECE492-45 on Piazza

Textbooks:

G. James, D. Witten, T. Hastie and R. Tibshirani, Introduction to Statistical Learning with Applications in R, Springer, 2013. [Online]
T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning, (12th Printing, 2017), 2nd Ed., Springer. [Online]
I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016. [Online]

Reference Books:

J. L. Devore, Probability and Statistics for Engineering and the Sciences, Ed. 9.
A. Leon-Garcia, Probability, Statistics, and Random Processes for Electrical Engineering, Ed. 3, 2008.
C. E. McCulloch, S. R. Searle, Generalized, Linear, and Mixed Models, Wiley & Sons, 2001.
K. P. Murphy, Machine Learning: A Probabilistic Perspective, 2012.
R. O. Duda, P. E. Hart, D. G. Stork, Pattern Classification, 2nd Ed., Wiley, 2001.
E. Alpaydin, Introduction to Machine Learning, 2nd Ed., MIT Press, 2009.

Course Description: With the availability of huge datasets and the recent advancement in computational power, machine learning as a predictive tool has been increasingly successful in virtually all aspects of our life. In order to achieve the best results out of applying such tool, both solid understandings of the underlying principles and hands-on experiences are needed. This course introduces fundamental concepts in class and exposes students to real-world applications via well-guided homework programming problems.

Topics:

Linear statistical models, Bayesian classifiers, neural networks, support vector machine (SVM), classification/decision tree, clustering, principal component analysis, naive Bayes, topic model, hidden Markov model.

Class #	Date	Topic	Lecture notes	Readings	HW Assignment
Class 1	8/21	Introduction		Video: Can We Build a Brain?	HW1 (Due 8/28)
Class 2	8/26	Statistical learning basics	HT Ch1	ISLR Ch1–2
Class 3	8/28	Regression function, Conditional expectation	HT Ch2, Supp	ISLR Ch2; Devore 3.3, 4.2, 2.4; Leon 3.2, 5.7	HW2 (Due 9/9)
Class 4	9/4	Curse of dimensionality, Model accuracy	HT Ch2	ISLR Ch2; Devore CH6
Class 5	9/9	Addressed HW questions, Bias-variance decomposition	HT Ch3	ISLR Ch3	HW3 (Due 9/18)
Class 6	9/11	Classification examples, Linear regression	HT Ch3, Supp	ISLR Ch3
Linear Regression
Class 7	9/16	Addressed HW questions, Matrix-vector form	HT Ch3	ESL 3.2
Class 8	9/18	Confidence interval, Hypothesis test	HT Ch3	ISLR 3.1, Devore 7.1, 8.1, 8.3	HW4 (Due 9/25)
Class 9	9/23	Multiple regression	HT Ch3	ISLR Ch3
Class 10	9/25	Variables selection, Interaction, Qualitative predictors	HT Ch3	ISLR Ch3	HW5 (Due 10/2)
Classification
Class 11	9/30	Logistic regression	HT Ch4	ISLR Ch4, ESL 4.4
Class 12	10/2	Linear discriminant analysis	HT Ch4	ISLR 4.4, Leon 6.3.1, 6.4	HW6 (Due 10/9)
Class 13	10/7	Error types, ROC, AUC, EER	HT Ch4	ISLR 4.4.3, Murphy 5.7.2.1
Class 14	10/9	Exam guide; Naive Bayes; Logistic vs. LDA; MLE	Handwritten	ESL 6.6.3, Murphy 3.5; ISLR 4.5; Devore 6.2	No HW. Prepare for Exam 1.
Class 15	10/14	MLE (cont'd), Invariance principle	Handwritten	Devore 6.2
Class 16	10/16	Exam 1			Exam 1
Class 17	10/21	Link function for GLM; Cross-Validation	HT Ch5, Supp	McCulloch 5.1–4 (updated); ISLR 5.1	HW7 (Due 10/28)
Resampling Methods
Class 18	10/23	Cross-Validation	HT Ch5, Supp 1, 2	ISLR 5.1
Class 19	10/28	Addressed HW questions; Cross-Validation (cont'd)	HT Ch5	ISLR 5.1	HW8 (Due 11/6)
Regularization and Dimensionality Reduction
Class 20	10/30	Bootstrap; Regularization	HT Ch5; HT Ch6	ISLR 5.2; 6.2
Machine Learning Algorithms
Class 21	11/4	Support vector machines	HT Ch9	ISLR Ch9	HW9 (Due 11/13)
Class 22	11/6	Support vector machines (cont'd); Regularization (cont'd)	HT Ch9	ISLR Ch9; 6.2	Project Proposal Guide Example Proposal (Due 11/18)
Class 23	11/11	Neural network intro	Slides	ESL 11.3–8, Goodfellow Ch6–9, Duda Ch6
Class 24	11/13	Gradient descent, Backpropagation	Slides		HW10 (Due 11/20)
Class 25	11/18	Autoencoders, Deepfake; CNN architectures	Slides	Goodfellow Ch14; CS231N Lec9
Class 26	11/20	Exam Review
Class 27	11/25	Exam 2			Exam 2
Class 28	12/2	PCA	HT Ch10	ISLR 10.2	HW11 (Due 12/9)
Class 29	12/4	K-Means; HMM; Topic model	HT Ch10; Supp 1; 2	ISLR 10.3; Topic model 1, 2	Project Report Submission Guide (Due 12/18)

Class #

Date

Topic

Lecture notes

Readings

HW Assignment

Class 1

8/21

Introduction

Video: Can We Build a Brain?

HW1 (Due 8/28)

Class 2

8/26

Statistical learning basics

HT Ch1

ISLR Ch1–2

Class 3

8/28

Regression function, Conditional expectation

HT Ch2, Supp

ISLR Ch2; Devore 3.3,
4.2, 2.4; Leon 3.2, 5.7

HW2 (Due 9/9)

Class 4

9/4

Curse of dimensionality, Model accuracy

HT Ch2

ISLR Ch2; Devore CH6

Class 5

9/9

Addressed HW questions, Bias-variance decomposition

HT Ch3

ISLR Ch3

HW3 (Due 9/18)

Class 6

9/11

Classification examples, Linear regression

HT Ch3, Supp

ISLR Ch3

Linear Regression

Class 7

9/16

Addressed HW questions, Matrix-vector form

HT Ch3

ESL 3.2

Class 8

9/18

Confidence interval, Hypothesis test

HT Ch3

ISLR 3.1, Devore 7.1,
8.1, 8.3

HW4 (Due 9/25)

Class 9

9/23

Multiple regression

HT Ch3

ISLR Ch3

Class 10

9/25

Variables selection, Interaction, Qualitative predictors

HT Ch3

ISLR Ch3

HW5 (Due 10/2)

Classification

Class 11

9/30

Logistic regression

HT Ch4

ISLR Ch4, ESL 4.4

Class 12

10/2

Linear discriminant analysis

HT Ch4

ISLR 4.4, Leon 6.3.1, 6.4

HW6 (Due 10/9)

Class 13

10/7

Error types, ROC, AUC, EER

HT Ch4

ISLR 4.4.3, Murphy 5.7.2.1

Class 14

10/9

Exam guide; Naive Bayes; Logistic vs. LDA; MLE

Handwritten

ESL 6.6.3, Murphy 3.5;
ISLR 4.5; Devore 6.2

No HW. Prepare for Exam 1.

Class 15

10/14

MLE (cont'd), Invariance principle

Handwritten

Devore 6.2

Class 16

10/16

Exam 1

Class 17

10/21

Link function for GLM; Cross-Validation

HT Ch5, Supp

McCulloch 5.1–4 (updated);
ISLR 5.1

HW7 (Due 10/28)

Resampling Methods

Class 18

10/23

Cross-Validation

HT Ch5, Supp 1, 2

ISLR 5.1

Class 19

10/28

Addressed HW questions; Cross-Validation (cont'd)

HT Ch5

ISLR 5.1

HW8 (Due 11/6)

Regularization and Dimensionality Reduction

Class 20

10/30

Bootstrap; Regularization

HT Ch5; HT Ch6

ISLR 5.2; 6.2

Machine Learning Algorithms

Class 21

11/4

Support vector machines

HT Ch9

ISLR Ch9

HW9 (Due 11/13)

Class 22

11/6

Support vector machines (cont'd); Regularization (cont'd)

HT Ch9

ISLR Ch9; 6.2

Project Proposal Guide
Example Proposal
(Due 11/18)

Class 23

11/11

Neural network intro

Slides

ESL 11.3–8, Goodfellow
Ch6–9, Duda Ch6

Class 24

11/13

Gradient descent, Backpropagation

Slides

HW10 (Due 11/20)

Class 25

11/18

Autoencoders, Deepfake; CNN architectures

Slides

Goodfellow Ch14;
CS231N Lec9

Class 26

11/20

Exam Review

Class 27

11/25

Exam 2

Class 28

12/2

PCA

HT Ch10

ISLR 10.2

HW11 (Due 12/9)

Class 29

12/4

K-Means; HMM; Topic model

HT Ch10; Supp 1; 2

ISLR 10.3;
Topic model 1, 2

Project Report
Submission Guide
(Due 12/18)