ECE 50024 / STAT 59800 Machine Learning

Spring 2023 Jan 9, 2023 - Apr 29, 2023

Recorded videos of the lectures can be found in Brightspace. The schedule is subject to change.

Week 1

• Jan 10, 2022. Lecture 0.
  • Topics: Course overview and linear regression I. Course Whiteboard
  • Reading:
    • Python Tutorial.
    • Python for Matrices. (Password is in BrightSpace.)
    • Python for Plotting. (Password is in BrightSpace.)
    • Linear Algebra Review.
    • Probability Review.
• Jan 12, 2022. Lecture 1.
  • Topics: Linear regression II. Course Whiteboard
  • Reading:
    • Chapter 7.1.0-7.1.4 of SC.

Week 2

• Jan 17, 2022. Lecture 2.
  • Topics: Examples of Linear Regression. Course Whiteboard
  • Reading:
    • Chapter 7.1.5 of SC.
    • Singular Value Decomposition
    • Linear Regression Examples.
    • Python Demo of Linear Regression with Outliers
    • Recommended linear algebra books, see logistics page.
• Jan 19, 2022. Lecture 3.
  • Topics: Examples of Linear Regression II, Ridge Regression. Course Whiteboard
  • Reading:
    • Chapter 7.4 of SC.
    • Python demo for Ridge and LASSO
    • Ridge Regression:
      • Stanford CS 229 Note on Linear Algebra
      • Lecture Note on Ridge Regression
      • Theobald, C. M. (1974). Generalizations of mean square error applied to ridge regression. Journal of the Royal Statistical Society. Series B (Methodological), 36(1), 103-106.

Week 3

• Jan 24, 2022. Lecture 4.
  • Topics: Ridge and LASSO Regression II. Course Whiteboard
  • Reading:
    • LASSO Regression:
      • ECE / STAT 695 (Lecture 1)
      • Statistical Learning with Sparsity (Chapter 2)
      • Elements of Statistical Learning (Chapter 3.4)
• Jan 26, 2022. Lecture 5.
  • Topics: Optimization I. Course Whiteboard
  • Reading:
    • Tutorial on Optimization.
    • CVX demo from the class
    • Extra CVX demo.
    • Unconstrained Optimality Conditions:
      • Nocedal-Wright, Numerical Optimization. (Chapter 2.1)
      • Boyd-Vandenberghe, Convex Optimization. (Chapter 9.1)
    • Convexity
      • Nocedal-Wright, Numerical Optimization. (Chapter 1)
      • Boyd-Vandenberghe, Convex Optimization. (Chapter 2 and 3)
      • CMU, Convex Optimization (Lecture 2 and 4)
      • Stanford CS 229 (Tutorial)
      • UCSD ECE 273 (Tutorial)
    • Constrained Optimization
      • Nocedal-Wright, Numerical Optimization. (Chapter 12.1)

Week 4

• Jan 31, 2022. Lecture 6.
  • Topics: Optimization II. Course Whiteboard
  • Reading:
    • Tutorial on Optimization.
    • Python demo for gradient descent
    • Python demo for constrained optimization
    • Gradient Descent
      • Boyd-Vandenberghe, Convex Optimization. (Chapter 9.2-9.4)
      • Nocedal-Wright, Numerical Optimization. (Chapter 3.1-3.3)
      • Y. Nesterov, “Introductory lectures on convex optimization”, Chapter 2.
• Feb 2, 2022. Lecture 7.
  • Topics: Optimization III. Linear Separability Course Whiteboard
  • Reading:
    • Stochastic Gradient Descent
      • CMU 10.725 Lecture
      • Kleinberg et al. (2018) “When Does SGD Escape Local Minima”

Week 5

• Feb 7, 2022. Lecture 8.
  • Topics: Linear Separability II. Bayesian Decision Rule. Course Whiteboard
  • Reading:
    • Separating Hyperplane:
      • Duda, Hart and Stork’s Pattern Classification, Chapter 5.1 and 5.2.
      • Princeton ORFE-523, Lecture 5 on Separating hyperplane.
      • Cornell ORIE-6300, Lecture 6 on Separating hyperplane
      • Caltech, Lecture Note
    • High Dimensional Gaussian
      • Bishop, Pattern Recognition and Machine Learning, Chapter 2.3
      • Stanford CS 229 Tutorial on Gaussian
• Feb 9, 2022. Lecture 9.
  • Topics: Bayesian Decision Rule. Course Whiteboard
  • Reading:
    • Python demo for Bayesian Decision Rule Special Case
    • Python demo for Bayesian Decision Rule with Gaussian Likelihood
    • Bayesian Decision Rule
      • Bishop, Pattern Recognition and Machine Learning, Chapter 4.1
      • Duda, Hart and Stork’s Pattern Classification, Chapter 2.1, 2.2, 2.6
      • UCSD ECE 271A, Lecture 4 and 5

Week 6
Course Note: ROC Curves
Course Note: Maximum Likelihood Estimation

• Feb 14, 2022. Lecture 10.
  • Topics: Classification Error and ROC curves. Parameter Estimation I. Course Whiteboard
  • Suggested Reading:
    • Probability of Error:
      • Duda, Hart and Stork’s Pattern Classification, Chapter 2.7, 3.1.
      • Poor, Intro to Signal Estimation and Detection, Chapter 2.
    • ROC Curve
      • ECE645 Note.
    • Parameter Estimation.
      • Duda, Hart and Stork’s Pattern Classification, Chapter 3.2
      • Iowa State EE 527
      • Purdue ECE 645, Lecture 18-20
      • UCSD ECE 271A, Lecture 6
      • Univ. Orleans
• Feb 16, 2022. Lecture 11.
  • Topics: Parameter Estimation II. Course Whiteboard
  • Suggested Reading:
    • Same as above.

Week 7
Course Note: Maximum a Posteriori
Course Note: Logistic Regression

• Feb 21, 2022. Lecture 12.
  • Topics: Parameter Estimation III. Logistic Regression. Course Whiteboard
  • Python demo 1
  • Python demo 2
• Feb 23, 2022. Lecture 13.
  • Topics: Logistic Regression. Course Whiteboard
  • Python demo: why prior is useful for MAP?

Week 8
Course Note: Principal Component Analysis

• Feb 28, 2022. Lecture 14.
  • Topics: Logistic Regression. Feature Extraction. Course Whiteboard
  • Python demo: Eigenface
• Mar 2, 2022. Lecture 15.
  • Topics: Feature Extraction. Kernel Methods.
  • Python demo: Kernel

Week 9
Course Note: Kernel Methods

• Mar 7, 2022. Lecture 16.
  • Topics: Multi-layer Perceptron. Course Whiteboard
  • Python demo: Multi-Layer Perceptron
  • Python demo: Convolutional Neural Network
• Mar 9, 2022. Lecture 17. Course Whiteboard
  • Topics: Convolutional Neural Network. Course Whiteboard

Week 10

• Spring Break

Week 11
Course Note: Convolutional Neural Network

• Mar 21, 2022. Lecture 18.
  • Topics: Convolutional Neural Networks II. Course Whiteboard
  • Python demo: Neural Style Transfer
• Mar 23, 2022, Lecture 19
  • Topics: Convolutional Neural Networks III. Adversarial Attacks. Course Whiteboard
  • Optional Reading: GAN Tutorial

Week 12
Course Note: Learning Feasibility

• Mar 28, 2022. Lecture 20.
  • Topics: Probability Inequality. Course Whiteboard
• Mar 30, 2022. Lecture 21.
  • Topics: Is Learning Feasible? Course Whiteboard

Week 13
Course Note: Generalization Bound

• Apr 4, 2022. Lecture 22.
  • Topics: Generalization Bound and Growth Function. Course Whiteboard
• Apr 6, 2022. Lecture 23.
  • VC dimension. Course Whiteboard
  • Optional Reading: VC dimension of MLPs

Week 14
Course Note

• Apr 11, 2022. Lecture 24.
  • Sample and Model Complexity. Bias and Variance. Course Whiteboard
• Apr 13, 2022. Lecture 25.
  • Bias and Variance. Overfitting. Course Whiteboard
  • Python Demo: Bias and Variance

Week 15 Course Note

• Apr 18, 2022. Lecture 26.
  • Overfitting. Regularization. Course Whiteboard
  • Python Demo: Overfitting
  • Python Demo: Regularization
• Apr 20, 2022. Lecture 27.
  • Validation. Course Whiteboard

Week 16

• Apr 25, 2022. Lecture 28.
  • Validation II. Course Whiteboard
  • Slides: Ethics of AI
  • Python Demo: Validation
• Apr 27, 2022. Lecture 29.