Introduction

What is this module about ?

• Understanding how the type of outcome variable is linked to the type of task that we want to solve.
• Learning how to setup a good supervised machine learning pipeline.
• Understanding the role of the training, validation and test sets.
• Understanding how to evaluate the results of machine learning algorithms (also with statistical tests!)
• Presenting some classical classification and regression techniques.

Outcome variables

In addition to the independent variables \(X_j\), we now have a dependent (= outcome) variable \(Y\).

Line vector \(x_{i.}\) represents one individual in a p-dimensional space \(\rightarrow\) “variable space” or “feature space”. To that line vector is associated a target \(y_i\).

Outcome variables

Different tasks will have different types for \(Y\).

Outcome variables

Example: a generative large language model receives as input a list of “tokens” (i.e. vectors in an “embedding” space) and outputs the “most likely next token” from a list of possible tokens.

Is it…

1. A multiple regression task
   
2. A multivariate regression task
3. A binary classification task
4. A multiclass classification task

Outcome variables

Example: a generative image diffusion model receives as input a list of “tokens” and outputs a matrix of pixel values forming an image that “matches” the tokens (the “prompt”).

Is it…

1. A multiple regression task
   
2. A multivariate regression task
3. A binary classification task
4. A multiclass classification task

Classification vs Regression

Supervised learning problem

Given a set of examples \(\mathcal{D} = \{(\vec{X_i}, \vec{Y_i})\}\), find a relationship \(\vec{Y} = f(\vec{X}, \vec{\theta})\) with \(\vec{\theta}\) a vector of parameters of the model, such that \(\vec{Y_i} - f(\vec{X_i}, \vec{\theta})\) is minimized for all \(Y_i, X_i\).

Very often (but not always) the process of solving the problem will be: choose the general form of \(f(\vec{X}, \vec{\theta})\) (the “model”), and then use an optimization technique to find the “best” \(\theta\) (“learn” the best parameters).

Example: simple linear regression \(\rightarrow\) set \(f(x; a, b) = ax + b\); find \(a, b\) so that \(\frac{1}{n} \sqrt{\sum_{i=1}^n (y_i - ax_i - b)^2}\) is minimized on \(\mathcal{D}\).

Supervised learning problem

A “supervised learning” solution will therefore include…

• A model with parameters.
• An optimization method to determine the best parameters.

Two main “classes” of methods:

• Direct method (ex: least square errors by matrix inversion: \(Y = XA \Rightarrow A = (X^TX)^{-1}X^TY\))
• Iterative method (ex: gradient descent \(\rightarrow\) start with initial parameters \(A = A_0\), compute gradient of loss function \(L\), update \(A \leftarrow A + \nu \frac{\partial L}{\partial A}\) with learning rate \(\nu\), repeat until stopping criterion is met.)

The supervised learning pipeline

Training set and overfitting

A training set can be simply defined as a dataset used for “training” the model, i.e. to find the “best” \(\vec{\theta}\) such that the error made by the model is minimized…

…but the best \(\vec{\theta}\) for the training set are not what we are looking for. We want the best \(\vec{\theta}\) for any data that our model may need to process. In other words, we need the best \(\vec{\theta}\) for the population of all potential test cases.

Training set and overfitting

Extreme example: if our model \(f(\vec{x}, \vec{\theta})\) is simply a look-up table of our training set, so that \(f(\vec{x_i}) = \vec{\theta_i} = \vec{y_i} \forall x_i, y_i \in \mathcal{D}_{train}\).

This certainly minimizes the error on the training set, but our model is actually undefined for any previously unseen case!

Over- and under-fitting

Overfitting is a phenomenon in machine learning where the model is too accurate on the training data, to the point that it doesn’t generalize well to new data. It is often a sign that the model is too complex.

Underfitting is when the model is too generic and fails to capture the complexity of the data. It is often a sign that the model is not complex enough.

Training, validation, test

To avoid (or at least measure) overfitting: split the training set into a training and a validation set.

The training set is used to determine the best \(\vec{\theta}\) of the model. The validation set is used to determine the best of the pipeline: choice of model, optimization parameters, stopping criterion, etc. Better scores on the validation set are more likely to result in better scores on the population.

A separate test set is used to provide a final evaluation of the complete solution.

Training, validation, test

It is very important to have a clear strategy of how to split the data before we start training models.

Main rules:

• Have the training and test data (e.g. if you have multiple images per patient, split at least at the patient level… or at the hospital level if you have that information!)
• You need enough data in your test set to be able to make reasonable conclusions about the results. (How much data is enough? \(\rightarrow\) see sample size and power computations!)

Training, validation, test

• You need enough data in your test set to be able to make reasonable conclusions about the results. (How much data is enough? \(\rightarrow\) see sample size and power computations!)

…

• But you also need enough data in your training set to find the best (hyper-)parameters! \(\rightarrow\) always a trade-off.

Training, validation, test

For the training and validation set, we can make better use of the available data with cross-validation.

Image: Gufosawa for Wikipedia. “Test data” should be “validation data” using our terminology.

Training, validation, test

Typical procedure with cross-validation:

1. For each set of hyper-parameters that you want to test (e.g. model type, learning rate, etc.), perform k-fold cross-validation with the training set and compute a score \(s_i\) on each of the \(k\) iterations.
2. Determine which hyper-parameters lead to the best results: e.g. highest average score \(\frac{1}{k}\sum_{i=1}^k s_i\). You can use a statistical test such as Friedman (if you used the same folds!) to check if the hyper-parameters have a significant impact.
3. Re-train your model on the entire training set using the best hyper-parameters
4. Evaluate your model on the test set.

Training, validation, test

IMPORTANT

You can (ideally) only use your test set once. It is a final test. After that, you have gained some knowledge about how your model behaves on the test set.

Any alteration of the model, training procedure, etc. that you do after that is done with knowledge of the test set. It is therefore no longer a test set: it has become part of the training set.

, you often have to do some back-and-forth with the test set, or the re-use the same benchmarks in different experiments. But you should be aware that each usage makes it more likely that you are overfitting the test set.

Quick quizz

To train a malign/benign cancer classifier, you gather data in a large, multi-centric study involving five hospitals and a total of 1.268 patients, which are distributed like this:

How you would split your dataset to maximize the generalizability of your results?

Evaluation

General evaluation procedure

We evaluate the ability of a system or algorithm \(\mathcal{A}\) to perform a task (classification, regression…) \(\mathcal{T}\) on a population \(\mathcal{P}\), on which we measure variables \(X_i\). On a training and test set \(D_{train}\) and \(D_{test}\), we have pairs of measures \((X_i, Y_i)\) where \(Y_i\) is the desired output of the system.

The system will produce an output: \(\hat{Y}_{i} = \mathcal{A}(X_i)\).

To evaluate the system, we have an assessment metric \(\mathcal{M}\) so that we can compute a set of scores \(\{S_{i} = \mathcal{M}(\hat{Y}_{i}, Y_i)\}\) on all examples in \(D_{test}\).

General evaluation procedure

• That \(D_{test}\) is representative of \(\mathcal{P}\).
• That \(\mathcal{M}\) is representative of the ability of \(\mathcal{A}\) to perform \(\mathcal{T}\).
• That the annotations \(Y_i\) are correct.
• That \(\mathcal{A}\) was constructed/trained without knowledge of \(D_{test}\).

\(\Rightarrow\) None of these assumptions are practically verified… the discussion of how right or wrong they are put limitations on the conclusions of a study.

Evaluation metrics

For classification problems, the evaluation metric is typically based on the confusion matrix. For \(m\) categories:

(global): \(ACC = \frac{1}{N}\sum_{i=1}^{m}C_{ii}\) with \(N\) the total number of cases in the test set. \(0 \leq ACC \leq 1\)

Evaluation metrics

(per-class): \(SEN_{i} = \frac{C_{ii}}{\sum_{j=1}^mC_{ij}} \rightarrow\) ratio of “real” \(k=i\) that are correctly predicted. (= true positive rate, recall, hit rate, power…). \(0 \leq SEN_i \leq 1\)

(per-class): \(PRE_{i} = \frac{C_{ii}}{\sum_{j=1}^mC_{ji}} \rightarrow\) ratio of “predicted” \(k=i\) that are really correct. (= positive predictive value). \(0 \leq PRE_i \leq 1\)

(per-class): \(F1_i = 2 \frac{PRE_i \times SEN_i}{PRE_i + SEN_i}\). \(0 \leq F1_i \leq 1\)

Evaluation metrics

(“simple average”) (global): \(MF1 = \frac{1}{m}\sum_{i=1}^mF1_i\). \(0 \leq MF1 \leq 1\)

(“harmonic of averages”) (global): other version which is confusingly given the same name, first compute \(PRE = \frac{1}{m}\sum_{i=1}^mPRE_i\) and \(SEN = \frac{1}{m}\sum_{i=1}^mSEN_i \Rightarrow MF1 = 2 \frac{PRE \times SEN}{PRE + SEN}\). \(0 \leq MF1 \leq 1\)

: Matthews Correlation Coefficient (\(-1 \leq MCC \leq 1\)), Cohen’s kappa (\(-1 \leq \kappa \leq 1\)), Geometric Mean of \(SEN_i\) (\(0 \leq GM \leq 1\))…

Evaluation metrics

Segmentation problems in image analysis = per-pixel classification \(\rightarrow\) we generally compute the confusion matrix per image. In a binary segmentation problem, we would have:

= F1-score of the positive class = \(\frac{2TP}{2TP+FP+FN}\).

= Jaccard Index = \(\frac{TP}{TP+FP+FN}\).

Example

Evaluation metrics

Regression problems:

• Mean Absolute Error (MAE): \(\frac{1}{n}\sum_{i=1}^n|\hat{y}_i-y_i|\). (positive unbounded)
• Mean Squared Error (MSE): \(\frac{1}{n}\sum_{i=1}^n(\hat{y}_i-y_i)^2\). (positive unbounded)
• Root Mean Squared Error (RMSE): \(\sqrt{\frac{1}{n}\sum_{i=1}^n(\hat{y}_i-y_i)^2}\). (positive unbounded)
• \(R^2\) (coefficient of determination): \(1 - \frac{\sum_{i=1}^n(\hat{y}_i-y_i)^2}{\sum_{i=1}^n(y_i-\bar{y})^2}\)

Comparison of multiple algorithms

If we consider the results of each algorithm as a random variable, observed on a sample \(D_{test}\) of the population \(\mathcal{P}\) of all possible test cases…

Then we can test the null hypothesis that the algorithms have the same results in \(\mathcal{P}\) based on the observed results in \(D_{test}\).

\(\Rightarrow\) tests on paired samples.

\(\Rightarrow\) tests on paired samples. Which tests? It depends on the type of metric!

Example

Classification

Classification problem

Given a set of examples \(\mathcal{D} = \{(\vec{X_i}, \vec{Y_i})\}\), with \(\vec{Y_i}\) a vector of categorical variables, find a relationship \(\vec{Y} = f(\vec{X}, \vec{\theta})\) with \(\vec{\theta}\) a vector of parameters of the model, such that \(\mathcal{L}(\{\vec{Y_i}, f(\vec{X_i}, \vec{\theta})\})\) is minimized, with \(\mathcal{L}\) as loss function (or error function).

If \(\vec{Y_i}\) is a single binary variable \(Y_i \in [0, 1]\), we have a binary classification problem.

If \(\vec{Y_i}\) is a single nominal variable \(Y_i \in \{C_1, ..., C_m\}, m>2\), we have a multi-class classification problem.

If \(\vec{Y_i}\) has multiple variables \(\vec{Y_i} = (Y_{i0}, ... Y_{ik})\) (where \(Y_{ij}\) may be binary or not), we have a multi-label classification (in the binary case) or a multi-output classification (\(m>2\) classes) problem.

Classification problem

Classification problem

Classification problem

Classifiers

• Bayesian classifier (and Gaussian Naive Bayes)
• Linear (and Quadratic) Discriminant Analysis
• Logistic regression
• k-Nearest Neighbours
• Multiclass extensions

Bayesian classifier

A Bayesian classifier frames the problem as a minimization of the probability of classification error:

\(P_e = \sum_{i=1}^n P(f(X) = C_i, Y \neq C_i)\)

This is done by computing (based on the training set) \(P(C_k | X) \forall C_k\), the a posteriori class probabilities for \(X\), and selecting the \(C_k\) such that \(P(C_k | X)\) is maximum. The a posteriori class probabilities can be computed using Bayes Theorem:

\(P(C_k | X) = \frac{P(C_k) P(X | C_k)}{P(X)}\)

Bayesian classifier

\(P(C_k | X) = \frac{P(C_k) P(X | C_k)}{P(X)}\)

\(P(C_k)\) is the a priori probability of \(C_k\), which can be estimated as the % of cases in the training set that belong to that class.

\(P(X | C_k)\) is the class density (probability of observing \(X\) in \(C_k\)).

\(P(X)\) is the probability of observing \(X\) independently of the class: it is therefore the density of the point cloud in feature space.

Bayesian classifier

\(P(C_k | X) = \frac{P(C_k) P(X | C_k)}{P(X)}\)

Since \(P(X)\) is completely independent from the classification function, maximizing \(P(C_k | X)\) only requires maximizing \(P(C_k) P(X | C_k)\).

How do we practically compute \(P(C_k | X)\)?

• Separate estimation of \(P(C_k)\) and \(P(X | C_k)\) by making some assumptions about the “shape” of \(P(X | C_k)\) (ex: Gaussian Naive Bayes: estimate the \(\sigma_k\) and \(\mu_k\) of each class, whose distribution is assumed to be Gaussian; Linear Discriminant Analysis).

• Direct estimation of \(P(C_k | X)\) \(\rightarrow\) logistic regression, neural networks…

• Other (non-Bayesian) approaches: local (k-Nearest Neighbours), recursive partition (Decision Trees)

Bayesian classifier

Discriminant analysis

Similar idea to PCA, but instead of finding the directions of overall maximum variance in the point cloud, we search for the direction which maximizes the separation between the classes.

The directions can then be used for dimensionality reduction (like PCA), but discriminant analysis can also be formulated as a method of classification.

Linear Discriminant analysis

Using the zero-centered point cloud (as with PCA), the first discriminant factor is the direction such that inter-class variance is maximum.

Let \(B\) be the weighted variance-covariance matrix of the class centroids \(g_k\): \(B_{m \times m} = \frac{1}{n}\sum_{k=1}^mn_kg_kg_k^T\), and \(S\) be the variance-covariance matrix of the (zero-centered) point cloud…

The discriminant factors will be the eigenvectors of \(S^{-1}B\).

Note that there will be at most \(m-1\) non-zero eigenvalues. If we have two classes, we have one discriminant factor, which is the axis of projection that maximizes inter-class separability.

Linear Discriminant analysis

Quadratic Discriminant analysis

Logistic regression

Logistic regression frames the classification as a regression problem followed by a thresholding. Let’s look at the binary classification case, where the two available classes are \(\{0, 1\}\), and with a single variable \(x\):

Logistic regression

Regularization

Regularization is a common strategy to limit the risk of overfitting. A very common way to do regularization is to add a term to a function to optimize (“loss function”, “cost function”, etc.) that penalizes complexity in the model.

Typical strategies:

• L1-regularization: \(R(\beta) = \sum_j |\beta_j|\)
• L2-regularization: \(R(\beta) = \frac{1}{2} \sum_j \beta_j^2\)

Logistic regression

Logistic regression is a more direct method to find linear separation, and will tend to produce the best (linear) separation (in the Bayesian sense). LDA will work better if the classes distributions are Gaussian, have similar variance-covariance matrices, and we have a good estimation of those variance-covariance matrices (i.e. enough representative data!).

Optimization

Many classification algorithms, such as logistic regression, rely on solving an optimization problem.

These are usually solved using an iterative approach (similar to what we’ve seen with the K-Means clustering):

1. Set initial conditions \(\beta_{(0)}\)
2. Evaluate cost function \(L(\beta_{(t)}, p, x, y)\)
3. Compute update step towards local minimum/maximum \(\beta_{(t+1)} = \beta_{t} + \delta(L)\), where \(\delta(L)\) is an update function.
4. Repeat 2-3 until stopping criterion is met.

Optimization

3. Compute update step towards local minimum/maximum \(\beta_{(t+1)} = \beta_{t} + \delta(L)\).

Very often, this step is based on the gradient of \(L\): \(\nabla_\beta L(\beta; p, x, y)\). Gradient descent methods use an update rule of the form:

\[\beta_{(t+1)} = \beta_{(t)} - \eta \nabla_\beta L(\beta; p, x, y)\]

Image shamelessly stolen from 3Blue1Brown, “Gradient descent, how neural networks learn”: https://www.youtube.com/watch?v=IHZwWFHWa-w

k-Nearest Neighbours

Nearest neighbour algorithm \(\rightarrow\) very likely to overfit. Regularization strategy: use \(k\) nearest neighbours and take the majority vote.

k-Nearest Neighbours

Distance-based method \(\rightarrow\) requires to choose the distance metric! (cf unsupervised methods).

• Very easy to implement and understand.
• Non-linear boundaries.
• Adapts to local shape of the point cloud.
• If \(k > 1\), automatically ignores outliers.

But…

• Choice of \(k\) and distance metric non-trivial.
• May require PCA and/or standardization to avoid scaling issues.
• May be computationally expensive if large number of points.

Multiclass extensions

Slightly confusing terminology:

• “Simple” multiclass: 1 target variable, \(>2\) categories.
• Multilabel \(\approx\) “multiclass detection”: \(>1\) target binary variables (i.e. several non-exclusive categories)
• Multiclass-multioutput: \(>1\) target variables with \(>2\) categories.

Multiclass extensions

For multiple target variables, this is equivalent to fitting multiple independent models.

For multiple categories (\(m>2\)), several strategies:

• One-vs-All (or One-vs-Rest): train \(m\) classifiers (1 per class), each time as a binary classifier between the target class and all others. Prediction: class with highest predicted probability.
• One-vs-One: train \(\frac{m(m-1)}{2}\) classifiers (1 per pair of classes). Prediction: class with the most votes.

Example: using the Iris dataset \(\rightarrow\) 3 different species of iris flowers (Setosa, Versicolor, Virginica), 4 features (Sepal/Petal length/width), here reduced to 2 with PCA.

Example with QDA

Example with QDA

Example with LDA

Example with LDA

Classification summary

• Classification = finding a decision hyper-surface to separate the points per-category in feature space.
• Simplest methods \(\rightarrow\) either find linear separations (i.e. hyper-planes) OR use local neighbourhood (no explicit model at the end).
• Recognize overfitting using (cross-)validation. Use regularization techniques to reduce it (e.g. regularization term in optimization function, number of neighbours in kNN…).
• Increasing complexity of the model to quadratic models… or Decision Trees, SVMs, Neural Networks, etc. (see INFO-H501 course) \(\rightarrow\) can find more complex shapes… at a higher risk of overfitting.
• Best model is generally the simplest model that works.
• “Fitting the model”: direct methods (exact or approximations) vs optimization-based methods (iterative, always approximative).

Regression

Regression problem

Given a set of examples \(\mathcal{D} = \{(\vec{X_i}, \vec{Y_i})\}\), with \(\vec{Y_i}\) a vector of numerical variables, find a relationship \(\vec{Y} = f(\vec{X}, \vec{\theta})\) with \(\vec{\theta}\) a vector of parameters of the model, such that \(\mathcal{L}(\{\vec{Y_i}, f(\vec{X_i}, \vec{\theta})\})\) is minimized, with \(\mathcal{L}\) as loss function (or error function).

If \(\vec{Y_i}\) is a single numerical variable \(Y_i\) and \(\vec{X_i}\) is a single numerical variable \(X_i\), we have a simple regression problem.

If \(\vec{Y_i}\) is a single numerical variable \(Y_i\) and \(\vec{X_i}\) has multiple variables, we have a multiple regression problem.

If \(\vec{Y_i}\) has multiple variables \(\vec{Y_i} = (Y_{i0}, ... Y_{ik})\), we have a multivariate linear regression problem.

Regression problem

Often represented with one axis as the target, but that means we can only look at one independent variable at a time (if we look in 2D) \(\rightarrow\) other visualizations possible.

Linear regression

In a linear regression (simple or multiple) problem, we are trying to fit a linear model through the point cloud:

\(\hat{Y_i} = \theta_0 + \sum_{j=1}^p \theta_j X_{ij}\)

If there are no exact solutions, we will in general have: \(\hat{Y_i} = \theta_0 + \sum_{j=1}^p \theta_j X_{ij} + \epsilon_i\), with \(\epsilon_i\) the residual error of the model for the \(i\)-th case.

In matrix form, we have \(\hat{Y} = X\theta + \epsilon\)

The Ordinary Least Squares method consists in optimizing \(\theta\) so that \(||X\theta - Y||^2\) is minimized on the dataset.

Linear regression

The Ordinary Least Squares method consists in optimizing \(\theta\) so that \(||X\theta - Y||^2\) is minimized on the dataset.

This minimization problem has an exact solution if \(X^TX\) is invertible:

\[\theta = (X^TX)^{-1}X^TY\]

with \(\theta = \begin{pmatrix}\theta_0 \\ ... \\ \theta_p\end{pmatrix}, X = \begin{pmatrix}1 & X_{11} & ... & X_{1p} \\ 1 & ... & ... & X_{2p}\\ 1 & ... & ... & ... \\ 1 & X_{n1} & ... & X_{np}\end{pmatrix}, Y = \begin{pmatrix}Y_1 \\ ... \\ Y_n\end{pmatrix}\)

def ordinary_least_squares(X: np.ndarray, Y: np.ndarray):
    X_ = np.ones((X.shape[0], X.shape[1]+1)) # add column
    X_[:, 1:] = X
    return np.linalg.inv(X_.T @ X_) @ X_.T @ Y

Linear regression

Example: diabetes dataset, after standardization + PCA (computed on training set), taking first component vs target.

Linear regression

This minimization problem has an exact solution if \(X^TX\) is invertible:

Conditions:

• \(n \leq p\) (more points than dimensions)
• \(X_j\) variables are linearly independent \(\rightarrow\) \(r_{ij}^2 << 1 \forall i \neq j\)

Otherwise:

• PCA \(\rightarrow\) use \(q\) first components \(\rightarrow\) uncorrelated and lower dimension.

Linear regression

• Very sensitive to outliers. \(\rightarrow\) we can minimize \(\sum|\epsilon_i|\) instead of \(\sum \epsilon_i^2\), and/or use regularization.

Example: Ridge regression \(\rightarrow\) uses ordinary least squares with L2 regularization \(\rightarrow\) minimizes \(||X\theta - Y||^2 + \alpha ||\theta||^2\)

Ridge Regression

Ridge Classification

Similarly to logistic regression, we can use the Ridge regressor as a classificator:

• For binary classification, convert from “classes \(\{0,1\}\)” to “values \(\{-1, 1\}\)”. Train regressor on these values. Predicted class is then the sign of the regressor’s predicted value \(\rightarrow\) \(Y_i = sgn(X_i\theta)\).
• For multiclass classification, train each possible class separately as above. Predicted class is then the class with the highest predicted value \(\rightarrow\) \(Y_i = argmax_j(X_i\theta_j)\).

Non-linear regression

Linear regression is very easy to compute and very easy to interpret, but we are obviously limited in the kind of functions that we can model. There are several options to introduce non-linearity.

Option 1: make strong assumptions about the shape of the data, and use optimization method to fit parametric model.

Non-linear regression

poly = PolynomialFeatures(degree=5)
Xt = poly.fit_transform(X[:, np.newaxis])
reg = Ridge(alpha=1.)
reg.fit(Xt, Y)    

Non-linear regression (or classification)

Option 2: find a transformation into a new feature space where linear regression (or classification) becomes possible.

This will often be a higher dimensionality feature space \(\rightarrow\) instead of the dimensionality reduction that we’ve done so far, we need to do dimensionality expansion!

Kernel trick

Option 2: find a transformation into a new feature space where linear regression (or classification) becomes possible.

Example with classification:

Transformation: \((x_1, x_2) \rightarrow (x_1, x_2, x_1^2-0.3 + x_2^2-0.6)\)

Kernel trick

In general: we can’t really know what the exact transformation is… but we don’t need to, thanks to the kernel trick.

Let \(F(x)\) be our transformation function, \(K(x, x') = F(x)F(x')^T\) is the corresponding Kernel function. For most linear models, the training (and prediction) process can be expressed in terms of \(K(x, x')\)! That means that we don’t actually need to formulate \(F(x)\), as long as we can formulate \(K(x, x')\).

Kernel trick

Example: polynomial kernel with degree 2. If \(x = (x_1, x_2)\) is two-dimensional, we have \(F(x) = (1, x_1, x_2, x_1^2, x_2^2, x_1x_2)\), a projection into a 6-dimensional space.

Then, we have \(K(x, x') = F(x)F(x')^T = (1 + x^Tx')^2\)

reg = KernelRidge(alpha=1., 
                  kernel='poly')
reg.fit(X, Y)

Ensemble learning

A common technique to get complex decision functions with less risk of overfitting is ensemble learning.

Instead of training one very complex model on the whole training set…

… train several smaller models on parts of the dataset, hiding some data points and/or some features

For prediction: majority voting (classification) or averaging (regression).

Towards deep learning

The core elements in machine learning

Through the different supervised and unsupervised techniques, we’ve seen the core elements of machine learning solutions:

• Looking at datasets as point clouds in feature space.
• Finding transformations of the feature space that make our task easier to solve (dimensionality reduction, removing covariances, increasing separability, dimensionality expansion).
• Defining a model of the relationship between our inputs and our target outputs.
• Defining an error function.
• Solving an optimization problem to find the best parameters of our model in order to minimize the error function on our training set.
• Evaluating our results based on an evaluation metric and an independent test set.

The core elements in machine learning

Example:

Given a training dataset \(\mathcal{D}_{train} = \{(x_i, y_i)\}\) and a test dataset \(\mathcal{D}_{test} = \{(x_j, y_j)\}\):

• Use PCA on \(x_i\) to find a transformation \(x_t = PCA(x)\).
• Define a polynomial model \(f(x_t) = \sum_k \theta_k x^k\)
• Find \(\beta_k\) so that \(\sum_i (-ylog(f(x_t)) - log(y)f(x_t))\) is minimized \(\forall (x_i, y_i)\), using gradient descent.
• Evaluate the results with the F1-score computed on \((y_j, f(PCA(x_j), \beta))\)

The core elements in deep learning

A deep learning model will perform the feature extraction/transformation and the model optimization steps all at the same time. This is generally done using large neural networks.

Example: AlexNet, from Krizhevsky et al., 2012.

The core elements in deep learning

Deep learning models = succession of layers which often perform “almost-linear” transformations of the feature space.

Example: “dense layer” \(\rightarrow f(x, \theta) = ReLU(\sum_i x_i \theta_i)\), where \(ReLU\) is the Rectified Linear Unit \(ReLU(x) = x \text{ if } x>0, 0 \text{ otherwise}\).

Successive linear transformations + small nonlinearities = completely non-linear relationship between the network inputs and outputs.

The core elements in deep learning

Layers can perform dimensionality reduction or dimensionality expansion, depending on their role in the network… or keep the dimensionality but change the semantics of the dimensions.

Example: convolutional layers in image networks, transforming for instance a \(128 \times 128 \times 3\) dimensional image input (dimensions = mostly independent localization) into a \(16 \times 16 \times 192\) dimensional feature space (dimensions = mostly independent characteristics).

Deep learning is machine learning is data analysis

• Deep neural networks often converge better if the inputs are standardized.
• Deep neural networks require regularization: in the loss function, but also through “dropout” layers, data augmentation, ensemble learning…

Example: GPT evolution

• Unsupervised learning \(\rightarrow\) embedding of the words into “token” space.
• Wearkly supervised learning \(rightarrow\) next-token prediction (or fill-in-the-blank) on all publicly avaible text.
• Reinforcement Learning from Human Feedback \(\rightarrow\) higher quality task-specific supervision.
• GPT 4 \(\rightarrow\) ensemble learning with several GPT-3.5-like models under the hood.

Conclusions

Module 1

• You can’t do data analysis without understanding the data.
• Data analysis starts from the acquisition: a biased or ill-conceived dataset will lead to a biased or ill-conceived analysis and/or predictive model.
• Data visualization tells a story \(\rightarrow\) choosing the right story to tell is an important choice in understanding the data.

Module 2

• Hypothesis testing is about answering: “is this observation compatible with this hypothesis?”
• Hypothesis testing never proves (or disproves) the hypothesis.
• Hypothesis testing can reject the (null) hypothesis… with some level of confidence.
• Level of significance \(\rightarrow\) acceptable probability of Type I error (incorrectly rejecting \(H_0\))
• Power \(\rightarrow\) probability of correctly rejecting \(H_0\) \(\rightarrow\) depends on the effect size and the sample size.
• Common univariate hypothesis tests: t-test on the mean, Wilcoxon signed-rank test on the median, chi-square test on the variance, Kolmogorov-Smirnov goodness-of-fit, Shapiro-Wilk normality…

Module 3

• The big decision tree of hypothesis tests:
  • What are we trying to compare? (same variables in different groups? different variables in same group? same variables in same group in different conditions?)
  • What is the type of the variables (numerical, ordinal, nominal)?
  • Are the application conditions for parametric tests met?
• Tests on unpaired samples: independent t-test / Mann-Whitney \(\rightarrow\) ANOVA / Kruskal-Wallis, \(\chi^2\) independence test…
• Tests on paired samples: paired t-test / Wilcoxon \(\rightarrow\) ANOVA / Friedman, McNemar, Cochran’s Q test…

Module 4

• Multivariate data = point clouds in feature space.
• Unsupervised learning = understand the structure of the data.
• Transformations of feature space \(\rightarrow\) PCA.
• Dimensionality reduction \(\rightarrow\) PCA, MDS, t-SNE, autoencoders…
• Clustering \(\rightarrow\) hierarchical, K-Means…
• Importance of distance metrics.

Module 5

• Classification = finding decision hypersurfaces in feature space.
• Class separability: linear separability \(\rightarrow\) linear classifiers. Nonlinear separability \(\rightarrow\) more complex functions… or transformation of feature space.
• Regression = fitting an hypersurface through a point cloud.
• Importance of a good supervised learning pipeline: independent training / test sets, cross-validation, evaluation metrics…
• Comparison of models \(\rightarrow\) statistical tests of significance.