{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "1 Introduction 1\n", "1.1 Example: Polynomial Curve Fitting . . . . . . . . . . . . . . . . . 4\n", "1.2 Probability Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 12\n", "1.2.1 Probability densities . . . . . . . . . . . . . . . . . . . . . 17\n", "1.2.2 Expectations and covariances . . . . . . . . . . . . . . . . 19\n", "1.2.3 Bayesian probabilities . . . . . . . . . . . . . . . . . . . . 21\n", "1.2.4 The Gaussian distribution . . . . . . . . . . . . . . . . . . 24\n", "1.2.5 Curve fitting re-visited . . . . . . . . . . . . . . . . . . . . 28\n", "1.2.6 Bayesian curve fitting . . . . . . . . . . . . . . . . . . . . 30\n", "1.3 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . 32\n", "1.4 The Curse of Dimensionality . . . . . . . . . . . . . . . . . . . . . 33\n", "1.5 Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 38\n", "1.5.1 Minimizing the misclassification rate . . . . . . . . . . . . 39\n", "1.5.2 Minimizing the expected loss . . . . . . . . . . . . . . . . 41\n", "1.5.3 The reject option . . . . . . . . . . . . . . . . . . . . . . . 42\n", "1.5.4 Inference and decision . . . . . . . . . . . . . . . . . . . . 42\n", "1.5.5 Loss functions for regression . . . . . . . . . . . . . . . . . 46\n", "1.6 Information Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 48\n", "1.6.1 Relative entropy and mutual information . . . . . . . . . . 55\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58\n", "2 Probability Distributions 67\n", "2.1 Binary Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 68\n", "2.1.1 The beta distribution . . . . . . . . . . . . . . . . . . . . . 71\n", "2.2 Multinomial Variables . . . . . . . . . . . . . . . . . . . . . . . . 74\n", "2.2.1 The Dirichlet distribution . . . . . . . . . . . . . . . . . . . 76\n", "2.3 The Gaussian Distribution . . . . . . . . . . . . . . . . . . . . . . 78\n", "2.3.1 Conditional Gaussian distributions . . . . . . . . . . . . . . 85\n", "2.3.2 Marginal Gaussian distributions . . . . . . . . . . . . . . . 88\n", "2.3.3 Bayes’ theorem for Gaussian variables . . . . . . . . . . . . 90\n", "2.3.4 Maximum likelihood for the Gaussian . . . . . . . . . . . . 93\n", "2.3.5 Sequential estimation . . . . . . . . . . . . . . . . . . . . . 94\n", "2.3.6 Bayesian inference for the Gaussian . . . . . . . . . . . . . 97\n", "2.3.7 Student’s t-distribution . . . . . . . . . . . . . . . . . . . . 102\n", "2.3.8 Periodic variables . . . . . . . . . . . . . . . . . . . . . . . 105\n", "2.3.9 Mixtures of Gaussians . . . . . . . . . . . . . . . . . . . . 110\n", "2.4 The Exponential Family . . . . . . . . . . . . . . . . . . . . . . . 113\n", "2.4.1 Maximum likelihood and sufficient statistics . . . . . . . . 116\n", "2.4.2 Conjugate priors . . . . . . . . . . . . . . . . . . . . . . . 117\n", "2.4.3 Noninformative priors . . . . . . . . . . . . . . . . . . . . 117\n", "2.5 Nonparametric Methods . . . . . . . . . . . . . . . . . . . . . . . 120\n", "2.5.1 Kernel density estimators . . . . . . . . . . . . . . . . . . . 122\n", "2.5.2 Nearest-neighbour methods . . . . . . . . . . . . . . . . . 124\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127\n", "3 Linear Models for Regression 137\n", "3.1 Linear Basis Function Models . . . . . . . . . . . . . . . . . . . . 138\n", "3.1.1 Maximum likelihood and least squares . . . . . . . . . . . . 140\n", "3.1.2 Geometry of least squares . . . . . . . . . . . . . . . . . . 143\n", "3.1.3 Sequential learning . . . . . . . . . . . . . . . . . . . . . . 143\n", "3.1.4 Regularized least squares . . . . . . . . . . . . . . . . . . . 144\n", "3.1.5 Multiple outputs . . . . . . . . . . . . . . . . . . . . . . . 146\n", "3.2 The Bias-Variance Decomposition . . . . . . . . . . . . . . . . . . 147\n", "3.3 Bayesian Linear Regression . . . . . . . . . . . . . . . . . . . . . 152\n", "3.3.1 Parameter distribution . . . . . . . . . . . . . . . . . . . . 153\n", "3.3.2 Predictive distribution . . . . . . . . . . . . . . . . . . . . 156\n", "3.3.3 Equivalent kernel . . . . . . . . . . . . . . . . . . . . . . . 157\n", "3.4 Bayesian Model Comparison . . . . . . . . . . . . . . . . . . . . . 161\n", "3.5 The Evidence Approximation . . . . . . . . . . . . . . . . . . . . 165\n", "3.5.1 Evaluation of the evidence function . . . . . . . . . . . . . 166\n", "3.5.2 Maximizing the evidence function . . . . . . . . . . . . . . 168\n", "3.5.3 Effective number of parameters . . . . . . . . . . . . . . . 170\n", "3.6 Limitations of Fixed Basis Functions . . . . . . . . . . . . . . . . 172\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173\n", "4 Linear Models for Classification 179\n", "4.1 Discriminant Functions . . . . . . . . . . . . . . . . . . . . . . . . 181\n", "4.1.1 Two classes . . . . . . . . . . . . . . . . . . . . . . . . . . 181\n", "4.1.2 Multiple classes . . . . . . . . . . . . . . . . . . . . . . . . 182\n", "4.1.3 Least squares for classification . . . . . . . . . . . . . . . . 184\n", "4.1.4 Fisher’s linear discriminant . . . . . . . . . . . . . . . . . . 186\n", "4.1.5 Relation to least squares . . . . . . . . . . . . . . . . . . . 189\n", "4.1.6 Fisher’s discriminant for multiple classes . . . . . . . . . . 191\n", "4.1.7 The perceptron algorithm . . . . . . . . . . . . . . . . . . . 192\n", "4.2 Probabilistic Generative Models . . . . . . . . . . . . . . . . . . . 196\n", "4.2.1 Continuous inputs . . . . . . . . . . . . . . . . . . . . . . 198\n", "4.2.2 Maximum likelihood solution . . . . . . . . . . . . . . . . 200\n", "4.2.3 Discrete features . . . . . . . . . . . . . . . . . . . . . . . 202\n", "4.2.4 Exponential family . . . . . . . . . . . . . . . . . . . . . . 202\n", "4.3 Probabilistic Discriminative Models . . . . . . . . . . . . . . . . . 203\n", "4.3.1 Fixed basis functions . . . . . . . . . . . . . . . . . . . . . 204\n", "4.3.2 Logistic regression . . . . . . . . . . . . . . . . . . . . . . 205\n", "4.3.3 Iterative reweighted least squares . . . . . . . . . . . . . . 207\n", "4.3.4 Multiclass logistic regression . . . . . . . . . . . . . . . . . 209\n", "4.3.5 Probit regression . . . . . . . . . . . . . . . . . . . . . . . 210\n", "4.3.6 Canonical link functions . . . . . . . . . . . . . . . . . . . 212\n", "4.4 The Laplace Approximation . . . . . . . . . . . . . . . . . . . . . 213\n", "4.4.1 Model comparison and BIC . . . . . . . . . . . . . . . . . 216\n", "4.5 Bayesian Logistic Regression . . . . . . . . . . . . . . . . . . . . 217\n", "4.5.1 Laplace approximation . . . . . . . . . . . . . . . . . . . . 217\n", "4.5.2 Predictive distribution . . . . . . . . . . . . . . . . . . . . 218\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220\n", "5 Neural Networks 225\n", "5.1 Feed-forward Network Functions . . . . . . . . . . . . . . . . . . 227\n", "5.1.1 Weight-space symmetries . . . . . . . . . . . . . . . . . . 231\n", "5.2 Network Training . . . . . . . . . . . . . . . . . . . . . . . . . . . 232\n", "5.2.1 Parameter optimization . . . . . . . . . . . . . . . . . . . . 236\n", "5.2.2 Local quadratic approximation . . . . . . . . . . . . . . . . 237\n", "5.2.3 Use of gradient information . . . . . . . . . . . . . . . . . 239\n", "5.2.4 Gradient descent optimization . . . . . . . . . . . . . . . . 240\n", "5.3 Error Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . 241\n", "5.3.1 Evaluation of error-function derivatives . . . . . . . . . . . 242\n", "5.3.2 A simple example . . . . . . . . . . . . . . . . . . . . . . 245\n", "5.3.3 Efficiency of backpropagation . . . . . . . . . . . . . . . . 246\n", "5.3.4 The Jacobian matrix . . . . . . . . . . . . . . . . . . . . . 247\n", "5.4 The Hessian Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . 249\n", "5.4.1 Diagonal approximation . . . . . . . . . . . . . . . . . . . 250\n", "5.4.2 Outer product approximation . . . . . . . . . . . . . . . . . 251\n", "5.4.3 Inverse Hessian . . . . . . . . . . . . . . . . . . . . . . . . 252\n", "5.4.4 Finite differences . . . . . . . . . . . . . . . . . . . . . . . 252\n", "5.4.5 Exact evaluation of the Hessian . . . . . . . . . . . . . . . 253\n", "5.4.6 Fast multiplication by the Hessian . . . . . . . . . . . . . . 254\n", "5.5 Regularization in Neural Networks . . . . . . . . . . . . . . . . . 256\n", "5.5.1 Consistent Gaussian priors . . . . . . . . . . . . . . . . . . 257\n", "5.5.2 Early stopping . . . . . . . . . . . . . . . . . . . . . . . . 259\n", "5.5.3 Invariances . . . . . . . . . . . . . . . . . . . . . . . . . . 261\n", "5.5.4 Tangent propagation . . . . . . . . . . . . . . . . . . . . . 263\n", "5.5.5 Training with transformed data . . . . . . . . . . . . . . . . 265\n", "5.5.6 Convolutional networks . . . . . . . . . . . . . . . . . . . 267\n", "5.5.7 Soft weight sharing . . . . . . . . . . . . . . . . . . . . . . 269\n", "5.6 Mixture Density Networks . . . . . . . . . . . . . . . . . . . . . . 272\n", "5.7 Bayesian Neural Networks . . . . . . . . . . . . . . . . . . . . . . 277\n", "5.7.1 Posterior parameter distribution . . . . . . . . . . . . . . . 278\n", "5.7.2 Hyperparameter optimization . . . . . . . . . . . . . . . . 280\n", "5.7.3 Bayesian neural networks for classification . . . . . . . . . 281\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284\n", "6 Kernel Methods 291\n", "6.1 Dual Representations . . . . . . . . . . . . . . . . . . . . . . . . . 293\n", "6.2 Constructing Kernels . . . . . . . . . . . . . . . . . . . . . . . . . 294\n", "6.3 Radial Basis Function Networks . . . . . . . . . . . . . . . . . . . 299\n", "6.3.1 Nadaraya-Watson model . . . . . . . . . . . . . . . . . . . 301\n", "6.4 Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 303\n", "6.4.1 Linear regression revisited . . . . . . . . . . . . . . . . . . 304\n", "6.4.2 Gaussian processes for regression . . . . . . . . . . . . . . 306\n", "6.4.3 Learning the hyperparameters . . . . . . . . . . . . . . . . 311\n", "6.4.4 Automatic relevance determination . . . . . . . . . . . . . 312\n", "6.4.5 Gaussian processes for classification . . . . . . . . . . . . . 313\n", "6.4.6 Laplace approximation . . . . . . . . . . . . . . . . . . . . 315\n", "6.4.7 Connection to neural networks . . . . . . . . . . . . . . . . 319\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320\n", "7 Sparse Kernel Machines 325\n", "7.1 Maximum Margin Classifiers . . . . . . . . . . . . . . . . . . . . 326\n", "7.1.1 Overlapping class distributions . . . . . . . . . . . . . . . . 331\n", "7.1.2 Relation to logistic regression . . . . . . . . . . . . . . . . 336\n", "7.1.3 Multiclass SVMs . . . . . . . . . . . . . . . . . . . . . . . 338\n", "7.1.4 SVMs for regression . . . . . . . . . . . . . . . . . . . . . 339\n", "7.1.5 Computational learning theory . . . . . . . . . . . . . . . . 344\n", "7.2 Relevance Vector Machines . . . . . . . . . . . . . . . . . . . . . 345\n", "7.2.1 RVM for regression . . . . . . . . . . . . . . . . . . . . . . 345\n", "7.2.2 Analysis of sparsity . . . . . . . . . . . . . . . . . . . . . . 349\n", "7.2.3 RVM for classification . . . . . . . . . . . . . . . . . . . . 353\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357\n", "8 Graphical Models 359\n", "8.1 Bayesian Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 360\n", "8.1.1 Example: Polynomial regression . . . . . . . . . . . . . . . 362\n", "8.1.2 Generative models . . . . . . . . . . . . . . . . . . . . . . 365\n", "8.1.3 Discrete variables . . . . . . . . . . . . . . . . . . . . . . . 366\n", "8.1.4 Linear-Gaussian models . . . . . . . . . . . . . . . . . . . 370\n", "8.2 Conditional Independence . . . . . . . . . . . . . . . . . . . . . . 372\n", "8.2.1 Three example graphs . . . . . . . . . . . . . . . . . . . . 373\n", "8.2.2 D-separation . . . . . . . . . . . . . . . . . . . . . . . . . 378\n", "8.3 Markov Random Fields . . . . . . . . . . . . . . . . . . . . . . . 383\n", "8.3.1 Conditional independence properties . . . . . . . . . . . . . 383\n", "8.3.2 Factorization properties . . . . . . . . . . . . . . . . . . . 384\n", "8.3.3 Illustration: Image de-noising . . . . . . . . . . . . . . . . 387\n", "8.3.4 Relation to directed graphs . . . . . . . . . . . . . . . . . . 390\n", "8.4 Inference in Graphical Models . . . . . . . . . . . . . . . . . . . . 393\n", "8.4.1 Inference on a chain . . . . . . . . . . . . . . . . . . . . . 394\n", "8.4.2 Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398\n", "8.4.3 Factor graphs . . . . . . . . . . . . . . . . . . . . . . . . . 399\n", "8.4.4 The sum-product algorithm . . . . . . . . . . . . . . . . . . 402\n", "8.4.5 The max-sum algorithm . . . . . . . . . . . . . . . . . . . 411\n", "8.4.6 Exact inference in general graphs . . . . . . . . . . . . . . 416\n", "8.4.7 Loopy belief propagation . . . . . . . . . . . . . . . . . . . 417\n", "8.4.8 Learning the graph structure . . . . . . . . . . . . . . . . . 418\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418\n", "9 Mixture Models and EM 423\n", "9.1 K-means Clustering . . . . . . . . . . . . . . . . . . . . . . . . . 424\n", "9.1.1 Image segmentation and compression . . . . . . . . . . . . 428\n", "9.2 Mixtures of Gaussians . . . . . . . . . . . . . . . . . . . . . . . . 430\n", "9.2.1 Maximum likelihood . . . . . . . . . . . . . . . . . . . . . 432\n", "9.2.2 EM for Gaussian mixtures . . . . . . . . . . . . . . . . . . 435\n", "9.3 An Alternative View of EM . . . . . . . . . . . . . . . . . . . . . 439\n", "9.3.1 Gaussian mixtures revisited . . . . . . . . . . . . . . . . . 441\n", "9.3.2 Relation to K-means . . . . . . . . . . . . . . . . . . . . . 443\n", "9.3.3 Mixtures of Bernoulli distributions . . . . . . . . . . . . . . 444\n", "9.3.4 EM for Bayesian linear regression . . . . . . . . . . . . . . 448\n", "9.4 The EM Algorithm in General . . . . . . . . . . . . . . . . . . . . 450\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455\n", "10 Approximate Inference 461\n", "10.1 Variational Inference . . . . . . . . . . . . . . . . . . . . . . . . . 462\n", "10.1.1 Factorized distributions . . . . . . . . . . . . . . . . . . . . 464\n", "10.1.2 Properties of factorized approximations . . . . . . . . . . . 466\n", "10.1.3 Example: The univariate Gaussian . . . . . . . . . . . . . . 470\n", "10.1.4 Model comparison . . . . . . . . . . . . . . . . . . . . . . 473\n", "10.2 Illustration: Variational Mixture of Gaussians . . . . . . . . . . . . 474\n", "10.2.1 Variational distribution . . . . . . . . . . . . . . . . . . . . 475\n", "10.2.2 Variational lower bound . . . . . . . . . . . . . . . . . . . 481\n", "10.2.3 Predictive density . . . . . . . . . . . . . . . . . . . . . . . 482\n", "10.2.4 Determining the number of components . . . . . . . . . . . 483\n", "10.2.5 Induced factorizations . . . . . . . . . . . . . . . . . . . . 485\n", "10.3 Variational Linear Regression . . . . . . . . . . . . . . . . . . . . 486\n", "10.3.1 Variational distribution . . . . . . . . . . . . . . . . . . . . 486\n", "10.3.2 Predictive distribution . . . . . . . . . . . . . . . . . . . . 488\n", "10.3.3 Lower bound . . . . . . . . . . . . . . . . . . . . . . . . . 489\n", "10.4 Exponential Family Distributions . . . . . . . . . . . . . . . . . . 490\n", "10.4.1 Variational message passing . . . . . . . . . . . . . . . . . 491\n", "10.5 Local Variational Methods . . . . . . . . . . . . . . . . . . . . . . 493\n", "10.6 Variational Logistic Regression . . . . . . . . . . . . . . . . . . . 498\n", "10.6.1 Variational posterior distribution . . . . . . . . . . . . . . . 498\n", "10.6.2 Optimizing the variational parameters . . . . . . . . . . . . 500\n", "10.6.3 Inference of hyperparameters . . . . . . . . . . . . . . . . 502\n", "10.7 Expectation Propagation . . . . . . . . . . . . . . . . . . . . . . . 505\n", "10.7.1 Example: The clutter problem . . . . . . . . . . . . . . . . 511\n", "10.7.2 Expectation propagation on graphs . . . . . . . . . . . . . . 513\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517\n", "11 Sampling Methods 523\n", "11.1 Basic Sampling Algorithms . . . . . . . . . . . . . . . . . . . . . 526\n", "11.1.1 Standard distributions . . . . . . . . . . . . . . . . . . . . 526\n", "11.1.2 Rejection sampling . . . . . . . . . . . . . . . . . . . . . . 528\n", "11.1.3 Adaptive rejection sampling . . . . . . . . . . . . . . . . . 530\n", "11.1.4 Importance sampling . . . . . . . . . . . . . . . . . . . . . 532\n", "11.1.5 Sampling-importance-resampling . . . . . . . . . . . . . . 534\n", "11.1.6 Sampling and the EM algorithm . . . . . . . . . . . . . . . 536\n", "11.2 Markov Chain Monte Carlo . . . . . . . . . . . . . . . . . . . . . 537\n", "11.2.1 Markov chains . . . . . . . . . . . . . . . . . . . . . . . . 539\n", "11.2.2 The Metropolis-Hastings algorithm . . . . . . . . . . . . . 541\n", "11.3 Gibbs Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . 542\n", "11.4 Slice Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546\n", "11.5 The Hybrid Monte Carlo Algorithm . . . . . . . . . . . . . . . . . 548\n", "11.5.1 Dynamical systems . . . . . . . . . . . . . . . . . . . . . . 548\n", "11.5.2 Hybrid Monte Carlo . . . . . . . . . . . . . . . . . . . . . 552\n", "11.6 Estimating the Partition Function . . . . . . . . . . . . . . . . . . 554\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556\n", "12 Continuous Latent Variables 559\n", "12.1 Principal Component Analysis . . . . . . . . . . . . . . . . . . . . 561\n", "12.1.1 Maximum variance formulation . . . . . . . . . . . . . . . 561\n", "12.1.2 Minimum-error formulation . . . . . . . . . . . . . . . . . 563\n", "12.1.3 Applications of PCA . . . . . . . . . . . . . . . . . . . . . 565\n", "12.1.4 PCA for high-dimensional data . . . . . . . . . . . . . . . 569\n", "12.2 Probabilistic PCA . . . . . . . . . . . . . . . . . . . . . . . . . . 570\n", "12.2.1 Maximum likelihood PCA . . . . . . . . . . . . . . . . . . 574\n", "12.2.2 EM algorithm for PCA . . . . . . . . . . . . . . . . . . . . 577\n", "12.2.3 Bayesian PCA . . . . . . . . . . . . . . . . . . . . . . . . 580\n", "12.2.4 Factor analysis . . . . . . . . . . . . . . . . . . . . . . . . 583\n", "12.3 Kernel PCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586\n", "12.4 Nonlinear Latent Variable Models . . . . . . . . . . . . . . . . . . 591\n", "12.4.1 Independent component analysis . . . . . . . . . . . . . . . 591\n", "12.4.2 Autoassociative neural networks . . . . . . . . . . . . . . . 592\n", "12.4.3 Modelling nonlinear manifolds . . . . . . . . . . . . . . . . 595\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599\n", "13 Sequential Data 605\n", "13.1 Markov Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607\n", "13.2 Hidden Markov Models . . . . . . . . . . . . . . . . . . . . . . . 610\n", "13.2.1 Maximum likelihood for the HMM . . . . . . . . . . . . . 615\n", "13.2.2 The forward-backward algorithm . . . . . . . . . . . . . . 618\n", "13.2.3 The sum-product algorithm for the HMM . . . . . . . . . . 625\n", "13.2.4 Scaling factors . . . . . . . . . . . . . . . . . . . . . . . . 627\n", "13.2.5 The Viterbi algorithm . . . . . . . . . . . . . . . . . . . . . 629\n", "13.2.6 Extensions of the hidden Markov model . . . . . . . . . . . 631\n", "13.3 Linear Dynamical Systems . . . . . . . . . . . . . . . . . . . . . . 635\n", "13.3.1 Inference in LDS . . . . . . . . . . . . . . . . . . . . . . . 638\n", "13.3.2 Learning in LDS . . . . . . . . . . . . . . . . . . . . . . . 642\n", "13.3.3 Extensions of LDS . . . . . . . . . . . . . . . . . . . . . . 644\n", "13.3.4 Particle filters . . . . . . . . . . . . . . . . . . . . . . . . . 645\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646\n", "14 Combining Models 653\n", "14.1 Bayesian Model Averaging . . . . . . . . . . . . . . . . . . . . . . 654\n", "14.2 Committees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655\n", "14.3 Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657\n", "14.3.1 Minimizing exponential error . . . . . . . . . . . . . . . . 659\n", "14.3.2 Error functions for boosting . . . . . . . . . . . . . . . . . 661\n", "14.4 Tree-based Models . . . . . . . . . . . . . . . . . . . . . . . . . . 663\n", "14.5 Conditional Mixture Models . . . . . . . . . . . . . . . . . . . . . 666\n", "14.5.1 Mixtures of linear regression models . . . . . . . . . . . . . 667\n", "14.5.2 Mixtures of logistic models . . . . . . . . . . . . . . . . . 670\n", "14.5.3 Mixtures of experts . . . . . . . . . . . . . . . . . . . . . . 672\n", "Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674\n", "Appendix A Data Sets 677\n", "Appendix B Probability Distributions 685\n", "Appendix C Properties of Matrices 695\n", "Appendix D Calculus of Variations 703\n", "Appendix E LagrangeMultipliers 707\n", "References 711" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 }