{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 线性回归总结\n",
    "\n",
    "我们先回顾一下线性回归问题，线性模型一般表示成：\n",
    "\n",
    "$$\n",
    "h(x) = h_\\theta(x) = \\theta_0 + \\theta_1x_1 + ... + \\theta_dx_d\n",
    "$$\n",
    "\n",
    "也可以写成向量表示：\n",
    "\n",
    "$$\n",
    "h_\\theta(x) = \\theta^TX\n",
    "$$\n",
    "\n",
    "最常用的求解方法是 **最小二乘法**，它采用 **平方损失** 作为损失函数：\n",
    "\n",
    "$$\n",
    "J(\\theta) = \\sum_{i=1}^m(h_\\theta(x^{(i)}) - y^{(i)})^2\n",
    "$$\n",
    "\n",
    "要求解的模型参数就是损失函数取最小值时参数取值：\n",
    "\n",
    "$$\n",
    "\\theta = \\min_{\\theta} J_\\theta\n",
    "$$\n",
    "\n",
    "最小二乘法有两种常见的求解思路，一种使用正规方程：\n",
    "\n",
    "$$\n",
    "\\theta = (X^TX)^{-1}X^Ty\n",
    "$$\n",
    "\n",
    "另一种使用优化算法梯度下降：\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j + \\alpha(y^{(i)} - h_\\theta(x^{(i)}))x_j^{(i)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 逻辑回归总结\n",
    "\n",
    "逻辑回归模型一般表示成：\n",
    "\n",
    "$$\n",
    "h_\\theta(x) = g(\\theta^Tx) = \\frac{1}{1 + e^{-\\theta^Tx}}\n",
    "$$\n",
    "\n",
    "其中，$g(z) = \\frac{1}{1 + e^{-z}}$，假设预测分类的概率满足伯努利分布：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "P(y = 1 | x; \\theta) &= h_\\theta(x) \\\\\n",
    "P(y = 0 | x; \\theta) &= 1 - h_\\theta(x)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "我们把上面两个式子合在一起，有：\n",
    "\n",
    "$$\n",
    "P(y\\ | x; \\theta) = h_\\theta^{y}(x) (1-h_\\theta(x))^{1-y}\n",
    "$$\n",
    "\n",
    "根据 **极大似然估计** 我们有：\n",
    "\n",
    "$$\n",
    "L(\\theta) = \\prod_{i=1}^m h_\\theta^{y^{(i)}}(x^{(i)}) (1-h_\\theta(x^{(i)}))^{1-y^{(i)}}\n",
    "$$\n",
    "\n",
    "对其取对数得到逻辑回归的损失函数：\n",
    "\n",
    "$$\n",
    "\\ell(\\theta) = \\log L(\\theta) = \\sum_{i=1}^m y^{(i)}\\log h_\\theta(x^{(i)})) + (1-y^{(i)}) (1 - h_\\theta(x^{(i)}))\n",
    "$$\n",
    "\n",
    "和线性回归一样，可以使用梯度下降法求 $\\ell(\\theta)$ 的最小值：\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j + \\alpha(y^{(i)} - h_\\theta(x^{(i)}))x_j^{(i)}\n",
    "$$\n",
    "\n",
    "可以看到梯度下降公式和线性回归是一样的，差别在于 $h_\\theta(\\cdot)$，线性回归的 $h_\\theta(x) = \\theta^TX$，而逻辑回归的 $h_\\theta(x) = g(\\theta^Tx)$，这个 $g(\\cdot)$ 通常称为**连接函数**（link function，或称为联系函数），它必须是单调可微的，通过连接函数得到的模型称为**广义线性模型**（generalized linear model）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 线性回归和逻辑回归的比较\n",
    "\n",
    "回归问题中经常使用基于欧式距离的损失函数，而分类问题中，损失函数是基于概率分布，学术上被称为 **交叉熵**（Cross Entropy）。虽然两种损失函数的形式不同，但是它们的值都对应着模型的预测误差，因此这个值越小越好，这也是损失函数参数估计的基本原则。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}